Jobs, queue and scale

Everything about running many jobs, or long ones, without watching them: resume semantics, the local queue, the control plane, workers, storage and support bundles.

Resume#

Both run and generate autosave a job_state.json and accept --resume. On resume:

• Capture gets skipped entirely if the dump already contains every requested frame.
• Offline stages continue from the last completed step.
• Re-running a finished job is a cheap no-op.

Cancelling a Studio run behaves the same way. Completed frames stay in the dump and a later resume picks up from there.

The local queue#

A persistent JSON-backed queue that runs commands sequentially and survives a restart.

nameframe queue-submit _local/queue.json -- generate --demo --samples 500 --seed 1 --output _local/runs/s1
nameframe queue-list _local/queue.json
nameframe queue-run _local/queue.json

Everything after -- is the command to enqueue.

The control plane#

When one machine and one queue are not enough, the local control plane holds projects, users, jobs, shards, workers, metrics and a durable audit trail.

nameframe control-init _local/control.json
nameframe control-worker-register _local/control.json --worker-id box-01
nameframe control-job-submit _local/control.json -- generate --job _local/jobs/beach.json --output _local/runs/beach

Sharding a large job

nameframe control-shard-submit _local/control.json --shards 8 -- generate --demo --samples 80000 --output _local/runs/big

The split is deterministic, so shard n always covers the same slice. That is what makes a partially-failed run resumable per shard instead of from the start.

Running work

nameframe worker-run _local/control.json --worker-id box-01

A worker claims jobs it is compatible with, heartbeats while running, and releases work it cannot finish. Re-register with the same worker id to heartbeat.

Watching it

Users, tokens and roles#

nameframe auth-user-create _local/control.json --user analyst --role operator
nameframe auth-token-create _local/control.json --user analyst
nameframe auth-roles _local/control.json
nameframe auth-check _local/control.json --user analyst --permission jobs:submit

Dataset versions and lineage#

Publishing a dataset version makes it immutable and traceable: which dump each sample came from, under which configuration.

nameframe dataset-version-publish _local/control.json _local/datasets/beach --name beach --version v1
nameframe dataset-versions _local/control.json
nameframe lineage-samples _local/control.json --version <version-id>

Object storage#

Local content-addressed storage, so the same bytes stored twice occupy one entry.

nameframe object-put _local/objects path/to/file
nameframe object-list _local/objects
nameframe object-get _local/objects <sha256-or-uri> out/path

Backup and restore#

nameframe metadata-backup _local/metadata.sqlite _local/backups/metadata_2026-08-10
nameframe metadata-restore _local/backups/metadata_2026-08-10 _local/metadata.sqlite

Support bundles#

When something is wrong and you need to hand the evidence to somebody else:

nameframe support-bundle _local/support/bundle_2026-08-10

It collects the diagnostics that matter and leaves out what should not travel. Check the contents before sending it anywhere.

Release readiness#

nameframe release-readiness _local/reports/readiness

Aggregates the gates into a single readiness view, so deciding whether a build is shippable is not guesswork.

On-prem packaging#

nameframe package-onprem _local/dist/nameframe_onprem

Builds a sanitized source bundle for installation on a machine that will not clone the repository.

The Results web page#

Studio's Results page starts a loopback-only API over an existing Metadata index and opens a browser UI for indexed runs, saved views, health and run comparisons. Configure the index path, the saved-views JSON, the report folder and the local port (default 8010) on that page, then press Start Results and Open Results. Stop Results stops the process Studio started.