Tutorial: headless and scripted runs

Generating datasets without anyone watching: a single output root, resumable jobs, a persistent queue, and a CI gate that fails the build when dataset quality drops.

One command, one output root#

generate is the headless entry point. It takes one of three sources and produces every artifact under a single root.

nameframe generate --job _local/jobs/beach.json --output _local/runs/beach_001 --dataset-task box --dataset-format yolo,coco --inspect --fail-under 80

Under _local/runs/beach_001 you get dump/, report/, dataset/, validation/ and inspector/. One folder per run, nothing to wire up.

Surviving interruptions#

Long runs get interrupted, whether by a crash, a reboot or a cancelled job. Both generate and run autosave progress and accept --resume.

nameframe generate --job _local/jobs/beach.json --output _local/runs/beach_001 --resume

On resume, capture gets skipped entirely if the dump already holds every requested frame, and the offline stages continue from the last completed step recorded in job_state.json.

Queueing several runs#

The local persistent queue runs commands sequentially and survives a restart.

nameframe queue-submit _local/queue.json -- generate --job _local/jobs/a.json --output _local/runs/a
nameframe queue-submit _local/queue.json -- generate --job _local/jobs/b.json --output _local/runs/b
nameframe queue-list _local/queue.json
nameframe queue-run _local/queue.json

Useful pattern: queue a seed sweep overnight.

foreach ($seed in 1..8) {
  nameframe queue-submit _local/queue.json -- generate --demo --samples 500 --seed $seed --output _local/runs/seed_$seed
}
nameframe queue-run _local/queue.json

A CI quality gate#

--fail-under makes the process exit non-zero when the dataset score drops below a threshold, which is all a CI system needs.

nameframe generate --demo --samples 4 --seed 7 --output _build/ci-smoke --dataset-task box --dataset-format yolo,coco --fail-under 80

That is the exact smoke test this repository's own CI runs on every push, after the unit suite, on Python 3.11 and 3.12.

What to gate on

Unreal automation tests are a separate engine-side gate. They are not part of the Python CI job.

Scaling past one machine#

For many jobs or many machines, the local control plane distributes work and records audit.

nameframe control-init _local/control.json
nameframe control-worker-register _local/control.json --worker-id box-01
nameframe control-job-submit _local/control.json -- generate --job _local/jobs/beach.json --output _local/runs/beach
nameframe control-shard-submit _local/control.json --shards 4 -- generate --demo --samples 40000 --output _local/runs/big
nameframe worker-run _local/control.json --worker-id box-01

Then watch it:

nameframe control-jobs _local/control.json
nameframe control-metrics _local/control.json
nameframe control-audit _local/control.json
nameframe control-logs _local/control.json

More detail in Jobs, queue and scale.

Making runs findable later#

Three runs are easy to keep track of. Three hundred are not. Build a Metadata v1 sidecar per run and index it:

nameframe metadata-build _local/runs/beach_001/dump _local/runs/beach_001/metadata
nameframe metadata-index _local/metadata.sqlite _local/runs/beach_001/metadata
nameframe metadata-index-summary _local/metadata.sqlite

Then query, compare and report without touching the raw dumps:

nameframe metadata-query _local/metadata.sqlite --limit 20
nameframe metadata-compare _local/metadata.sqlite --baseline run_a --candidate run_b
nameframe metadata-report _local/metadata.sqlite --run run_b --out _local/reports/run_b

Save a filter you use often as a view, and regenerate its report on demand:

nameframe metadata-view-save _local/views.json --name nightly --filter ...
nameframe metadata-views _local/views.json
nameframe metadata-view-report _local/metadata.sqlite _local/views.json --name nightly --out _local/reports/nightly