Quality, checks and scoring
A synthetic dataset that looks right and is subtly wrong costs more than no dataset at all. NameFrame checks in two places: pre-flight gates against the raw dump before labelling, and a scored validation of the assembled dataset afterwards.
The 12 pre-flight checks#
nameframe verify <dump> runs all of them and writes
verify.json into the dump. Each one exists because of a real failure.
| Check | What it proves |
|---|---|
schema | The dump matches the raw capture contract at all. |
configuration-snapshot-integrity | The recorded configuration matches what was actually run. |
spawn-manifest-integrity | The spawn manifest agrees with the actors present in the frames. |
color-collisions | No two instances share an ID colour. A collision merges two objects into one label. |
determinism | Re-running the same seed reproduces the same frames. |
fragments | Instances are not shattered into disconnected pixel islands. |
pixels-vs-transforms | The pixels agree with the recorded transforms. This is the core anti-drift check. |
missing-visible-people | Nothing visible in the image is missing from the labels. |
native-bones-vs-offsets | Engine bone positions agree with derived offsets, within 0.30 m. |
environment-ranges | Environment values actually varied across the run. |
actor-ground-contact | Actors stand on the ground instead of floating or sinking. |
population-spacing | The population respects the configured minimum spacing. |
If a check raises, it gets reported as failed, loudly, rather than skipped. Silence is never treated as success.
Reading a failure
| Failing check | Usually means | Fix |
|---|---|---|
color-collisions | Too many instances for the ID colour space, or duplicated IDs | Reduce instances per frame, or check for duplicated actors |
pixels-vs-transforms | The scene moved after the transform was recorded | Raise settle time, and check that nothing animates mid-capture |
missing-visible-people | Actors are visible but unassigned to a class | Fix the class assignment, or mark them ignored |
environment-ranges | You configured a range but got one value | The randomiser is not being applied. Check the weather driver and the recipe |
actor-ground-contact | No ground actor was hit | Set the ground actor prefix, or pick ground on the zone |
population-spacing | Spacing too large for the zone area | Enlarge the zone, lower the count, or reduce spacing |
Dataset scoring#
nameframe validate <dataset> inspects the assembled dataset rather than
the dump, and scores it. Every check contributes a bounded, documented penalty, subtracted from
100.
nameframe validate _demo/dataset --out _demo/validation| Score | Grade | Reading |
|---|---|---|
| 90 to 100 | A | Ship it. |
| 80 to 89 | B | Usable. Read the penalties before scaling up. |
| 70 to 79 | C | Something real is wrong. Fix it before a large run. |
| 60 to 69 | D | Do not train on this. |
| under 60 | F | A mechanism is broken, not a parameter. |
When both YOLO and COCO are exported, each gets scored separately and the reported score is the lower of the two. A dataset is only as good as its worst format.
What the validator looks at
- Images: corrupt files, unreadable images, blank or near-blank frames, brightness and contrast outliers.
- Duplicates and leakage: perceptual hashing finds near-identical images and flags the ones split across train and val.
- Label integrity: malformed lines, out-of-range coordinates, degenerate boxes, class indices with no name.
- Class balance: per-class counts and size buckets, so a class that is nominally present but actually rare becomes visible.
- Split health: empty splits, wildly skewed fractions.
- Format conformance:
data.yamlagreement with the labels on disk, COCO structural validity, andkpt_shapeplusflip_idxfor pose.
Gating on the score
nameframe generate --demo --samples 200 --seed 7 --output _local/runs/x --fail-under 80--fail-under makes the process exit non-zero below the threshold, which is all
CI needs. See the CI section.
Reproducibility and corruption#
nameframe hardening-smoke _build/hardening --frames 8 --seed 42An offline smoke test that checks the pipeline is reproducible and that corrupt inputs get rejected rather than silently mislabelled. Fast enough to run in CI.
Comparing runs#
A single score tells you whether a dataset is sound. Comparison tells you whether a change helped.
nameframe metadata-compare _local/metadata.sqlite --baseline run_a --candidate run_bIt compares stable metrics and frozen configuration between two indexed runs, so a difference in the numbers is attributable to a difference in the configuration.
Closing the loop with a model#
Dataset quality only really means "did the model get better". These commands connect the two:
Profile both sides
nameframe real-dataset-import _local/real _local/profiles/real.json nameframe dataset-profile _local/datasets/synth _local/profiles/synth.jsonMeasure the gap
nameframe compare-profiles _local/profiles/real.json _local/profiles/synth.json _local/profiles/cmp.json nameframe domain-gap _local/profiles/cmp.json _local/reports/gapEvaluate by slice
nameframe model-eval _local/results.json _local/eval.json nameframe failure-mine _local/eval.json _local/failures.jsonGenerate against the failures
nameframe hard-cases _local/failures.json _local/plans/hard_cases.jsonGate the next release
nameframe regression-campaign --name nightly --evaluations _local/eval.json --out _local/campaign.json
Coverage works the same way. Declare the axes you care about, evaluate what you have, and plan corrections for what is missing.
nameframe coverage-eval _local/coverage/axes.json _local/coverage/observations.json _local/coverage/matrix.json