Cross-split leakage
Consecutive frames of one capture look nearly identical. If one lands in train and its neighbour in validation, your validation score measures memorisation. Images are hashed and compared across splits.
Dataset validation
A generator that cannot fail is a generator you cannot trust.
Producing a million labelled frames is the easy half. Knowing whether they are correct is the half that decides whether the dataset is worth anything, and it is the half most synthetic pipelines leave to the person who downloads the result.
- Checks on every capture
- 13
- Failed on the reference run
- 1
- Dataset grade
- A · 92.4/100
- Raw health
- B · 82/100
Two layers, asking different questions
Quality gates ask whether the capture is usable at all. Are there empty frames, missing files, black renders, enough instances per frame? These are cheap, blunt and fail-closed: a run that trips one stops rather than exporting something that looks fine in a file listing.
Verification checks ask the harder question: do the labels agree with the geometry that produced them? Actor positions are projected through the camera and tested against the blobs in the identity buffer. Identity colours are checked for collisions. Two frames are labelled twice and compared byte for byte to confirm the pipeline is deterministic.
The second layer is only possible because the capture records both the pixels and the transforms. A pipeline that produced masks alone would have nothing to check them against.
Every check on the reference capture
Including the one that failed. A validation feature that only shows green results is a screenshot, not a feature.
12/13checks passed on this run. The one that did not is listed below with the rest.
| Check | Kind | Result | Recorded message |
|---|---|---|---|
| empty-frame-rate | Quality gate | PASS | |
| empty-seg-rate | Quality gate | PASS | |
| total-instances | Quality gate | PASS | |
| mean-instances-per-frame | Quality gate | PASS | |
| black-rgb | Quality gate | PASS | |
| missing-files | Quality gate | PASS | |
| small-object-rate | Quality gate | PASS | |
| determinism | Verification | PASS | |
| configuration-snapshot-integrity | Verification | PASS | |
| spawn-manifest-integrity | Verification | PASS | |
| color-collisions | Verification | PASS | |
| pixels-vs-transforms | Verification | PASS | |
| missing-visible-people | Verification | FAIL |
What the failing check found
This is the part worth reading, because it is the part that tells you what validation is for.
The reference capture failed missing-visible-people. Three of ninety visible people in the inspected sample had no bounding box, a 3.3% false-negative rate on that class. Their pixels were claimed by a neighbouring instance in the identity buffer, so from the labeller’s point of view they were not there.
Nothing about that failure is visible in the rendered frames. You would find it by training a model, noticing it under-detects people in crowded areas, and spending a week working out why. The check finds it in seconds because it compares two independent descriptions of the same scene.
The result is published here, in the report inside the downloadable sample, and on the quality page. Hiding it would not make the dataset better; it would only move the discovery to whoever trained on it.
Checks that run on the export, not the capture
A dataset can be built from a perfectly good capture and still be broken by how it was split.
Near-duplicate images
Within a split, near-identical frames waste training time without adding information. Detected by perceptual hash, reported as a count rather than silently removed.
Corrupt and black frames
An image that will not decode, or that rendered black because the capture ran before the scene finished streaming. Both are counted, and both are gates rather than warnings.
Empty annotations
A frame with no objects is a legitimate negative example. A dataset that is mostly empty files is a broken run. The distinction is a threshold you set.
# grade an exported dataset and write both machine and human reports
nameframe validate _out/dataset --out _out/validation --fail-under 85
# validation/validation.json carries the score, the grade, per-format reports,
# every check with its count and penalty, and the top issues in order--fail-under makes the grade a build gate: below the threshold, the command exits non-zero and your pipeline stops. The reference export scored 92.4 out of 100, grade A, across 4,378 instances in 40 frames.