How to check a synthetic dataset before training

A broken dataset does not throw. It trains, the loss falls, the run finishes, and the model is quietly worthless. Every check here catches a failure that produces no error message, which is why they are worth an hour before the first run rather than a week after it.

Time to run all seven: Under an hour
What they have in common: None of them raise an exception
The two that need eyes: Drawn boxes, and mirrored labels
When to do it: Before the first training run

1. Count the boxes per class

The single most informative minute you can spend. A corpus where one class holds ninety per cent of the instances will train a model that predicts that class, and the accuracy will look respectable while the thing you actually cared about is never found.

The number to look at is instances, not images. Fifty images containing a hundred cars each and five hundred images containing one person each are not the balance the image count suggests.

2. Check the split for near-duplicates

This is the check that most often turns a good validation score into an honest one. Rendered frames from the same scene, the same orbit or the same few seconds look alike. If one lands in training and its near-twin in validation, the model is being tested on something it has effectively already seen, and the score is measuring memory.

Splitting by scene, by capture, or by camera position rather than by shuffling frames removes most of it. Expect the validation number to drop when you do this. That drop is the point: the old number was wrong.

3. Prove the class ids line up

Every format numbers classes and every format disagrees about where to start. YOLO counts from zero, COCO category ids conventionally start at one, and a converter between them is exactly where an off-by-one arrives. The result is a model trained to call cars people, with no error anywhere.

The check is to read the highest id in the labels and compare it against the length of the name list, then to look at one image of each class and confirm the name matches what is in the picture. Both halves are necessary: the arithmetic catches a shift past the end, and only your eyes catch a swap in the middle.

4. Look at twenty frames with the boxes drawn on

Chosen at random, not chosen well. This is the check people skip because it feels unscientific, and it is the one that catches the failures no metric reports: labels shifted by one frame, coordinates in the wrong convention, boxes drawn around the wrong half of the object, a whole class silently missing.

Twenty is enough. If something is systematically wrong it will be wrong in most of them, and if nothing looks wrong in twenty random frames the gross failures are not there.

5. Decide what an empty frame means to you

A frame with no objects is a legitimate negative for detection and teaches background. For pose or for segmentation it usually teaches nothing and costs a training step. Neither answer is wrong; not having decided is.

Count them first. A corpus that turns out to be a third empty is telling you something about how it was captured, and that is worth knowing before you tune a learning rate around it.

6. Find the labels nobody can check

The one specific to rendered data. A visibility check reads geometry: it knows how much of an object is hidden behind another object, and it answers that exactly. Fog is not geometry. Neither is dust, shadow, a dark hour, or a grey vehicle against a grey wall.

In all of those the object is scored fully visible, the label is correct, and the picture contains nothing a person could confirm. On our own published capture, 2.3% of labelled instances have neither contrast against their surroundings nor any variation inside them, which is to say there is nothing there to see.

Whether those belong in your training set is a real question with two defensible answers. Discovering them afterwards is not one of them.

Measure under the labels

Average the brightness of the pixels each mask covers rather than of the whole image. A dawn frame with a bright sky averages fine while every labelled object in the street below it is black.

Compare against the surroundings

An object the same shade as what is behind it can be perfectly bright and still indistinguishable. Contrast against a ring of background is the measure, not absolute brightness.

Two numbers, not one

A patterned object whose average matches its background has no contrast and is easy to see; the pattern gives it away. Only when contrast and internal variation are both low is there nothing to find.

7. For pose, mirror one image by hand

Only for keypoint datasets, and it takes five minutes. Flip an image and its labels using your flip map, then draw both skeletons. If a wrist ends up on the same side of the body in both, the map is wrong.

A missing or wrong flip map produces no error at all. The model is shown one limb labelled two ways in roughly equal numbers for the whole run and learns that the limb is unpredictable. Loss falls normally the entire time.

Doing it with NameFrame

Several of these are computed for you and shipped with the run rather than left as homework: the class counts, the empty-frame count, the per-instance readability, and a report with drawn boxes on sampled frames. The rest are yours, because they depend on what the corpus is for.

nameframe report      D:/runs/street D:/runs/street/report
nameframe obscuration D:/runs/street --every 5 --out readability.json

The readability measure and the numbers above come from the page about what a visibility gate cannot see, measured on a capture you can download and check.