How to train a model on synthetic data

Our own gap, first

A person detector trained on eight rendered captures and nothing else, at drone altitude, quantized to run on an Edge TPU. Recall then precision.

Both are true. The first says the model learned the task as the renderer poses it. The second says roughly two people in three go unfound on footage from a real camera. If the only number you ever compute is the first one, you will ship something that fails and you will not know why until it does.

The real number comes from 120 VisDrone frames the model had never seen, at a confidence threshold of 0.15. Dropping the threshold to 0.10 moves recall to 0.374 and costs precision, which is the ordinary trade and does not rescue anything.

Worth saying plainly, because it is the obvious objection: that first number was not inflated by a careless split. The held-out tiles came from camera zones the model never trained on. The gap is not leakage. It is the distance between a rendered world and a real camera, and it survives doing the split properly.

Why a synthetic score runs optimistic

It is not that rendered images are unrealistic in some vague way. Three specific things inflate that first number. The first is the one most people have and we did not; the other two we have as much as anyone.

Near-duplicate frames split across the sets. Frames from one orbit of one scene look alike. If two of them land one in training and one in validation, the model is being tested on something it has effectively seen. Holding out whole captures or whole camera zones, rather than shuffling frames, is the fix, and it will lower your validation score, which is the point of it.

Labels that are correct in a way real labels are not. A rendered corpus has no missing annotations and no sloppy boxes. Real evaluation sets do. Some of the gap is your model being penalised for finding things the ground truth forgot, and some is genuinely worse detection; the two look identical in a recall number.

One distribution, sampled a lot. Three hundred frames of one map on one seed is not three hundred independent observations. It is a handful of situations, re-rolled. Variety across maps buys more than volume within one.

What actually narrows it

The check to run before you train at all

Two lines of work that save a week. First, count instances per class; a corpus where one class holds ninety per cent of the boxes will train to predict that class and little else. Second, look at twenty frames with the boxes drawn on them, chosen at random rather than chosen well.

The second one catches the failures that no metric reports: labels shifted by a frame, a class mapped to the wrong id, boxes drawn around objects the camera cannot see. That last one is more common than it sounds, because a visibility check that reads geometry cannot tell that an object is lost in fog or darkness. We measure it and publish the number.

Doing it with NameFrame

The eight captures those numbers come from are published in full, with their labels, so the experiment can be repeated rather than believed. Whole captures can be held out rather than frames shuffled, for the reason above, and the validation report ships with each run.

The full write-up of the experiment, including the changes that moved the number and the ones that did not, is on the experiments page. The captures themselves are on the datasets page.