The data is dangerous to collect
Collisions, spills, fires, people in places they should not be. You need thousands of examples of the thing your safety process exists to prevent.
Synthetic data
Synthetic data for computer vision, without the sales pitch.
Synthetic data means several unrelated things depending on who is saying it. This page is about one of them: images rendered from a three-dimensional scene, together with labels derived from that scene rather than drawn by a person. What it is good for, what it costs, and how to tell whether it is working for you.
- Scope
- rendered computer vision data
- Not covered
- tabular, text, privacy
- Worked example
- 40 real frames
- Instances
- 4,378
What the phrase does and does not mean here
Search for “synthetic data” and you will find four different industries sharing a name. There is tabular synthetic data, used to share records without sharing people. There is text generated by language models for training other language models. There is privacy-preserving data synthesis in finance and healthcare. And there is the one this page is about: rendered imagery for computer vision.
They have almost nothing in common beyond the word. The techniques are different, the failure modes are different, and the question “does synthetic data work?” has a different answer in each. Everything below is about vision.
Three ways people make synthetic images
- Augmentation. Take real images and transform them: flip, crop, recolour, paste objects. Cheap and effective, but it cannot invent a viewpoint or an object you never photographed.
- Generative models. Diffusion and GAN output. Visually impressive, and genuinely useful for some tasks, but the labels are the problem: a model that generates an image does not know where the objects are, so you are back to annotating, or to trusting another model’s guess.
- Rendering from a 3D scene. Build or acquire a scene, place a camera, render. The labels come from the scene description, which is the entire advantage: they are known, not estimated.
The third is what a game engine is unreasonably good at, because it has spent thirty years optimising exactly this operation for other reasons.
What you actually get, per frame
The reason to render rather than photograph is not the picture. It is everything that arrives alongside it.
| Modality | What it is | What it costs to get from real data |
|---|---|---|
| RGB | The rendered image | A camera and someone to hold it |
| 2D boxes | Extremes of each object's visible pixels | Seconds per object, per frame, by hand |
| Instance masks | Exact per-object pixel membership | Minutes per object, and annotators disagree |
| Depth | Metres per pixel, float | A depth sensor, with its own noise and range limits |
| 3D positions | Every object's location, in metres | Motion capture, or not at all |
| Camera pose | Position, rotation, intrinsics | Calibration rigs and careful logging |
The row that changes projects is the third one. Instance masks are the most expensive annotation in common use and the one most likely to be inconsistent between annotators. From a render they are free and exact, because the renderer already had to decide which object owns which pixel in order to draw anything at all.
When it is worth doing
Synthetic data is not free. You pay in scene work, and you pay again in the gap between a render and the world. Here is when that trade is obviously good.
The data does not exist yet
A product not yet manufactured, a site not yet built, a sensor not yet installed. The model has to be ready before the world is.
The labels are the bottleneck
You have plenty of footage and no budget to annotate it. A dense aerial frame takes an hour by hand; the reference capture on this site averages 109 annotated instances per frame.
You need controlled variation
One variable changes and everything else holds still. That is an ablation you can trust, and it is close to impossible with collected data.
When it is not
The honest cases against, which you will find yourself within a month if nobody tells you now.
You already have plenty of labelled real data
If collection and annotation are cheap for your problem, use them. Synthetic data earns its cost where real data is expensive, not everywhere.
Your task depends on appearance you cannot model
Fine-grained texture, material subtleties, sensor artefacts specific to your camera. A renderer approximates these, and for some tasks the approximation is the whole problem.
You have no way to measure the result
Synthetic data without a real validation set is a guess. You need enough real data to evaluate on, even if you have nowhere near enough to train on.
You need it this afternoon
Once a scene exists, frames are minutes. Getting the scene to exist is real work, and anyone who tells you otherwise is selling something.
The domain gap, concretely
This is the objection everyone raises and it is the right one. What matters is being specific about where the gap actually lives.
A render differs from a photograph in ways that are easy to list and hard to eliminate: sensor noise, lens imperfection, chromatic aberration, rolling shutter, compression artefacts, the endless irregularity of real surfaces, and lighting that never quite behaves like a light transport approximation.
Some of that can be simulated. Lens profiles, exposure behaviour, weather and time of day are parameters rather than accidents, so they can be varied deliberately and widely, which is itself a form of robustness training. Some of it cannot, and pretending otherwise is how synthetic data got its reputation.
How to find out what it costs you
- Train on synthetic, evaluate on real. The number you get is the only one that matters, and it is specific to your task.
- Then fine-tune on a small real set. The common finding across the literature is that synthetic pre-training plus a modest real set beats the real set alone. Whether that holds for you is an experiment, not a promise.
- Compare the distributions, not just the score. If your synthetic set covers 40 to 120 metres and your real deployment is at 15, the gap you are measuring is coverage, not realism.
NameFrame ships tooling for the third one, because a distribution mismatch is both the most common cause of disappointing results and the cheapest to fix.
What to look for in a generator
Independent of which tool you pick, these are the questions worth asking. They are the ones this site tries to answer about itself.
Where do the labels come from?
Derived from the scene, or estimated afterwards by a model? The first is a measurement. The second is a dataset of one model’s opinions.
Is it reproducible?
Same seed and same configuration, same frames? If not, you cannot re-run an experiment, and you cannot debug a bad batch.
What does it refuse to do?
A generator with no failure modes has no checks. Ask what it rejects, and ask to see a run where something failed.
Can you see the distributions?
Distance, object size, view angle, class balance. Without them you are buying frames by the thousand and hoping.
For the capture used throughout this site: labels come from the engine’s identity buffer, the run is seeded at 67 with a hashed configuration snapshot, twelve of thirteen checks passed and the one that failed is published, and the distributions are plotted from the buffers rather than authored. Instances sit between 33.2 and 187.8 metres from the camera, with a median object width of 23.26 pixels.