How to choose an Unreal Engine synthetic data tool

Ask this first, because it is the one nobody asks and it decides whether the rest matters. There are three common answers and they are not equivalent.

The tool renders a segmentation image and stops. You get pixels coloured by class or by object, and turning those into boxes, into instance counts, into anything a detector trains on, is code you write and maintain.

The tool projects each object’s bounding volume through the camera. This is fast and it is wrong the moment anything is occluded: the box covers the whole object including the half hidden behind a wall, and a model trained on it learns to hallucinate the hidden half.

The tool measures the box from the pixels the object actually drew. This is the only one of the three that gets occlusion right, and it is the reason a separate identity pass exists at all.

Set a seed, run it twice, compare the two folders byte for byte. If they differ, you cannot ablate anything: every experiment you run afterwards is measuring your change plus whatever else moved.

This is harder than it sounds and most of the difficulty is not in the random number generator. Level streaming, auto exposure adapting over frames, physics settling for a different number of ticks, an animation that starts wherever the editor left it: all of these make two runs differ while the seed stays the same. Ask what the seed actually controls.

The expensive failure in synthetic data is not rendering too few images. It is rendering a hundred thousand pictures of the same scene from a hundred thousand angles and calling it a dataset. A model trained on that learns the scene.

So ask what varies between frames beyond the camera: object identity, position, pose, material, lighting, weather, time of day, and whether anything stops two objects being placed inside each other. A tool that only moves the camera is a renderer, and you will be writing the rest.

Keyframing a path is fine for a hundred frames and unworkable for fifty thousand. At that size you want the camera sampled under constraints: a distance range, a height range, keep the subject in frame, reject the viewpoint if the subject is behind a wall or the frame is nearly black.

The question behind the question is what happens to a bad viewpoint. Silently keeping it puts unusable frames in your training set. Rejecting and resampling costs render time and gives you a dataset whose camera distribution is no longer the one you asked for, which you should also know about.

Synthetic labels are widely assumed to be correct because a computer made them. They are correct only if the pipeline that made them is, and pipelines have bugs that produce plausible, wrong data at scale.

Ask whether anything reopens the output and disagrees with it: are the boxes where the geometry says they should be, do two objects share an identity colour, does labelling the same frame twice give the same answer. A check that has never failed has never been tested, so ask to see one failing.

YOLO and COCO are the two formats most training code reads. If a tool does not write them, you write the converter, and the converter is where the coordinate convention bugs live: origin at the corner or the centre, y down or y up, normalised or pixels, class ids from zero or from one.

Also ask what is kept that the export throws away. Depth in metres, camera intrinsics, per-instance visibility: these are cheap to record at capture time and impossible to recover later.

A dataset is rarely built once. It is built, looked at, adjusted and built again, usually overnight. If every run needs a person clicking in the editor, that loop is as long as your working day.

Ask whether a run can be started from a command line, whether it can be resumed after a crash, and whether the settings live in a file you can diff and commit.

The most boring question and the one that ends most evaluations. A plugin built against 5.2 does not load in 5.8. Porting it is possible and it is your afternoon, not theirs, and it is an afternoon you will spend again at the next engine upgrade.

Ask which versions are verified rather than which are claimed, and what verified means to them.

The whole field has a shortage of this. Plenty of tools show you a rendered frame with a neat box on it. Far fewer show a model trained on their output and measured on real images, because that number is often disappointing and publishing it is uncomfortable.

Ask for it anyway, and treat an honest bad number as better evidence than a good one with no method attached. If the answer is a synthetic test split, that measures the renderer against itself and tells you nothing about your real footage.

Two of the tools compared on this site are MIT licensed and free. That is a real advantage: you can read the code, change it, and nobody can withdraw it. The cost is that when it breaks at midnight it is yours to fix, and when the engine moves it is yours to port.

Paying for something buys the opposite trade and only that. Decide which of the two failure modes you would rather have at the point in the project where it will actually happen.

We built NameFrame around questions one, two, three and five, because those were the ones we kept getting wrong ourselves. The boxes are measured from an identity pass rather than projected. A seed replays a run. Scene variation is a first-class setting rather than a script. The output is graded and gated before it can be exported, and the report is published including the checks that failed.

Question nine is the one where we come off worst and it is on the site anyway. The single end-to-end experiment we have run trained a detector only on our frames, improved on the synthetic split, and lost recall on real drone footage. It is written up with the failure pictures.

On question eight we support one engine version, 5.8. On ten we are a commercial preview and not on sale, which for some readers ends the conversation, and that is a reasonable place for it to end.