Reproducible synthetic data: one seed, one dataset

A determinism test on a still scene with one camera proves nothing about a real one, so this corpus was made to be the worst case available. Every frame draws new positions for seventy-four people across five spawn zones, a fresh character for each of them out of a pool of eighteen hundred, a camera position, an altitude, an orbit radius, one of eleven weather presets and a time of day between six in the morning and six in the evening.

When a viewpoint is poor the camera draws another, up to twenty-five times. That one is the sharpest test in the set: a rejected viewpoint consumes randomness, so the number of attempts a frame took is a running tally of decisions, and two runs that diverge anywhere will disagree there first and then stay wrong.

Every mask came back bit for bit identical, on all three hundred frames, and so did every depth buffer, every camera pose and every retry count. So did the weather preset and the time of day each frame drew, and every label, box, keypoint and instance id in the corpus.

That is the half a model trains on. Whatever the photograph does, the answer sheet is the same answer sheet.

A rendered image is not bit-identical between two runs. The GPU adds floating point in whatever order its scheduler chooses, and the sum of a million samples depends on that order. Across the corpus, two thirds of all pixels came back identical anyway and 94.97% landed within one grey level of each other, with a mean difference of 0.31 of 255.

The difference is not spread evenly. It sits on the sunlit facades and the lit road, while the shadowed roofs and the tree are black, meaning identical. It is proportional to how much indirect light a surface receives, which points at the engine’s global illumination converging slightly differently rather than at anything upstream of it.

The first pair of runs was not reproducible, and the way it failed is worth more than the number that replaced it.

The two runs agreed on every decision for twenty-nine frames and then diverged. The seed was not the problem. For those twenty-nine frames the weather, the times, the camera poses and the retry counts all matched and the segmentation matched bit for bit. What differed from frame zero was the photograph.

The cause. Two of the viewpoint checks read the rendered image: a luma floor and a focus score. At frame thirty one of them landed either side of its threshold, a viewpoint was re-drawn in one run and not the other, and from there the two runs consumed the random stream at different rates. A gate that reads the rendered image makes the reproducibility of the whole run depend on the reproducibility of the renderer.

The repair. Both checks are gone. What remains counts targets and reads the identity mask and the depth buffer, all three of which reproduce exactly, so the decisions made from them do too. Warm-up went from one render to sixteen at the same time, which is where the depth buffer stopped differing at all: below that the renderer’s accumulation is still converging when the frame is taken.

Routing the RGB through a dedicated scene capture rather than the editor viewport cut the remaining difference tenfold. It was rejected. That path renders flat and washed out, with no shadows and no contrast, and a corpus that reproduces exactly because it stopped being worth looking at has bought the number by throwing away the thing the number was about. Lumen, shadows, anti-aliasing and the viewport render path are all exactly as they were.

Turning off anti-aliasing, Lumen, global illumination, reflections, volumetric fog, bloom, particles, shadows and translucency all at once, and fixing the exposure, left about a hundred pixels a frame still differing. That is not a setting either.

The two jobs differ in their output directory and in nothing else, which the preparation step prints so it can be checked rather than believed. The comparator keeps three tiers apart so a weak result cannot hide behind a strong one: pixels and depth are compared with no normalisation at all, records after removing where a run was written and when, and every key removed is printed with the result.

It was checked against two runs known to differ before it was pointed at two meant to match. A comparison that cannot fail proves nothing.

The scene script, the recipe, the comparator and every number on this page are in the repository under docs/showcase/determinism.