The capture pipeline

What happens inside Unreal between pressing Generate and having a raw dump on disk. Everything on this page is native: it runs in the engine, and none of it can be redone without rendering again.

Per frame, in order#

1. Populate

   The spawner resolves its rules against the zones and places actors. Every decision, accepted or refused, is written to that frame's spawn manifest before anything is rendered.

2. Place the camera

   A camera position is sampled inside its zone according to the camera policy. It is a candidate at this point, not a decision.

3. Validate the viewpoint

   The candidate is tested: how many targets are in frame, how much of the scene is visible, whether depth is valid, whether the camera is inside geometry. A placement that fails is rejected and another is sampled, up to the retry budget.

   This is why the reference capture tried 181 camera placements to accept 40 frames. The rejections are the point; each one is a frame that would have been of nothing.

4. Apply the environment

   Weather and time of day are applied and allowed to settle. Settling is not instant: clouds move, lighting rebuilds, and capturing mid-transition produces a frame that does not match its own recorded environment.

5. Render RGB

   The frame as a camera in that position would see it, at the declared quality settings.

6. Render the identity buffer

   The same frame again, with every object painted a unique flat colour and no lighting. This is the pass every label is later derived from, and it is the expensive one: the benchmark shows the ID pass is a material share of per-frame cost.

7. Write depth

   Float metres per pixel, when depth is enabled. It is optional because it is large: roughly 8 MB per 1080p frame.

8. Record the frame

   Camera pose and field of view, every actor's position and label, the environment state, the viewpoint validation result and the performance timings. Written as frame.json.

Frames complete or they do not exist#

A frame is finalised atomically. A run that is interrupted mid-frame leaves the previous frames intact and does not leave a half-written one behind for the labeller to choke on.

This is what makes --resume meaningful, and what makes a 50,000-frame unattended run survivable.

Performance profiles#

Capture exposes profiles rather than a wall of individual switches. The one to use for a normal training workflow is the balanced profile; the others exist for when you have a specific reason.

The knobs that actually move the cost are the resolution, whether depth is written, and how many identity targets are in view. See scene authoring for the capture performance section, and operations for long runs.

What comes after#

Nothing above is a label. Boxes, masks and datasets are all derived afterwards, over the dump, by the dataset pipeline.