From identity pixels to labels

Nothing Unreal writes is an annotation. Boxes, masks and keypoints are all derived afterwards, in Python, from the identity buffer and the recorded scene. This page is that derivation.

What the derivation reads#

seg.png       one flat colour per instance
frame.json    every actor, its label, its position, the camera
capture.json  the colour-to-actor mapping and the QA thresholds
depth.npy     distance per pixel, when present

The colour mapping is the hinge. Each declared class carries the identity colours assigned to its actors, so a pixel's colour is looked up rather than classified.

The derivation#

1. Find the blobs

   Every distinct colour in the buffer that maps to a declared actor becomes a candidate instance, with its pixel set.

2. Reassemble fragments

   An object seen through a railing arrives as several disconnected patches of one colour. They are merged into one instance rather than counted as several, which is the difference between one person and four stripes.

3. Derive geometry

   The box comes from the extremes of the visible pixels. The mask comes from the pixels themselves. Keypoints are projected from native bones for pose-enabled classes.

4. Apply the QA gates

   Four thresholds decide whether an instance survives:

   min_box_px        smallest allowed box side
   max_box_px        largest, when bounded
   max_aspect_ratio  reject implausibly elongated boxes
   min_fill_ratio    reject boxes their object barely occupies

   A rejected instance is dropped here rather than exported as a target nothing can learn.

5. Attach metadata

   Class, instance id, pixel count, distance in metres and the actor it came from.

Why the published count is smaller than the visible count#

A scan of the identity buffer counts more instances than a dataset publishes, and the difference is these gates. An instance eight pixels across is visible in the buffer and is not a useful training target.

Both numbers are true; they answer different questions. What matters is knowing which one you are looking at.

Why this is offline#

Because it can be run again. A new format, a different threshold, a rule that did not exist at capture time: all of them are a re-derivation over a dump you already have. See architecture.