Which object is this?
One colour per object. This is what you need for counting, tracking, and any task where two touching objects of the same class must not merge into one blob.
Instance segmentation
Which one is this, not just what is this.
Semantic segmentation tells you a pixel is a crate. Instance segmentation tells you which crate, and that distinction is the difference between counting objects and merely detecting their presence. In a rendered scene the answer is not inferred, it is painted: every object gets its own colour in a buffer drawn for exactly this purpose.
- Instances in this frame
- 105
- Across the capture
- 4,378
- Colour tolerance
- 0
- Fragments reassembled
- 1,409
How the buffer is made
The scene is rendered a second time with every material replaced by a flat, unlit colour unique to that object. No lighting, no shadows, no transparency tricks: just identity. The result is an image in which pixel colour is object identity, and reading a mask is a matter of selecting a colour.
Two properties follow from this that are hard to get any other way. Occlusion is already resolved, because the renderer had to decide which surface was in front to draw the buffer at all. And boundaries are exact to the pixel, with no polygon simplification and no annotator’s wobble.
Colour collisions, and why tolerance is zero
The whole scheme depends on colours being unique. If two objects share one, they become one instance and nobody notices until a model trained on the result behaves strangely. So the pipeline checks every capture and matches colours exactly, with a tolerance of 0: no “close enough”, because close enough is how anti-aliasing quietly invents objects.
Reference capture result: 289 colours across 289 classes, no collisions.
Fragments
An object seen through a railing, behind a lamp post, or split by a foreground crate arrives in the buffer as several disconnected patches of the same colour. Treated naively that is several instances of one object, which inflates counts and teaches a model that a person is a collection of stripes.
Those patches are reassembled into one instance. 1,409 fragments were merged across the reference capture, and the distribution of how many pieces each object arrived in is recorded in the report rather than smoothed away.
Instances and classes are two questions
The same buffer answers both, but they are not the same answer, and a pipeline that only stores one of them has thrown away information it cannot recover.
What kind of thing is this?
One colour per class, collapsed from the instance buffer. Cheaper to look at and what a semantic segmentation model consumes, but you cannot recover instances from it afterwards.
Derived, never separately captured
The class view is computed from the instance view, so the two cannot disagree. Capturing them independently would allow exactly the kind of drift that is impossible to debug later.
What you get in the export
Masks are only useful if they arrive in a format a trainer accepts.
- The raw identity buffer as a PNG per frame, which is the source of everything else and lets you derive whatever representation you prefer.
- COCO polygons traced from that buffer, so an existing COCO pipeline works without changes.
- Per-instance metadata: class, instance id, pixel count, distance in metres and the actor it came from, per frame.
The sample packs include the identity buffers alongside the COCO annotations, so you can check one against the other rather than take this page’s word for it.