Identity, cross-paint and withholding
The identity buffer is the foundation everything else rests on. This page is about the two ways it can mislead you, one deliberate and one a defect, and how to tell them apart.
Withheld identity, on purpose#
Some actors should appear in the picture and never in the labels: scenery, distractors, the building itself. Those are rendered normally and carry no identity, so they never become instances.
The report counts them separately as identity_withheld. That count is what
turns "this object has no label" from an ambiguous observation into a decision you can look
up.
Colour collisions#
The scheme depends on every object having a unique colour. If two share one, they merge into a single instance, and nothing downstream can tell.
Every capture is therefore checked for collisions, with a colour tolerance of zero. Not "close enough": anti-aliasing produces colours that are nearly an object's identity, and accepting near-matches is how a pipeline invents instances at object boundaries.
The reference capture reported 289 colours across 289 classes with no collisions. That check exists because the failure is silent, not because it fires often.
Cross-painting#
The subtler failure, and the one the reference capture actually hit.
When a thin object sits in front of another, the pixels along the boundary are contested. The identity buffer assigns each to one object or the other, and the loser can end up with so few pixels that it falls below the QA gates and gets no annotation at all. The object is plainly visible in the RGB frame and absent from the labels.
Three of ninety visible people had no bounding box: a 3.3% false-negative rate on that class, published in the report rather than filtered out. Nothing about it is visible in the rendered frames. You would otherwise find it by training a model, noticing it under-detects people in crowded areas, and spending a week working out why.
How it is caught#
By comparing two independent descriptions of the same scene. Recorded actor positions are projected through the camera and tested against the blobs the buffer says belong to them. An actor that is visible, projects into frame, and has no blob is a false negative.
This is only possible because the capture records both the pixels and the geometry. A pipeline that produced masks alone would have nothing to check them against, and would report a clean run.
What to do about it#
- Read the check result. It is in every report, with the affected instance ids.
- Fewer overlapping thin objects in the same depth band reduces contested boundaries. Spacing and placement rules are the lever.
- Capture at a higher resolution: contested boundaries are a fixed number of pixels wide, so more pixels per object means fewer objects lost to them.
- Decide deliberately whether a 3% false-negative rate on one class matters for your task. Sometimes it does not. The point is deciding rather than not knowing.