Instance segmentation

The annotation that is worst by hand and free by render.

A bounding box is four numbers a person can produce in a few seconds. A mask is a boundary, and two careful annotators will disagree about where it goes. From a rendered scene there is nothing to disagree about: the renderer already decided which object owns which pixel, and the mask is that decision written down.

Ground truth: per-object pixel mask
Formats: COCO polygons, raw ID buffer
Instances in the sample: 1,813
Classes: 6

The identity buffer of an airbase frame, every object in its own colour — Per-instance identity bufferMap_Airbase_Demo, frame 000004Unreal Engine 5.8, 1920×1080

The schema, and the thing beside it

Masks ship twice. Once as COCO polygons, which is what most segmentation code reads, and once as the identity buffer itself, which is what the polygons were traced from. Shipping both is the point: you can check one against the other, and you can derive a representation nobody thought to export.

# COCO: a polygon per instance, plus the true pixel area
{ "image_id": 4, "category_id": 2,
  "bbox": [1697, 564, 223, 200],
  "area": 26339,
  "segmentation": [[1715, 580, 1697, 649, 1698, 652, ...]] }

# and the buffer it came from
segmentation/plugin_000004.png    # one flat colour per instance, 0 tolerance

import cv2, numpy as np

seg = cv2.imread("segmentation/plugin_000004.png")[:, :, ::-1]
colours, counts = np.unique(seg.reshape(-1, 3), axis=0, return_counts=True)
print(len(colours) - 1, "instances in this frame")   # minus the background

mask = np.all(seg == colours[7], axis=-1)            # one object, exactly
print(mask.sum(), "pixels")

Two problems this format solves that annotation does not

Both are invisible in a box dataset and both matter as soon as objects overlap.

Occlusion, exactly

An object behind another has the mask of the part that is visible, to the pixel. No interpolation across the occluder, no annotator guessing where the hidden half would have been.

Fragments, reassembled

An object seen through a railing arrives as several disconnected patches. Those are merged back into one instance rather than counted as several: 1,409 fragments across the reference capture.

And one it does not

Amodal masks, the full extent of an object including the hidden part, are not produced. The engine knows it; the capture does not currently write it. If you need amodal supervision, this is not it yet.

Limitations, specific to segmentation

The failure mode below is the one the reference capture actually hit, which is why it leads.

Cross-painting. When a thin object sits in front of another, the identity buffer can assign contested pixels to the wrong one, and the loser ends up with no mask at all. On the reference capture the verification pass found three of ninety visible people with no annotation for exactly this reason: a 3.3% false-negative rate on that class, published on the quality page rather than quietly absorbed.

Polygon fidelity. Masks are traced to polygons for the COCO export, which is lossy at very small sizes. Three annotations in the reference training split carry a box and an empty polygon because the visible region was too small to trace. The raw buffer still has them, which is the argument for shipping it.

Colour budget. The scheme depends on every object having a unique identity colour. The reference capture used 289 colours across 289 classes without collision, and the check runs on every capture because a collision silently merges two objects into one.