Free space, terrain, coverage
Anything where the question is about area rather than objects: drivable surface, vegetation cover, water. Counting is not part of the task, so instances would be wasted precision.
Semantic segmentation
One colour per class, computed from one colour per object.
Semantic segmentation labels every pixel with what it is. It is the cheaper, coarser sibling of instance segmentation, and in a rendered pipeline it should never be captured separately: it is derived from the instance buffer, which is the only way to guarantee the two describe the same pixels.
- Classes
- 6
- Instances in this frame
- 105
- Across the capture
- 4,378
- Derived from
- the instance buffer
Why derivation, not capture
The obvious implementation is to render a second buffer with one colour per class. It works, and it is a mistake, because you now have two independent renders that can disagree. An object assigned to the wrong class in one and not the other produces a dataset where the instance mask and the semantic mask contradict each other, and nothing in the pipeline notices.
Collapsing the instance buffer instead makes disagreement impossible. Every pixel already belongs to exactly one object, and every object already belongs to exactly one class, so the semantic mask is a lookup rather than a second opinion.
What you lose going from instance to semantic
Adjacency. Two crates touching in the frame are two objects in the instance buffer and one continuous blob in the semantic one. That is fine if your task is “which pixels are crate” and fatal if it is “how many crates are there”.
The direction matters: instance to semantic is a lossless-to-lossy projection you can always perform, while semantic to instance is not recoverable at all. That is why the capture keeps the instance buffer and derives the rest.
The classes in the reference capture
Six classes, wildly unequal, because the scene is. The palette on this page and the pixels in the image above come from the same table.
| Class | Pixels labelled in | Instances | Frames present |
|---|---|---|---|
| barrel | #f93636 | 1,471 | 40 / 40 |
| crate | #36f9f9 | 1,216 | 40 / 40 |
| person | #3636f9 | 996 | 40 / 40 |
| container | #36f936 | 361 | 39 / 40 |
| car | #f9f936 | 254 | 40 / 40 |
| tank | #f936f9 | 80 | 40 / 40 |
One class, container, appears in 39 of 40 frames rather than all of them, because a camera zone at one end of the airfield cannot see the containers at the other. A per-class presence count is worth reading before training: a class absent from a third of your frames is a class the model sees a third less often than the instance count suggests.
When to use which
Counting, tracking, grasping
Anything where two of the same thing must not merge. In dense scenes this is almost always what you want, and it is what the capture stores natively.
Panoptic-style work
Countable objects plus uncountable background. Both masks come from the same buffer, so they can be combined without reconciliation.