Occlusion in synthetic data: what it costs a label

Four cameras hold the same moment. One stands inside the room with a clear view, one outside the glazing, one behind a solid wall, and one inside with a partition across its view. The subjects and their instance ids are the same in all four, so a figure that is hidden from one camera can be followed to another that can see it.

Occlusion is measured rather than guessed. The engine draws the targets on their own, compares that against a full-scene depth capture, and writes the ratio for every object in every frame. A figure that is 70 per cent hidden is 70 per cent hidden because it was counted, not because it looked about right.

The camera behind the solid wall writes nothing. Not an empty box, not a box with a flag on it: no instance at all, and no skeleton either, because the pose follows the instance.

That is worth saying plainly because the opposite is easy to imply. Labelling reads the identity image, so an object with no pixels has no box. Supervision for an object that is entirely hidden, the amodal case, is not something this pipeline produces today. The machinery for it exists nearby, in the landmark projector, which places a known world point in the image and keeps it whether or not anything can see it. Extending that to an object’s extent is work that has not been done.

A pole down the middle of one figure and a bar across another’s waist, both placed on the sightline by projection rather than by eye. The mask comes apart into five pieces and three pieces respectively. The question is whether that produces two boxes over half a person each.

It does not. One box per person, and the fragment count is written into the record rather than discarded, so a consumer can filter on it.

The order of operations is the reason. Blobs are extracted, blobs that belong to one person are merged, and only then are the size gates applied. Gating each blob separately, which is the obvious way round, lets the smaller half fall under the minimum box size and vanish, leaving a box over part of a person. Measured on a real capture at the time: 38 per cent of people fragmented, six lost outright, and surviving boxes up to 338 per cent too small.

A panel sized to hide the near half of a figure without cutting it. Nothing fragments, so nothing has to be reassembled, and the mask that comes out is one unbroken blob of half a person.

The box is that half. It answers where the evidence is, not where the person is, which is the correct behaviour for a modal box and worth knowing before you train on it. Then something less defensible happens: at 22 by 166 pixels the box has an aspect ratio of 7.55, past a gate set at 6, and the label is dropped entirely with nothing reported. The skeleton goes with it. Half a standing person is shaped like nothing that gate expects, and in this scene it happens twice: two of the five figures are refused this way, and neither refusal is recorded anywhere.

There is a threshold for this. Below a set fraction of visible pixels an instance is refused rather than labelled, because past a certain point the box stops describing the object at all. The same capture labelled twice shows it working: at the shipped 0.10 every figure survives, and at 0.30 the one that is 70 per cent hidden is refused.

The third limit is that you cannot ask for it. The capture writes that threshold as a fixed 0.10, so there is no job field and no command-line flag that changes it. The figure below was produced by editing the run’s own manifest. Wiring it up is a small change and it has not been made yet.

A skeleton is thirteen joints placed from the engine, each carrying its own state: seen, hidden behind something, or outside the picture. Those are three different facts and they are kept apart, because a joint that is occluded needs different handling from one that is simply not in frame.

Occlusion does not break the skeleton. A figure standing behind a pillar with a beam across it still carries all thirteen joints in the right places, six of them flagged as hidden. A model trained on that learns where an elbow is when the elbow cannot be seen.

A figure standing two and a half metres from the camera and half out of its frame has four of its thirteen joints projecting outside the image. They are still placed. The geometry and the camera are both known, so the projection is arithmetic rather than observation, and nobody can mark a joint that is not in the picture by hand at any price.

The export cannot carry them, and that is the format rather than us: COCO and YOLO both define a keypoint outside the image as not labelled and write zeros, so the pose file keeps the count and loses the coordinate. The projector takes a policy with three settings, one of which keeps the real coordinate under a non-zero flag for a consumer who wants that geometry supervised. Nothing in the pipeline sets it today, so the default always wins.

Glass in a game engine is a solid mesh carrying a translucent material, so whether it hides what is behind it is a decision somebody has to make. The capture has a switch for exactly that. Set both ways, with a verified dielectric pane between the camera and the subjects, on both segmentation backends, the two runs come out identical to the pixel.

The likely reason is that the identity pass draws only the targets, and a glazed wall is not a target, so it cannot occlude there whatever the switch says. If that is right the setting only bites when the labelled object is itself translucent, which is not what its name suggests. It is not fixed. It is here so that the documentation and the behaviour cannot quietly drift apart.

The room is built by a script rather than modelled by hand, so it can be rebuilt from nothing: walls, glazing, staff, poses and all four cameras. The occluders are placed by projection, which is why a panel hides exactly the near half of a figure rather than approximately. The figures on this page are generated from the captures, so a figure that disagrees with the data cannot be produced.

The scene script, both capture jobs, the figure script and a written record of every number are in the repository under docs/showcase/occlusion_lab.