One Unreal Blueprint, several labels: child bind
A composite actor is one thing to whoever authored it and several things to a dataset. A person in a wheelchair is moved, swapped and spawned as a single Blueprint, and a detector needs the person without the chair. Child bind names the branch that carries each label, so the chair still renders and still occludes without landing in anybody’s box.
- Authored
- One Blueprint, spawned once
- Labelled
- One branch, or several, by name
- The prop
- Still renders, still occludes, still writes depth
- Found by a reader
- One defect, root-caused and fixed
One actor, as many labels as the scene needs
The same three Blueprints are captured three times, and the captures differ in nothing except what each target asks for. Without bindings, each Blueprint is one label and its furniture is inside it. Binding the person branches gives a label per person. Binding the furniture as well gives it a class of its own, which is what a buyer wants when the bench is also something they train on.
| What the job asks for | Blueprints in the scene | Instances written |
|---|---|---|
| Nothing bound: one label per Blueprint | 3 | 5 |
| The person branches bound | 3 | 8 |
| People and furniture, each its own class | 3 | 13 |
What leaves the box
The bench is the clear case: one box over the furniture and both sitters becomes two boxes, one per person, and neither contains the bench.
The wheelchair is the one worth reading carefully, and it is turned to three quarters for that reason. Square on, the chair hides behind the person sitting in it and the numbers have to be taken on trust. At this angle the near wheel and the push handles stand outside the person’s silhouette. The box narrows from 248 pixels to 168 while its height does not move at all, because what leaves it is the chair beside the person rather than any part of the person.
| Frame | Without child bind | With child bind |
|---|---|---|
| A person in a wheelchair | 1 box, 248 by 336 | 1 box, 168 by 336 |
| Two people on a bench | 1 box, 417 by 373 | 2 boxes, 150 by 317 and 194 by 364 |
It is the mask, not just the box
Boxes are a summary and a summary can hide things. The identity image is what the box is derived from, and it says the same thing more plainly: with only the person bound, the chair carries no identity at all; with both bound, each is its own mask.
The first limit: a second label costs the occlusion measurement
An actor that yields one label is measured. An actor that yields two is not, and the split follows the number of bindings rather than anything about the scene.
| Blueprint | Labels per actor | Instances measured | Not measured |
|---|---|---|---|
| BP_ManWheelchair01 | 1 | 2 | 0 |
| BP_ManBenchPair01 | 2 | 0 | 4 |
| BP_WomanBenchPair02 | 2 | 0 | 2 |
Instances from a one-binding actor carry their mask pixel count, their occlusion ratio and their visible ratio. Instances from a two-binding actor carry nothing in all three, while keeping their box, their mask, their fragment count and their thirteen keypoints.
Two consequences, said here rather than discovered later. Nothing downstream can ask how much of those people reached the film. And the visibility threshold cannot fire on them, because it compares against a ratio that is not there: a bench person hidden to two per cent is kept where a wheelchair person hidden to two per cent is refused.
A defect this page found, and the fix
The figures above are the second version. The first showed a whole-actor box that did not contain the whole actor, and a reader said so before any check did.
Measured on the wheelchair frame at the time: 21,957 identity pixels for the person alone, 42,542 when the person and the chair were asked for as two labels, and 36,414 for the whole actor, which should have been the same as the two together. The region it dropped ended on a straight vertical edge through the middle of the near wheel, and a straight line across a round wheel is not geometry. On the bench it was 11,849 pixels, and its box was 323 wide where the bench alone is 417.
The cause. A target that asks for keypoints takes its bounds from the skeleton’s evaluated bones rather than from its components, which exists for a good reason: a skeletal component’s own bounds can still describe the reference pose rather than the current one. But it returned the bone box alone, and the identity capture accepts pixels inside the projected bounds, so any non-skeletal mesh the same actor owns was drawn and then clipped exactly where the person’s bones ended.
The proof. The same capture with keypoints turned off produced 42,542 pixels and lost nothing, against 36,414 with them on. One field, the same scene, 6,128 pixels.
After the fix. Zero pixels missing in both frames. The whole-actor box on the wheelchair went from 192 by 336 to 248 by 336, and on the bench from 323 by 364 to 417 by 373, which is the bench.
Running it yourself
The square is built by a script from street modules, so it can be rebuilt from nothing, and the component names the bindings point at were read out of the Blueprints rather than guessed. The figures on this page are generated from the captures, so a figure that disagrees with the data cannot be produced.
The scene script, the three capture jobs, the figure script and every number behind them are in the repository under docs/showcase/child_bind.