How to create a pose estimation dataset in Unreal Engine

A keypoint dataset is a list of joint positions per person, each carrying whether the camera could actually see it. The positions are the easy part. The visibility flag is where hand annotation becomes guesswork and where a rendered scene has an answer, and the flip map is where a whole training run quietly comes to nothing.

What a joint carries: A position and a visibility state
The hard state: Known, but hidden
The silent failure: A missing flip map
Formats: COCO keypoints, YOLO pose

Three states, and only one of them is easy

COCO gives every joint a visibility value, and the three values are not degrees of confidence. They are different claims.

State	Value	What it claims
Not labelled	v = 0	the joint is not in the annotation at all
Labelled but not visible	v = 1	the position is known, something is in front of it
Labelled and visible	v = 2	the camera can see it

The middle one is the whole problem. Asking a human to mark where an elbow is when a table is in front of it is asking them to guess, and different annotators guess differently. Most real datasets are inconsistent here, and a model trained on them learns that hidden joints are unpredictable, which they are, in the labels.

A rendered scene does not have to guess. The skeleton has a position whether or not anything is in front of it, and whether something is in front of it is a question the renderer answered in order to draw the frame at all. The middle state stops being expensive.

Two people with skeleton overlays, one behind a pillar and a diagonal beam and one cut across the waist by a bar, with hidden joints drawn in amber and visible joints in green — Joints the camera can see, and joints it cannot, on the same two peopleThe hidden ones still carry a position, because the position is known

The joint that is outside the picture

A person walking out of frame has a shoulder in the image and a wrist beyond its edge. The wrist has a real position; the picture simply does not reach it.

Both formats define this away. A keypoint outside the image is written as not labelled, with its coordinates zeroed, and the information is gone. That is a limit of the formats rather than of the capture, and it is worth knowing before you design an experiment around partially visible people: the answer exists, and the file you export cannot carry it.

What you can do is decide deliberately rather than by default. Zeroing is the correct, conservative choice and what most training code expects. Clamping the joint to the frame edge is a lie that looks like data. Keeping the true out-of-frame coordinate is useful for analysis and will confuse a standard loader. Pick one, and write down which.

A laboratory frame outlined on a black canvas, with skeleton joints of the nearest figure drawn beyond the edge of the image where they actually project — The lit rectangle is the photograph. The joints beyond it are where they really projectEvery format in common use writes these as not labelled

The flip map, and the day it costs you

This is the one to read even if you skip the rest.

Nearly every training pipeline mirrors images as augmentation. Mirroring a photograph of a person turns their left side into their right, so the labels have to swap too: the left wrist in the original is the right wrist in the mirrored copy. Doing that requires a table saying which keypoint index becomes which.

Without that table, nothing fails. No exception, no warning, no malformed file. The left wrist simply keeps its index on the mirrored copy, so the model is shown one limb labelled two different ways, in roughly equal numbers, for the whole run. It learns that limb is unpredictable. Loss falls, the model trains, and the wrists never converge.

The table is mechanical once the joint order is fixed: every left index maps to its right counterpart, every right to its left, and anything on the midline maps to itself. Ship it with the dataset. A pose dataset without a flip map is a dataset with a trap in it.

Which skeleton, and why it matters more than the count

Seventeen-joint COCO, twenty-five-joint OpenPose and the various hand and face extensions are different vocabularies, not different levels of detail. A model trained on one cannot read the other, and a dataset that mixes them is unusable without a remapping step somebody has to write and verify.

So the decision to make first is which downstream code you intend to feed, and then to keep the joint order fixed and published alongside the data. An order that changes between two exports of the same dataset is the same failure as the missing flip map, arriving from a different direction.

Checking it before you train

Draw the skeleton back on

Ten frames, joints connected by limb. A swapped left and right is instantly visible as crossed legs and invisible in any metric.

Mirror one image by hand

Flip a frame and its labels with your flip map and draw both. If the wrists end up on the same side of the body in both, the map is wrong.

Count the visibility states

A dataset where almost nothing is state one is a dataset that is guessing about occlusion or has none. Both are worth knowing before training on it.

Doing it with NameFrame

Keypoints are asked for per class in the capture job, and they come out of the same run as the picture, the masks and the boxes, from the same instant. Each joint carries its visibility, taken from what the renderer drew rather than estimated, and the flip map is written into the dataset beside the joint order.

nameframe label   D:/runs/street  D:/runs/street/labels
nameframe dataset D:/runs/street  D:/runs/street/dataset \
    --task pose --format yolo,coco

What happens to a joint that is occluded, and what happens to one outside the frame, are both measured rather than asserted on the occlusion page, which publishes the results including the ones that are limits.