Projecting 3D into 2D
Every actor position is recorded in the same metres as the camera, so you can reproject them yourself and check the labels. That is precisely what the verification pass does on every capture.
Camera metadata
Where the camera was, written down every time.
A frame without its camera is a picture. A frame with its camera is a measurement you can project into, reproject out of, and compare against another frame. Every capture records the pose, the field of view and the zone the placement came from, in a coordinate frame it names explicitly.
- Coordinate frame
- ned-z-down
- Field of view
- 64° horizontal
- Intrinsics
- exported
- Camera zones declared
- 4
One frame’s camera, exactly as recorded
Abridged only by rounding. This is frame plugin_000004 of the reference capture, the same frame every other page on this site uses.
"camera": {
"pos": [-137.746, -23.778, -31.576],
"quat": [-0.232439, -0.047249, -0.951994, 0.193514],
"euler_deg": { "pitch": -27.442, "yaw": -157.02, "roll": 0 },
"look_at": [-191.264, -46.473, -1.39],
"fov_deg": 64,
"zone": "sunlit_far",
"policy": "camera_zones_look_at_target"
}Both rotation representations are written, which is deliberate. The quaternion is what you want for maths; the Euler angles are what you want when you are trying to work out why a frame looks wrong at two in the morning. Deriving one from the other is easy and getting the convention wrong is easier, so the capture does it once and records both.
The conventions, stated rather than implied
- Positions are in metres. Unreal works in centimetres; the conversion happens once, at capture.
- The frame is ned-z-down. Naming it in the capture manifest means nothing downstream has to guess which axis points where.
- Quaternions are xyzw ordered, not wxyz. Both are common; only one is right for a given library.
look_atis the point the camera was aimed at, in the same metres, which makes the framing intent recoverable rather than inferred from the rotation.
Where the camera was allowed to be
Camera placement is authored as volumes in the level and sampled inside them, so a viewpoint distribution is a thing you designed rather than a thing that happened.
| Property | Value |
|---|---|
| Placement policy | camera_zones_look_at_target |
| Mode | orbit |
| Aiming | look at target |
| Zones declared in the level | 4 |
| Zones actually used | sunlit_far, sunlit_high |
| Pitch below horizontal | 13.41° to 43.44° |
| Distance to subject | 33.19–187.81 m |
Two of the 4 declared zones produced every frame in this run. That is not a bug, it is what viewpoint validation does: placements that could not see enough of the scene were rejected and resampled, and the two zones with a clear line to the subject won. It is also a coverage warning worth reading before training, and exactly the kind of thing that is invisible without the metadata.
What this makes possible
Multi-view geometry
Two frames of the same scene from known poses is a stereo pair with a known baseline. Nothing about that has to be estimated.
Coverage analysis
Pitch, yaw and distance per frame turn into the distributions on the quality page, which is how you find out your dataset only ever looked from one angle.
Reproducing a shot
A frame you liked can be captured again exactly, in different weather or with a different population, because the pose is a number rather than a memory.