Depth estimation

Metric depth, dense, and suspiciously perfect.

Depth is the modality where synthetic data has the largest advantage and the sharpest catch. The advantage is that every pixel has a true distance in metres, with no dropouts. The catch is that no real sensor behaves like that, and a model trained only on flawless depth learns to expect it.

Ground truth: float32 metres per pixel
Resolution: 1920×1080
Median depth: 63.8 m
Sample pack: 3 frames, 14 MB

A colour-mapped depth visualisation of an airbase frame — Scene depth, in metresMap_Airbase_Demo, frame 000004Unreal Engine 5.8, 1920×1080Display mapping only; the file is float32 metres

What arrives with each frame

A float32 array of 1080 by 1920 values, 2,073,600 in total, each the distance in metres from the camera to what that pixel shows.
The camera pose and field of view that produced it, so the depth map can be unprojected into a point cloud without estimating anything.
The identity buffer for the same frame, so depth can be attributed per object rather than per pixel.

100% of the values in the reference frame are finite. Depth spans 24.8 metres at the first percentile to 194.7 at the ninety-ninth, with a median of 63.8.

import numpy as np

depth = np.load("depth/plugin_000004.npy")     # metres, float32

# unproject to a point cloud using the recorded field of view
h, w = depth.shape
fx = fy = (w / 2) / np.tan(np.radians(64) / 2)
xs, ys = np.meshgrid(np.arange(w) - w / 2, np.arange(h) - h / 2)
points = np.stack([xs * depth / fx, ys * depth / fy, depth], axis=-1)

Training on perfect depth

The whole question for this task is how to keep the advantage without inheriting the flaw. Three practical answers.

Degrade it deliberately

If you deploy against stereo or LiDAR, simulate that sensor’s failure modes: dropouts on dark and reflective surfaces, range ceilings, quantisation, noise that grows with distance. Perfect depth is an input you can always make worse; you cannot make bad depth better.

Train relative, not absolute

Scale-invariant and ordinal losses transfer better across the synthetic-to-real boundary than absolute metric regression, because they depend on structure rather than on the exact calibration of your rig.

Check the range you actually need

This capture is aerial: the mass sits between 24.8 and 194.7 metres. An indoor robot needs 0.3 to 8, and no amount of frames from this capture will teach it that band.

The practical constraint nobody mentions

Depth is enormous, and it is why the depth sample pack is three frames rather than twelve.

2,073,600 float32 values is 8 MB per frame, uncompressed. A ten-thousand frame run carries 80 GB of depth alone, several times the size of everything else in the capture put together. That is why depth writing is a switch rather than an always-on default, why long soak runs turn it off, and why the every-modality sample is deliberately small.

If you need depth at volume, plan the storage before the run rather than discovering it at frame four thousand.