Skip to content
NAMEFRAMEApply for Pilot
Apply for Pilot

How to build a multi-camera capture rig in Unreal Engine

Rendering from several cameras is easy. Rendering from several cameras at the same instant, and being able to prove afterwards that you did, is the part that decides whether the dataset is usable for stereo, for multi-view, or for anything that compares one view against another.

The hard part
one instant, not one loop
Per camera
its own pose and intrinsics
Per frame
the same scene state
Fails as
plausible parallax

The loop most rigs start as

The obvious rig is a loop: for each camera, position it, render, save. That produces the right number of images with the right names, and it is wrong in a way nothing in the output announces.

Between the first render and the last, the world moved. Animations advanced, the physics stepped, foliage swayed, a character’s foot came down. The images are not views of one moment, they are views of consecutive moments from different places, and the difference between the two is indistinguishable from parallax.

A stereo network trained on that learns a disparity that partly encodes time. It performs well on more of the same and fails on a real rig, which genuinely is synchronised. Nothing in the pipeline reports a fault, so it survives to training.

One scene state, then every camera

One frame means one scene state and many cameras, in that order. Advance the world once. Freeze it. Then render every camera against the frozen state, write every view, and only then let time move again.

That is a different program from the loop above, and the difference is where the world tick sits. If any part of the engine is still stepping while the cameras take turns, the rig is a sequence of single camera captures wearing a multi camera name.

One camera's view from a capture. A multi-camera rig writes several of these per frame, all of the same instant, each with its own pose.
Rendered frameBeachDemo, frame 000245Unreal Engine 5.8, 1280×960One view of one frame; a rig writes several, all of this instant

Per-camera metadata

The second failure is subtler and shows up months later, when somebody tries to use the dataset for geometry. Two cameras in a rig share a scene and share a moment, and share nothing else. Each has its own position, its own orientation, and, if the lenses differ, its own intrinsics.

A capture that writes one camera record per frame, because the single camera case only ever needed one, silently attributes every view to the first camera’s pose. Reprojection then fails for every view except one, and the failure looks like a calibration problem rather than a bookkeeping problem.

frames/
  plugin_000004/
    cam_left/
      rgb.png
      seg.png
      depth.npy
      frame.json        # this camera's pose and intrinsics
    cam_right/
      rgb.png
      seg.png
      depth.npy
      frame.json        # a different pose, same instant
    scene.json          # what was true of the world for both of them

The shape matters more than the exact names. Anything shared by the views, what the objects were and where, belongs once at the frame level. Anything that differs per view belongs beside that view. A layout that cannot express the difference will eventually be asked to, and will answer with the wrong camera.

Naming the cameras

Give every camera a stable id in the level and use it in the path. Not an index: indices renumber when somebody deletes a camera, and a dataset where cam_1 means the left camera in the first half of the run and the right camera in the second half is worse than one with no ids at all, because it looks fine.

As long as the same physical camera keeps the same name from the first frame to the last, the pairing survives every later change.

RigWhat it is forThe thing that ruins it
Stereo pairDisparity and depth supervisionAny world motion between the two renders, which reads as extra disparity
Surround, four to six camerasCoverage without moving the platformSharing one pose across views, so only the first reprojects correctly
Chase camera plus a mounted sensorShowing a sensor's output against a third-person viewAssuming the sensor sees what the camera sees, when parallax means it does not

The third row is worth a note because it is the one people underestimate. A sensor mounted on a vehicle and a camera watching that vehicle from fifteen metres away do not share an occlusion set. The sensor sees round things the camera cannot, and those returns land, in the camera’s image, on whatever is in the way. The rig is working correctly. A check that compares the two views has to account for it, or it reports a fault nobody has.

Two checks on a rig

Put something moving in the scene, fast, and capture. Then look at that object in every view of a single frame. If the rig is synchronised, it is in one place, seen from several angles. If it is not, it is in several places, and the faster it moves the more obvious the smear across the views.

Do this with an object crossing the frame quickly rather than with a walking character. A slow subject hides the fault, and the whole point of the check is to make the fault impossible to miss.

The second check is arithmetic. Take an object the capture recorded a 3D position for, project it into every view using that view’s own intrinsics and pose, and confirm each lands inside that object’s box in that view. One view landing correctly and the rest drifting is the shared-pose bug above.