Raw ground truth

The format everything else is derived from.

YOLO and COCO are exports. What the engine writes is a dump: the frames, the buffers, and a record of the world at the moment each frame was taken. If you are building something the exporters do not cover, this is the thing to read, and it is included in the sample packs for exactly that reason.

Per frame: 5 files
Coordinate frame: ned-z-down
Units: metres
Reference capture: 40 frames

dump/
├── capture.json                    the run contract: classes, camera, seed, thresholds
├── spawn_manifests.jsonl           one line per frame manifest, for reconciliation
└── frames/
    └── plugin_000004/
        ├── rgb.png                 the rendered frame
        ├── seg.png                 per-instance identity buffer, one flat colour per object
        ├── depth.npy               float32 metres, 1080 × 1920
        ├── frame.json              camera, every actor, environment, validation, timings
        └── spawn_manifest.json     what was asked for, placed and refused, hashed

Why the dump is kept

Derivation happens outside the engine. That single decision is what makes a capture re-exportable: a run from six months ago can become a dataset in a format that did not exist when it was captured, without opening the editor or re-rendering a frame.

It also makes the labels auditable. Every box in an export can be traced back through an instance colour to an actor name, to a spawn decision, to a seed. Without the dump, an exported label is a claim you have to take on faith.

What each file carries

File	Contents	Notes
capture.json	Classes, camera policy, spawn contract, seed, render and QA thresholds	Hashed as the configuration snapshot
rgb.png	The rendered frame	Lossless, exactly as rendered
seg.png	One flat colour per instance	Matched with 0 tolerance
depth.npy	float32 metres per pixel	Unnormalised; 8 MB per 1080p frame
frame.json	Camera pose and FOV, every actor's position and label, environment state	Positions in metres
spawn_manifest.json	Requested, planned, accepted and refused placements with reasons	Hashed per frame

A frame record, abridged

The interesting part is the actors array: every object in the scene, whether or not it ended up visible, with the position that a label can be checked against.

{
  "id": "plugin_000004",
  "camera": { "pos": [-137.746, -23.778, -31.576], "fov_deg": 64,
              "zone": "sunlit_far", "policy": "camera_zones_look_at_target" },
  "viewpoint_validation": { "accepted": true, "attempts": 8,
                            "visible_targets": 73, "depth_valid_percent": 100 },
  "actors": [
    { "name": "SM_Crate_Stack_05_70", "label": "crate", "label_id": 4,
      "pos": [-109.49, -85.87, -1.39],
      "ground": { "hit": true, "clearance_m": 0.0000031, "slope_deg": 0.0000095 } }
  ],
  "env": { "weather": "scene", "weather_settle_seconds": 1.8 },
  "files": { "rgb": "rgb.png", "seg": "seg.png", "depth": "depth.npy" }
}

viewpoint_validation is worth noticing: this frame was the eighth camera placement tried, and the seven before it were rejected for not seeing enough of the scene. That record is how a coverage question gets an answer rather than a guess.

ground per actor is the other one. Clearance and slope are traced and stored, which is what lets a check afterwards say how many objects are resting on something and how many are floating, and name the surface each one is resting on.

Reading it yourself

Nothing here needs the NameFrame package. It is PNGs, NumPy arrays and JSON.

import json, cv2, numpy as np

frame = json.load(open("metadata/plugin_000004.frame.json", encoding="utf-8-sig"))
seg   = cv2.imread("segmentation/plugin_000004.png")[:, :, ::-1]
depth = np.load("depth/plugin_000004.npy")

cam = np.array(frame["camera"]["pos"])
for actor in frame["actors"][:5]:
    d = np.linalg.norm(np.array(actor["pos"]) - cam)
    print(f"{actor['label']:10} {d:6.1f} m")

The BOM

JSON is written UTF-8 with a byte order mark on Windows. Open with encoding="utf-8-sig" or the first parse fails in a way that reads like a corrupt file.

Channel order

OpenCV reads BGR. The identity colours in capture.json are RGB. Reverse one or the other, consistently.

Size

The reference capture’s dump is roughly 500 MB for 40 frames, most of it depth. The every-modality sample pack is 3 frames at 14 MB for that reason.

Across the reference run the manifests account for 3,360 placement candidates, and the reconciliation check confirms the per-frame files and the run index agree.