How to label a point cloud for 3D semantic segmentation

Semantic segmentation in 3D wants a class on every point, not a box around a cluster of them. Producing that by hand means a person rotating a viewport and deciding, point by point, where a pedestrian ends and the bollard behind them begins. A rendered scene knows which surface each ray struck, so the question never arises.

What each point carries: A position and a class
Optionally also: An instance id, for panoptic work
Hand labelling: Guesswork exactly at object edges
The built-in problem: Class counts follow geometry

Where hand labelling actually breaks

The cost of labelling a real cloud is not the middle of objects. A car is obviously a car and a wall is obviously a wall. The cost is the boundary: the points where a person’s shoulder meets the railing behind them, the returns that clipped an edge, the sparse tail of a distant object mixed with the ground.

Those boundary points are also where a model learns most, and they are the ones a human is least consistent about. Two annotators labelling the same cloud disagree at exactly the places that matter, and no amount of review makes that go away, because there is genuinely no way to tell from the points alone.

A ray in a rendered scene struck one surface belonging to one object. The class is not inferred from the geometry; it is carried by the thing that was hit. The boundary stops being a judgement.

A top-down point cloud of one LiDAR revolution over a farmyard, dense near the sensor and sparse at range — One revolution of a scan over a farmyard, every point placed by where its ray landedColour here encodes firing time; the class travels with each point in the file

Semantic, instance, panoptic

Three words for three different files, and the difference is the same as it is in 2D.

Semantic gives every point a class. Two cars parked together are one region of class “car”. This is what most published 3D benchmarks ask for and the easiest to consume.

Instance gives every point an object id, so the two cars separate. Needed for counting, for tracking, and for anything that reasons about individual objects.

Panoptic is both, per point, and is what you should export if you are unsure: it collapses into either of the others, and neither of the others recovers it. Writing both arrays costs a few bytes a point and saves a re-capture.

The imbalance that is built into the geometry

Every point cloud is dominated by large flat surfaces. Ground and buildings will hold the overwhelming majority of points in almost any outdoor scan, not because they are important but because they are big and close and a laser sweeping a scene hits them constantly.

Train naively on that and the loss is minimised by calling everything ground. Three things help and all three are decisions to make before capturing rather than after: weight the loss by inverse class frequency, sample the scene so rare classes appear near the sensor rather than only at range, and count points per class before training rather than discovering the ratio from a confusion matrix.

The counting is the cheap one and the one people skip. A histogram of points per class takes a minute and tells you whether the corpus can teach the thing you want it to teach.

Three things to check in the file

One label per point, in order

The label array must be exactly as long as the point array and in the same order. An off-by-one shifts every class by one point and produces a cloud that looks almost right when viewed.

Classes kept out of intensity

Intensity means something. A class id smuggled into it will be consumed as a return strength by the first library that reads the file properly.

The same instant as the frame

A scan taken at a different moment than the picture beside it describes a different world. If you are fusing them, the capture should state they came from one instant rather than leave you to hope.

The beam pattern still matters here

A cloud whose rings are evenly spaced has a density profile no real sensor produces: real heads crowd their beams near the horizon and leave gaps above and below. For semantic segmentation this changes which classes are dense and which are sparse, which changes the imbalance above, which changes what the model learns to ignore.

If the intent is to train something that will run on a particular sensor, scan with that sensor’s published beam table. It is in the calibration file and it costs nothing to use.

Two point clouds of the same farmyard scene: one sensor's scan lines bunch tightly around the horizon while an even ladder spreads them flat across the whole field of view — The same scene scanned two ways: a real beam table, and thirty-two evenly spaced ringsWhich points exist at all differs, so which classes are dense differs with it

Doing it with NameFrame

Points come out labelled because the ray knows what it hit, and the scan is written from the same instant as the picture and the depth. Export is PLY for looking at and KITTI binary for feeding to published detection code.

nameframe capture-unreal job.json     # lidar is a modality of the run, not a separate pass
nameframe label D:/runs/farm D:/runs/farm/labels

What the simulation covers and what it does not, including the parts it does not model at all, is in two tables on the LiDAR page. The beam tables and a measured comparison against an even ladder are on the sensor page.