Off-frame landmarks: annotating points outside the image

A camera standing on a football pitch sees perhaps a fifth of it. The far corner, the penalty mark at the other end, the halfway line off the left edge: all still known, because the ground and the camera both are. Those are facts rather than observations, so they can be written down even though no pixel in the frame contains them.

Grounds: A football pitch and a tennis court, to regulation
Projections: 147 across six frames
Outside their frame: 53, each with a coordinate
Checks: Two, and both refuse rather than warn

The one thing nobody can do by hand

Ask an annotator to mark the far corner of a pitch in a frame that does not contain it and there is nothing for them to click. The information is not in the picture. It is in the world, and a renderer knows the world and the camera, so the projection is arithmetic rather than observation.

Anyone recovering a homography from partial views needs exactly this and cannot get it from real footage. It is the clearest case on this site of a thing synthetic data provides that no amount of labelling budget buys.

A football pitch seen from high on the halfway line, with green dots on landmarks inside the frame and pink dots outside it on black, the touchlines running out to the far corners — The lit rectangle is the photograph. Everything on black is outside itL_FootballPitch, marked to the IFAB Laws at 105 by 68 metres

Camera	In the frame	Outside it	Behind it
High on the halfway line	23	12	0
Eye level at the touchline	15	14	6
Close on the east penalty area	12	8	15
Total	50	34	21

From the touchline, most of the pitch is elsewhere

Standing where a touchline camera stands, at eye level, fifteen landmarks are in the frame, fourteen are outside it and six are behind the camera. The pitch has not moved.

A low camera on a football touchline, with most of the pitch landmarks drawn on black to the left and right of the frame — Eye level at the touchlineNamed points are the ones furthest out

A camera close to a penalty area, with the penalty area corners drawn on black below and left of the frame, joined to it by the painted lines — Close on the east penalty areaFifteen of the thirty-five are behind this camera and are not drawn at all

A small ground is not a fully observed one

A pitch makes the point by being enormous. A tennis court makes it more sharply, because it is small enough that a viewer expects to see all of it. From behind the baseline the whole court appears to be in shot, and four of its fourteen landmarks are below the bottom edge.

Across the three court cameras it comes out at exactly half: nineteen landmarks in their frames and nineteen outside them.

A tennis court from behind the baseline, apparently entirely in frame, with four pink landmarks drawn on black below the picture where the near corners fall — Behind the baseline. The court looks complete; four corners are below the frameL_TennisCourt, marked to the ITF regulation

Camera	In the frame	Outside it	Behind it
Behind the baseline	10	4	0
Beside the net at eye level	5	7	2
Inside the court, on the service boxes	4	8	2
Total	19	19	4

A camera beside a tennis net, with baseline landmarks drawn far to the left and right on black — Beside the net at eye levelInside the court, on the service boxes

A camera inside a tennis court looking at the service boxes, with most of the court's landmarks outside the frame

Three states, not two

Whether a point is inside the frame and whether anything is hiding it are different questions, and one flag forces a choice between them. So every landmark carries both, separately, and a point at or behind the camera plane carries a third: outside the frustum, occlusion is not false, it is undefined, and a refusal to answer is not a coordinate.

Twenty-one of the pitch’s hundred and five projections and four of the court’s forty-two are behind their camera. They are counted and they are not drawn.

How an invisible label is checked

An off-frame label cannot be eyeballed. There is no pixel to look at, so a wrong one looks exactly like a right one, which means the usual defence against a bad label does not exist here.

Every frame is therefore projected twice, once through the vector form the labeller uses and once through an independent matrix form, and the worst disagreement is recorded. Where the points are coplanar, which on a marked ground they are, a homography is fitted from metres to pixels and its residual is recorded too.

Check	Worst, football	Worst, tennis	Refuses above
The two projections disagree	3.0e-09 px	0.0 px	0.05 px
Homography residual	4.2e-12 px	1.2e-11 px	1.0e-06 px

Both limits refuse to write rather than warn. That is not caution for its own sake: an axis-convention flip in this repository’s own sweep once produced zero visible corners from every elevated camera and looked entirely plausible. The reprojection check would have caught it on the first frame.

The limit: no export can carry these

COCO and YOLO both define a point outside the image as not labelled and write zeros. An off-frame landmark cannot travel in a pose file at all, so this is a limit of the formats rather than of the capture, and it is the reason landmarks are their own concept with a file of their own rather than extra keypoint rows.

What that means in practice: a consumer who wants this geometry reads landmarks.json, which carries the world position, the image coordinate, the three states and the depth for every point in every frame.

Running it yourself

Both grounds are built by script and marked to regulation. For each, the painted lines in the level and the landmarks projected through the camera come from one module. That is not tidiness: a corpus whose geometry and whose recorded points are generated separately is how a dataset ships with its lines most of a metre out while every self-consistency check it has passes.

The scene scripts, the capture jobs, the point files, the figure script and every number behind them are in the repository under docs/showcase/off_frame_landmarks.