COCO

COCO datasets, with polygons traced from the render.

COCO is the format most segmentation code already speaks, and its polygons are the part that is expensive to produce by hand. Generated from an Unreal scene they are traced from the identity buffer, which means they follow the object outline exactly rather than approximating it in a dozen clicks.

Categories: 6
Annotations in the sample: 1,813
Tasks: detection, instance segmentation
Loader: pycocotools

One real annotation, in full

This is annotation 372 from the exported sample, copied out of the file. It is the car in the top right of the frame this site keeps showing: 223 by 200 pixels, 26,339 of which actually belong to it, outlined by a seventeen-point polygon.

{
  "id": 372,
  "image_id": 4,
  "category_id": 2,
  "bbox": [1697, 564, 223, 200],
  "area": 26339,
  "iscrowd": 0,
  "segmentation": [[
    1715, 580, 1697, 649, 1698, 652, 1713, 661, 1710, 662, 1709, 666,
    1713, 668, 1712, 671, 1724, 668, 1735, 675, 1731, 683, 1732, 685,
    1752, 690, 1823, 717, 1919, 763, 1919, 639, 1778, 564
  ]]
}

Two details are worth noticing. Area is not width times height. The box is 44,600 pixels; the area is 26,339, because that is how many pixels the car actually occupies. Most hand-annotated COCO sets fill area from the box or from a rough polygon, and models that use it for size-aware losses inherit the error.

The polygon runs off the right edge at x = 1919. The car is truncated by the frame, and the outline says so rather than pretending the object ends where the image does.

What the file contains

info and licenses, filled from the run rather than left as COCO’s placeholder strings.
images: file name, width and height per frame.
categories: 6 entries, ids starting at 1 as COCO requires, which is not the same numbering YOLO uses.
annotations: box, area, iscrowd, and a polygon per instance.

import json
from pycocotools.coco import COCO

coco = COCO("annotations/instances.json")
print(len(coco.imgs), "images", len(coco.anns), "annotations")

# every annotation in the first image, with its mask
for ann in coco.loadAnns(coco.getAnnIds(imgIds=[1])):
    mask = coco.annToMask(ann)          # HxW binary array
    name = coco.loadCats([ann["category_id"]])[0]["name"]
    print(name, ann["bbox"], mask.sum(), "pixels")

Category ids, and the off-by-one that bites everyone

COCO numbers categories from 1. YOLO numbers classes from 0. The same export writes both, and the two numberings are not interchangeable.

Class	COCO id	YOLO id	Annotations
person	1	0	996
car	2	1	254
tank	3	2	80
container	4	3	361
crate	5	4	1,216
barrel	6	5	1,471

If you train from the COCO file and evaluate against the YOLO labels without noticing, every prediction is off by one class and the result looks like a model that learned nothing. Both files ship in the same pack precisely so you can check this before it costs you a day.

Honest limits of this export

Three things worth knowing before you build a pipeline on it.

Polygons, not RLE

Segmentation is written as polygons. For most training code that is what you want. If you need run-length encoding, convert with pycocotools rather than expecting it in the file.

A few instances have no polygon

Three of the 3,356 annotations in the reference training split carry a box but an empty polygon, because the visible region was too small to trace. They are kept rather than quietly dropped, and are worth filtering if your loader is strict about it.

iscrowd is always zero

Every instance is a distinct object with its own identity, so nothing is a crowd region. That is a property of engine-derived labels, not an omission.