Annotation and labelling
How raw data becomes training data: writing the guideline, measuring agreement, estimating cost, and letting models help.
The idea
A model learns from examples with correct answers. Someone has to assign those answers. That is annotation, and in most projects it is the single largest cost.
The flow
- 01
Write the guideline
With examples, especially for borderline cases. Definitions alone are not enough.
- 02
Pilot run
Have 50 to 100 cases annotated independently by two people.
- 03
Measure agreement
Where agreement is low, revise the guideline, not the people.
- 04
Main run with sampling
Have five to ten percent double-annotated to monitor quality continuously.
A usable guideline
- One page per class: definition, three unambiguous examples, three borderline cases with reasoning.
- A decision order for cases where several classes apply.
- An explicit "cannot be decided" category so nobody has to guess.
- A change log. Every adjustment mid-run affects the cases already collected.
Estimating cost
| Task | Time per unit | 10,000 units |
|---|---|---|
| Text classification, 3 classes | 5 to 15 s | 14 to 42 h |
| Text classification, 20 classes | 20 to 60 s | 56 to 167 h |
| Marking entities in text | 1 to 4 min | 167 to 667 h |
| Object boxes in an image | 20 to 60 s | 56 to 167 h |
| Instance segmentation | 3 to 15 min | 500 to 2,500 h |
That table belongs at the start of every project. It leads to a change of task definition more often than to a budget request.
Active learning
Rather than annotating at random, you pick the cases the current model is least certain about.
Empirically, active learning reaches the same quality with 30 to 60 percent of the annotations. Two caveats:
- The resulting dataset is no longer representative and is unsuitable as a test set. The test set is always drawn at random.
- Pure uncertainty selection favours edge cases and outliers. Blending in 20 to 30 percent random cases keeps the dataset healthy.
Pre-annotating with a model
| Step | Effect |
|---|---|
| Model proposes, human confirms or corrects | Factor 3 to 5 faster |
| Only present cases below a confidence threshold | Factor 5 to 20, with risk |
| Accept without review | Model errors become training data |
The third row is the mistake that quietly ruins projects: the model learns its own errors and confirms them. Sampling at least five percent of all automatically accepted labels is the floor.
The link to the AI Act
For high-risk systems, Art. 10 requires among other things information on data collection procedures, provenance, and assumptions about what the data is meant to represent. The annotation guideline, the measured agreement and the change log are therefore part of the technical documentation. Keeping them properly discharges that obligation as a by-product. See Preparing for audit.
Related courses and sources
Segment Anything
Segmentation without task-specific training, steered by points and boxes. Changes the preparatory work in image analysis considerably.
For image analysis with little data of your own; it changes the preparatory work noticeably.
spaCy
A library for classical language processing. For recognising names, parts of speech and structure it is often faster, cheaper and more checkable than a language model.
When names, parts of speech or structure are needed: often faster, cheaper and more checkable than a language model.