Building datasets
The part that is 80 percent of the work: defining scope, sampling, splitting, versioning and settling the legal basis.
The idea
A model can only be as good as the examples it learns from. Assembling those examples is not preparatory work; it is the work.
The four questions at the start
- 01
What exactly is to be predicted?
And how will you later tell that the answer was right?
- 02
Which cases belong in scope?
The boundary matters more than the volume. A clearly bounded dataset of 2,000 cases beats an unclear one of 50,000.
- 03
Where does the data come from?
Provenance, period, collection method. Without those, nothing can later be judged.
- 04
Are we allowed to use it?
Legal basis, purpose limitation, third-party rights. Before the first training run, not after.
The learning curve as a decision aid
import numpy as np
from sklearn.model_selection import learning_curve
fractions = np.array([0.1, 0.25, 0.5, 0.75, 1.0])
# If the validation curve is still rising markedly at the right edge, more
# data pays. If it is flat, only a different model or better features help.That curve answers "more data or a better model?" empirically rather than by guesswork, and regularly saves a quarter of annotation work.
The data card
A short document per dataset answering:
- Purpose: what was it assembled for, and explicitly not for?
- Scope: count, period, sources, language, regions.
- Collection: how selected? Randomly, exhaustively, filtered? By what criteria?
- Labels: who assigned them, under what guideline, with what agreement?
- Known gaps: which cases are missing or under-represented?
- Law: legal basis, personal data share, retention, erasure.
- Version: identifier, checksum, date, changes from the previous version.
Sampling methods
| Method | When | Trap |
|---|---|---|
| Simple random sample | Homogeneous population | Rare classes go missing |
| Stratified | Known subgroups | Strata must be fixed in advance |
| Time-blocked | Time series, processes | Never shuffle |
| By group | Several cases per person | Groups must not straddle splits |
| Targeted at hard cases | Sharpening | Distorts the distribution, training only |
The last row is important and often got wrong: deliberately collecting hard cases improves training but makes the test set unusable if the same selection is applied there.
Freezing the split
The advantage over a seeded random split: when new cases are added, no existing case migrates to a different part. Without that property, comparing two model versions over time is worthless.
The legal frame
- Purpose limitation. Data collected to perform a contract is not thereby usable for model training. A change of purpose must be assessed and justified.
- Art. 10 AI Act. For high-risk systems, requirements apply to training, validation and test data: relevance, representativeness, freedom from error as far as possible, examination for bias.
- Erasure. An erasure request affects the dataset. Whether the trained model is affected is unsettled, which is why personal data belongs removed before training.
- Third-party rights. Third-party texts and images are subject to copyright; the text and data mining exception applies only with lawful access and no effective reservation of use.
See GDPR and AI and Copyright and AI.
Related courses and sources
Common Crawl
The open crawl of the web that a large share of language model training data comes from. It shows concretely what a pre-training corpus actually contains.
For anyone asking where a model's knowledge comes from, and for the question of opt-out reservations.
Datasheets for Datasets
Documenting the provenance, composition and limitations of a dataset. The template today's documentation duties come from.
For anyone documenting datasets; the template behind today's duties.
Hugging Face datasets
Open datasets with description, licence and preview. Useful for evaluation sets, risky as training data without checking provenance.
Good for evaluation sets; as training data only with a provenance and licence check.
ImageNet
The dataset image processing measured itself against for a decade. Historically important and well documented in its biases.
For anyone reading benchmarks: almost every image recognition figure refers back to it.
Kaggle datasets
Real, untidy data to practise on. That is exactly what makes them valuable, because tidied examples hide the actual work.
For practising on untidy data, because tidied examples hide the actual work.
Training Compute-Optimal Large Language Models
The calculation showing that most large models were trained on too little data. Data volume has not been a side issue since.
For anyone comparing model sizes who needs to know why data volume counts.