How a language model is trained
From raw text to a usable model: pre-training, supervised fine-tuning and alignment, with the orders of magnitude of each stage.
The three stages
| Stage | What happens | Data volume | Share of cost |
|---|---|---|---|
| Pre-training | Predict the next token | Trillions of tokens | ~99 % |
| Fine-tuning | Learn to follow instructions | 10,000 to 1 million examples | ~1 % |
| Alignment | Learn preferred behaviour | 10,000 to 500,000 comparisons | under 1 % |
What is actually learned in pre-training
The task is always the same: here is a piece of text, predict the next piece. To do that well, a model has to grasp grammar, retain facts, resolve references and recognise patterns of argument. None of it is taught separately.
The pre-training loss
What is remarkable is that every token is simultaneously input and target. A text of a million tokens yields a million training examples with nobody annotating anything. That is why the task is feasible at this scale at all.
What is in the data
- Deduplication is the single most effective quality step. Repeated texts get memorised.
- Quality filtering beats sheer volume, measured on downstream tasks.
- Order matters: high-quality data late in training has a stronger residual effect.
- A language's share of the training data determines later quality in that language almost directly.
Scaling laws
A 7-billion-parameter model would therefore be compute-optimal at 140 billion tokens. In fact such models are trained on 2 to 15 trillion tokens, far beyond the optimum.
The reason is a different objective. Compute-optimal minimises cost to a finished model. Anyone planning to run that model millions of times wants the smallest possible model at a given quality, and accepts a multiple of the training effort for it. Operating cost dominates the overall calculation.
Fine-tuning
Forget that mask and you train the model to generate prompts too. The result is answers that repeat the question: a common and easily missed mistake.
Orders of magnitude for your own project
| Plan | Effort | Realistic for |
|---|---|---|
| Own pre-training, 7 bn | 10⁵ to 10⁶ GPU hours | Very few organisations |
| Full fine-tuning, 7 bn | 100 to 1,000 GPU hours | A larger department |
| LoRA fine-tuning, 7 bn | 5 to 50 GPU hours | Any team |
| Prompting and RAG | 0 GPU hours | The normal case |
The last row is the right answer for the overwhelming majority of business tasks. See Fine-tuning in depth.
Related courses and sources
Common Crawl
The open crawl of the web that a large share of language model training data comes from. It shows concretely what a pre-training corpus actually contains.
For anyone asking where a model's knowledge comes from, and for the question of opt-out reservations.
Language Models are Few-Shot Learners
The paper showing that examples in the prompt can replace fine-tuning. The origin of what is now called prompting.
For understanding why examples in the prompt can replace fine-tuning.
Training Compute-Optimal Large Language Models
The calculation showing that most large models were trained on too little data. Data volume has not been a side issue since.
For anyone comparing model sizes who needs to know why data volume counts.
Training Language Models to Follow Instructions
How a text continuation engine becomes an assistant. The paper behind alignment from human feedback.
For anyone asking how a text continuation engine becomes an assistant.