The transformer architecture
The blueprint behind almost every current model: blocks, position encoding, the parameter calculation, and what has actually changed since 2017.
The idea
A transformer is a stack of identical blocks. Each block does two things: it lets every token look at all the others, and then it processes each token on its own. That repeats twenty to a hundred times.
At the end comes a prediction about the next token, that is a probability for every entry in the vocabulary.
The structure of one block
- 01
Normalise
Bring the values into a usable range.
- 02
Attention
Every token mixes in information from all previous ones.
- 03
Add
The result is added to the input, not substituted for it.
- 04
Feed-forward
A small network processes each token individually, again with normalisation and addition.
The parameter calculation
Worked through for L = 32, h = 4096, f = 11008, V = 128,000:
| Item | Calculation | Parameters |
|---|---|---|
| Attention, 32 blocks | 32 × 4 × 4096² | 2.15 bn |
| Feed-forward, 32 blocks | 32 × 2 × 4096 × 11008 | 2.88 bn |
| Embedding | 128,000 × 4096 | 0.52 bn |
| Total | 5.55 bn |
With a tied output layer it stays there; otherwise another 0.52 billion is added. The calculation also shows why the feed-forward layer is the largest item, and why mixture of experts targets exactly it.
Position encoding
| Method | Principle | Behaviour on longer context |
|---|---|---|
| Learned absolute | One table entry per position | Breaks beyond the training length |
| Sinusoidal | A fixed function of position | Generalises somewhat better |
| RoPE | Rotates query and key by position | Can be extended by scaling |
| ALiBi | Penalises long distances in the score | Generalises best |
RoPE prevailed because it encodes relative distances directly in the dot product and can be stretched afterwards to extend context.
RoPE
That is the decisive property: the model learns relative distances rather than
absolute positions. Extending context then works by stretching the frequencies,
for instance by raising b or scaling the position.
What changed since 2017
| Then | Now | Reason |
|---|---|---|
| Post-norm | Pre-norm | Stability at depth |
| LayerNorm | RMSNorm | Faster, equally good |
| ReLU in the feed-forward | SwiGLU | Measurably better quality |
| Absolute positions | RoPE or ALiBi | Longer context possible |
| Multi-head attention | Grouped queries | KV cache 4 to 8 times smaller |
| Encoder-decoder | Decoder only | Simpler, scales better |
| Dense feed-forward | Partly mixture of experts | More capacity at the same cost |
The basic structure, a stack of identical attention plus feed-forward blocks with residuals, is unchanged. Every change improves stability, efficiency and context length, not the principle.
Grouped queries
Rather than computing separate keys and values per head, several query heads share one key-value head.
With 32 query heads and 8 key-value heads, g = 4 and the cache is a quarter of
the size. The measured quality loss is a few tenths of a point of perplexity; the
gain in possible context length is a factor of four. See
Memory and bandwidth.
Related courses and sources
Attention Is All You Need
The 2017 paper that introduced the transformer. Everything called a language model today rests on these eight pages.
The eight pages everything called a language model today rests on.
BERT
Pre-training on masked text, then fine-tuning per task. The pattern that shaped language processing before the large models.
For understanding what shaped language processing before the large models.
Deep Learning Specialization
Five courses from the basics of neural networks to sequence models. Thorough, with programming exercises, and in places older than current practice.
For anyone who can program and wants to work through the field completely.
Hugging Face NLP course
Tokenisation, transformers, fine-tuning and deployment, with code throughout. Assumes Python, and in exchange you end up working with real models.
For development with language models once it has to go beyond calling an interface.
Understanding Deep Learning
A modern textbook with unusually clear figures that already covers transformers and diffusion models in full. Free as a PDF.
For a modern entry point; it already covers transformers and diffusion in full.