Optimisation
What a minimum is, why high dimensions have few local traps, and how learning rate, momentum and Adam work together.
The idea
You are standing on a hillside in fog and have to get to the valley. You can see only the ground at your feet. So you feel which way is downhill, take a step, and repeat.
Steps that are too large overshoot the valley. Steps that are too small take forever. That trade-off is the whole art of optimisation.
What it is good for
Every training run is such a descent, only in a space with billions of directions rather than two.
The three dials
| Dial | If too small | If too large |
|---|---|---|
| Learning rate | Training crawls, sits on plateaus | Loss jumps or turns NaN |
| Batch size | Very noisy gradients | Fewer steps per epoch, often worse generalisation |
| Warmup | Early instability, especially in transformers | Wasted compute |
# Learning-rate range test: raise the rate exponentially per step and
# record where the loss falls most steeply.
import numpy as np
lrs = np.geomspace(1e-6, 1e-1, 60)
# In practice: train one batch per lr, keep the loss, look at the curve.
# Pick roughly one order of magnitude below the minimum of that curve.Common mistakes
- Copying a learning rate from someone else's recipe without scaling for batch size.
- No warmup on transformers: the first steps then wreck normalisation.
- Forgetting gradient clipping, so one outlier batch ruins the model.
- Applying weight decay through the optimiser and as an L2 term in the loss.
The formulas
What Adam costs
Adam keeps two extra states per parameter. For a 7-billion-parameter model in
float32:
| Item | Bytes per parameter | Total |
|---|---|---|
| Weights | 4 | 28 GB |
| Gradients | 4 | 28 GB |
Adam m | 4 | 28 GB |
Adam v | 4 | 28 GB |
| Total, excluding activations | 16 | 112 GB |
Which is why a 7-billion model fits on a single 24 GB card for inference but not for full training. That calculation is precisely the motivation for LoRA and parameter-efficient training.
Why local minima are overrated
In d dimensions a critical point is a minimum only if all d eigenvalues of
the Hessian are positive. Under a random sign distribution the probability of
that is on the order of 2^(−d). For d in the millions that is effectively
zero: almost every critical point is a saddle, and saddles are what momentum
walks off.
Related courses and sources
Adam
The optimiser practically every network today is trained with. Short and readable.
Short and readable; the optimiser practically every network today is trained with.
Deep Learning
The standard work by Goodfellow, Bengio and Courville, free to read. Mathematically dense, complete, and in its foundational parts timeless.
For the systematic route. Strong as a reference, too dense as a first read.
MIT 18.065 Matrix Methods
Singular value decomposition, principal components and optimisation applied to data. The bridge between linear algebra and what models actually compute.
For the step from pure mathematics to what models actually compute.