Regularisation

The idea#

A student who memorises every past exam question aces the repeat and fails the new paper. Regularisation is everything that forces them to learn the principle instead of the answers.

The four levers#

What helps when#

1. 01

   First check whether there is overfitting at all

   Only if validation loss rises while training loss falls. Otherwise regularisation is the wrong tool.

2. 02

   Data augmentation before algorithms

   Flips, crops and colour jitter for images; back-translation and paraphrase for text. Nearly always stronger than a hyperparameter.

3. 03

   Tune weight decay

   Over a logarithmic search across three orders of magnitude, judged on validation.

4. 04

   Early stopping with patience

   Do not stop at the first worse value but after a fixed number of epochs without improvement.

# Early stopping, as it actually looks
best, patience, waited = float("inf"), 5, 0
for epoch in range(200):
    val = train_one_epoch()                  # placeholder
    if val < best - 1e-4:
        best, waited = val, 0
        save_best_model()                    # not the last one, the best one
    else:
        waited += 1
        if waited >= patience:
            break

The formulas#

L1 induces sparsity because the gradient of the absolute value is a constant ±λ, so small weights get pushed all the way to zero. For L2 the gradient is 2λθ and shrinks with the weight itself, so nothing reaches exactly zero.

Weight decay is not L2#

Under Adam the two are not equivalent. L2 adds the penalty into the gradient, so it is scaled by √v̂ and acts more weakly on parameters with large gradients. AdamW instead subtracts the decay from the weight directly:

That difference is why AdamW consistently outperforms Adam with L2 in the loss on transformers.

Dropout, scaled correctly#

At training time each activation is zeroed with probability p and the rest scaled by 1/(1−p). The expectation stays the same, so nothing has to be adjusted at inference. Forget the scaling and at p = 0.5 you get a model whose outputs in production are systematically twice as large as during training.