How do I identify overfitting reliably?

By the gap between training and validation loss. If one falls while the other rises, the case is unambiguous.

What do I do about underfitting?

More capacity, better features, longer training or less regularisation, check in that order.

Does the bias-variance trade-off apply to large language models?

Only partly. Very large models show double descent: test error rises with size at first and then falls again.

Overfitting, bias and variance

The idea

Two ways of being wrong. First: you consistently aim two metres to the left. That is bias, a systematic error that more practice will not fix. Second: you scatter wildly around the centre. That is variance, and practice helps a lot.

How to tell them apart

Observation	Diagnosis	Remedy
Training poor, validation poor	Bias, underfitting	Bigger model, better features, train longer
Training good, validation poor	Variance, overfitting	More data, regularisation, smaller model
Both good, production poor	Distribution shift	Re-check the test set, add current data

Reading learning curves

Two curves over the number of training examples: training error and validation error.

Both curves meet at a high level: bias. More data will not help.
Large gap, validation still falling: variance. More data helps immediately.
Large gap, validation flat: variance, but data alone is no longer enough.
Training error near zero at moderate data volume: memorisation.

from sklearn.model_selection import learning_curve
from sklearn.ensemble import RandomForestClassifier
from sklearn.datasets import make_classification
import numpy as np

X, y = make_classification(n_samples=4000, n_features=25,
                           n_informative=6, random_state=0)
sizes, tr, va = learning_curve(
    RandomForestClassifier(n_estimators=200, random_state=0),
    X, y, train_sizes=np.linspace(0.1, 1.0, 6), cv=5, n_jobs=-1)
for n, a, b in zip(sizes, tr.mean(1), va.mean(1)):
    print(f"{int(n):5d}  train {a:.3f}   valid {b:.3f}   gap {a-b:.3f}")

The decomposition

Bias-variance decomposition of expected error

E[(y − f̂(x))²] = ( E[f̂(x)] − f(x) )² + Var[f̂(x)] + σ² bias² variance noise

Expected squared error splits into systematic deviation, spread across samples, and unavoidable noise.

f̂: the model learned from a random sample
f: the true function
σ²: irreducible noise in the labels

The third term is the ceiling on any achievable quality. Estimating it, through agreement between two independent annotators, is the single most useful piece of work to do before choosing a model.

Double descent

The classical U-curve holds for models below the interpolation threshold, that is while parameters < data points. Beyond it, test error rises sharply at first, peaks around parameters ≈ data points, and falls again, often below the first minimum. Modern language models operate permanently to the right of that threshold, which is why "smaller model to fight overfitting" is the wrong reflex there.

Measuring distribution shift

A simple and harsh test: train a classifier to distinguish whether a record came from training or from production. If it reaches ROC-AUC well above 0.5, the distributions differ measurably and every validation figure is worthless for production. See MLOps.

The idea#

How to tell them apart#

Reading learning curves#

The decomposition#

Double descent#

Measuring distribution shift#