One complete pass through the training data. Large models often see their data exactly once, small models dozens of times.

Why not use all the data at once?

Because memory will not hold it, and because the noise of small batches helps escape flat plateaus.

How do I know training is broken?

Loss rises, jumps sharply, or turns NaN. In that order you check learning rate, data preparation, and number format.

Gradient descent

The idea

The model makes a prediction, compares it with the right answer, and shifts its parameters a little in the direction that reduces the error. That is the whole algorithm. Everything else is a variation on those three steps.

What it is good for

It is why models with billions of parameters are trainable at all: you never have to try every possibility, only follow the local slope.

The three variants

Variant	Batch size	Property
Full-batch	all data	Smooth but very expensive per step
Stochastic (SGD)	1	Very noisy, rarely sensible
Mini-batch	8 to 4096	The normal case, a good compromise

A complete training loop by hand

import numpy as np
rng = np.random.default_rng(0)

X = rng.normal(size=(500, 3))
w_true = np.array([2.0, -1.0, 0.5])
y = X @ w_true + rng.normal(0, 0.1, 500)

w, lr, batch = np.zeros(3), 0.05, 32
for epoch in range(30):
    idx = rng.permutation(500)                 # shuffling each epoch matters:
    for start in range(0, 500, batch):         # otherwise the model learns
        b = idx[start:start + batch]           # the ordering as well.
        pred = X[b] @ w
        grad = 2 * X[b].T @ (pred - y[b]) / len(b)
        w -= lr * grad
    if epoch % 10 == 0:
        print(epoch, np.round(w, 3), round(float(((X @ w - y) ** 2).mean()), 5))

Diagnosis from the loss curve

Loss falls then flattens: healthy.
Loss jumps sharply: lower the learning rate or raise the batch size.
Loss falls, validation rises: overfitting, see regularisation.
Loss stays flat: learning rate too small, data badly scaled, or no gradient flowing.
Loss turns NaN: number format or missing gradient clipping.

The update rule

Mini-batch gradient descent

θ ← θ − η · (1/|B|) · Σ_{i∈B} ∇θ L(f(xᵢ; θ), yᵢ)

The parameters move by the learning rate times the mean gradient over the mini-batch.

θ: the parameters
η: the learning rate
B: a mini-batch of size |B|
L: the per-example loss

Why noise helps

A mini-batch gradient is an unbiased estimator of the full gradient with variance proportional to 1/|B|. That noise acts like a temperature: it keeps the system out of narrow, deep minima. Empirically, wide minima generalise better, which is why an overly large batch without a learning-rate adjustment measurably degrades test scores.

The usual remedy is the linear scaling rule: double the batch, double the learning rate, with a warmup over the first few hundred steps. Beyond roughly 8,000 examples per batch the rule stops holding.

Schedules

Schedule	Shape	Used for
Constant	unchanged	Short runs, debugging
Step	×0.1 every k epochs	Classical image classification
Cosine	smoothly to zero	Language models, today's default
One-cycle	up then down	Fast fine-tuning

Cosine with warmup is effectively standard for transformers: the first 2 to 5 percent of steps raise the rate linearly, after which it follows a cosine arc down to nearly zero.

The idea#

What it is good for#

The three variants#

A complete training loop by hand#

Diagnosis from the loss curve#

The update rule#

Why noise helps#

Schedules#