Why is bfloat16 preferred over float16 for training?

Because it has the same range as float32 and therefore overflows far less often. It has less precision, but during training range matters more than precision.

Does int8 cost a model quality?

Measurably little, if the scale is determined per channel. At int4 the loss becomes visible and has to be weighed against the memory saved.

How do I recognise numerical trouble?

NaN or Inf in the loss, results that differ noticeably between two runs, and gradients that become exactly zero.

Numbers in a computer

The idea

A computer does not store numbers, it stores approximations. 0.1 + 0.2 gives 0.30000000000000004 in almost every programming language. That is not a bug but a consequence of 0.1 having an infinite binary expansion.

In one calculation this goes unnoticed. Across billions of chained calculations it does not.

What it is good for

Knowing how imprecise a format is lets you decide how small a model may become before it gets worse. That decision is the starting point of every hardware question.

The formats compared

Format	Bits	Exponent	Mantissa	Largest value	Smallest relative resolution
float32	32	8	23	3.4e38	~1.2e-7
float16	16	5	10	65,504	~9.8e-4
bfloat16	16	8	7	3.4e38	~7.8e-3
int8	8	n/a	n/a	127	one step per scale unit
int4	4	n/a	n/a	7	16 levels in total

bfloat16 is float32 with a truncated mantissa. Conversion is therefore trivial and the range identical, which is why it won out for training.

import numpy as np

a = np.float16(60000)
print(a * 2)                       # inf  -> float16 overflows at 65504

x = np.float32([1e8, 1.0, -1e8])
print(x.sum())                     # 0.0  -> the 1 has been cancelled out
print(np.float64(x).sum())         # 1.0

Common mistakes

Accumulating loss values in float16 instead of float32.
Computing softmax without subtracting the maximum: exp(800) is inf at once.
Computing variance as E[x²] − E[x]²; with large means this cancels catastrophically.
Quantising a model to int4 and then quoting benchmarks from the float16 run.

Stable softmax

Softmax, numerically stable

softmax(z)ᵢ = exp(zᵢ − m) / Σⱼ exp(zⱼ − m), m = maxⱼ zⱼ

The maximum is subtracted inside the exponent; the result is mathematically identical but cannot overflow.

zᵢ: the logits, the raw outputs before normalisation
m: the maximum over all logits

Without that subtraction, exp(zᵢ) at zᵢ = 800 is already inf in float32, and inf/inf gives NaN. With it, the largest exponent is exactly exp(0) = 1 and the computation is independent of the scale of the logits.

Quantisation, worked through

Symmetric int8 quantisation

s = max|w| / 127 q = round(w / s) ŵ = q · s

The scale is the largest magnitude in the channel divided by 127; the weight is mapped onto it and scaled back when computing.

w: a weight in floating point
s: the per-channel scale factor
q: the integer value between −127 and 127

Example: a channel with max|w| = 0.42. Then s = 0.42/127 = 0.003307. A weight w = 0.1234 becomes q = round(37.32) = 37, and back ŵ = 0.12235. The error is 0.00105, that is 0.85 percent of the value and 0.25 percent of the channel maximum. Across thousands of weights these errors largely cancel, as long as they are uncorrelated: which is the real reason quantisation works.

Outliers break that assumption. A single weight with |w| = 5 in a channel whose other values are below 0.5 forces s up tenfold and makes every other weight ten times coarser. Which is why methods such as LLM.int8() and AWQ handle outlier channels separately, see Quantisation.

Determinism

Floating-point addition is not associative. Two runs of the same matrix multiplication with different thread partitioning do not produce bit-identical results. Anyone who needs reproducible outputs, for evidence under Logging, must force deterministic kernels and accept the throughput penalty, or log the result and the model version instead of relying on reproducibility.

The idea#

What it is good for#

The formats compared#

Common mistakes#

Stable softmax#

Quantisation, worked through#

Determinism#