Quantisation
Making models smaller without breaking them: int8 and int4, outlier channels, the methods compared, and how to measure the loss.
The idea
A weight is normally stored as a 16-bit number. Store it instead as a 4-bit integer with a shared conversion factor per group and the model shrinks to a quarter, and fits on a card it did not fit on before.
What that buys
| Format | 7 bn model | 70 bn model | Typical quality cost |
|---|---|---|---|
| float16 | 14 GB | 140 GB | Baseline |
| int8 | 7 GB | 70 GB | under 1 % |
| int4 | 3.5 GB | 35 GB | 1 to 3 % |
| int3 and below | 2.6 GB | 26 GB | Substantial, task-dependent |
The methods
| Method | Principle | Needs calibration data |
|---|---|---|
| Round to nearest | Simply round | no |
| GPTQ | Compensate error layer by layer | yes, a few hundred examples |
| AWQ | Protect important channels | yes |
| SmoothQuant | Shift scale between weight and activation | yes |
| Quantisation-aware training | Simulated during training | full training |
- Always calibrate with data resembling production, not arbitrary text.
- After quantising, measure against your own fixed evaluation set, not against someone else's numbers.
- Watch the group size: smaller groups are more accurate and need more memory for the scales.
- Often keep the output layer and embedding at higher precision; they are sensitive and small.
The calculation
The overhead from the scales
At n = 4 and g = 128 that is 4 + 32/128 = 4.25 bits per weight. At g = 32
already 5.0 bits, that is 18 percent more memory for noticeably better accuracy.
That trade-off is the real decision when picking a quantisation format.
Outliers
In large language models a few dimensions carry activations a hundred times larger than the rest. They force the scale factor up and make every other value in the same group correspondingly coarser.
Three counter-strategies:
- Separate handling. Outlier channels stay in float16 while the rest computes in int8.
- Shifting. Part of the scale is moved from the activation into the weight so both become well behaved.
- Importance weighting. Channels with large activations are scaled before quantisation so their error comes out smaller.
Measure rather than assume
- 01
Assemble your own evaluation set
100 to 300 tasks from real use, with expected results.
- 02
Measure the float16 baseline
Without that number every comparison is meaningless.
- 03
Measure at each stage
int8, then int4, each against the same set.
- 04
Break out by task type
A single overall figure hides that arithmetic suffers more than phrasing.
For a high-risk system under the AI Act this measurement is not optional: changing the numeric representation changes the system and belongs, with its effects, in the technical documentation.
Related courses and sources
Distilling the Knowledge in a Neural Network
A large model teaches a small one what it knows. The basis of the small models running in production today.
For anyone deploying small models in production who wants the basis for it.
Hugging Face model hub
Hundreds of thousands of open models with licence, model card and weights. The first place to look when checking whether a local model is enough for a task.
The licence is on the model card, and not every open model allows commercial use.
llama.cpp
Running quantised models on ordinary hardware, down to single cards and small boards. The reference implementation for operating without a data centre.
For running without a data centre, down to single cards and small boards.