Speeding up inference
Batching, caching, speculative decoding and distillation: the levers that actually reduce response time and cost per request.
The idea
A graphics card serving a single request sits over ninety percent idle: it is waiting for weights to arrive from memory. Serve several requests together and it uses the same weights for all of them, waiting only once.
The levers, by effect
| Lever | Typical gain | Quality loss |
|---|---|---|
| Continuous batching | 5 to 20 times throughput | none |
| Prompt caching | 30 to 90 % less prefill | none |
| Speculative decoding | 1.5 to 3 times per request | none |
| Quantisation to int8 | 2 times | small, measurable |
| Quantisation to int4 | 4 times | noticeable, must be checked |
| Distillation to a smaller model | 5 to 20 times | substantial, task-dependent |
Continuous batching
Classical batching waits until every request in the batch is finished. Since answers differ in length, the batch waits for the longest. Continuous batching admits a new request as soon as any one finishes.
Classical: [====A====][==B==]......[========C========] done after C
Continuous: A and B run; when A ends D starts, then E ...
The gain depends on the spread of answer lengths. With widely varying lengths, the normal case, it is a factor of five or more.
Prompt caching
When many requests share a system prompt or a document, its KV cache need only be computed once.
- Put the invariant part at the very front of the prompt, or the cache does not engage.
- With RAG, keep the order of retrieved passages stable.
- Measure the cache hit rate; below 30 percent the effort rarely pays.
Speculative decoding
Worked through for γ = 4, α = 0.8, c = 0.05:
E = (1 − 0.8⁵)/(1 − 0.8) = (1 − 0.328)/0.2 = 3.36 tokens per round, divided by
1 + 4 × 0.05 = 1.2, giving a gain of about 2.8 times.
At α = 0.5 that falls to E = 1.94 / 1.2 = 1.6 times. Acceptance rate is
therefore the decisive quantity, and it depends on how well the small model
matches the large one. A draft model distilled from the same model reaches far
higher rates than an arbitrary small one.
Important: the method is lossless. The output distribution stays exactly that of the large model, because rejected proposals are correctly resampled.
The order in which to optimise
- 01
Measure which phase is stuck
Record time to first token and tokens per second separately. Without that split you optimise blind.
- 02
Serving before model
Continuous batching, prompt caching, paged attention. Lossless and needing no evaluation effort.
- 03
Formats
KV cache to int8 first, then the weights. Measure against a fixed evaluation set after each step.
- 04
Model
A smaller model, distillation, or a router sending easy requests to a small model and hard ones to a large one.
What a router is worth
If 70 percent of requests are answered equally well by a model at a tenth of the
cost, total cost falls to 0.7 × 0.1 + 0.3 × 1.0 = 0.37, a 63 percent reduction.
The difficulty lies in the classification: it must itself be cheap and must not
misroute hard cases. In practice a conservative rule that escalates when in doubt
works well.
Related courses and sources
Switch Transformers
Only a fraction of the parameters compute per token. The paper that brought mixture of experts into wide use.
For understanding why large models do not compute every parameter per token.
vLLM
A high-throughput server with continuous batching and paged attention cache. The reason self-hosting becomes economical at volume.
For self-hosting at volume; the reason it can pay off at all.