Memory and bandwidth
The calculation that precedes every hardware purchase: how much memory a model needs, what the KV cache costs, and where the real limit sits.
The idea
A model must fit entirely into the graphics card's memory. If it does not, data is swapped, and access over the system bus is about fifty times slower. The practical consequence is not "a bit slower" but "unusable".
The rule of thumb
| Format | Bytes per parameter | 7 bn | 13 bn | 70 bn |
|---|---|---|---|---|
| float32 | 4 | 28 GB | 52 GB | 280 GB |
| float16 | 2 | 14 GB | 26 GB | 140 GB |
| int8 | 1 | 7 GB | 13 GB | 70 GB |
| int4 | 0.5 | 3.5 GB | 6.5 GB | 35 GB |
Add 10 to 20 percent runtime overhead plus the context memory.
The KV cache
This is the item routinely missing from plans.
Worked through for a model with 32 layers, h_kv = 4096, float16:
| Context | 1 request | 8 requests | 32 requests |
|---|---|---|---|
| 4,096 tokens | 2.1 GB | 17.2 GB | 68.7 GB |
| 32,768 tokens | 17.2 GB | 137 GB | 550 GB |
At 32,000 tokens of context the cache is already larger than the 7-billion model
itself. Which is exactly why modern architectures use grouped queries: with eight
query heads sharing one key-value head, h_kv falls by eight and the cache with it.
- Budget model, cache and a safety margin separately.
- Cap the maximum context length in production, or a single user determines capacity.
- With tight memory, quantise the cache first; it costs less quality than int4 weights.
The roofline model
For a card with P_max = 1,000 TFLOP/s and B = 3,000 GB/s the knee sits at
I = 333 FLOPs per byte. Typical values:
| Operation | Arithmetic intensity | Limited by |
|---|---|---|
| Generating a single token | ~2 | Bandwidth |
| Generation with batch 64 | ~128 | Bandwidth |
| Generation with batch 512 | ~1,000 | Compute |
| Prompt processing (prefill) | ~500 | Compute |
| Training | ~2,000 | Compute |
The table explains an often surprising observation: processing a long prompt runs at full compute while the subsequent token-by-token generation barely loads the card. The two phases have different bottlenecks and are planned separately in modern systems.
A complete sizing calculation
Requirement: a 13-billion-parameter model, 8,000 tokens of context, 16 concurrent
requests, float16 for weights and cache.
| Item | Calculation | Memory |
|---|---|---|
| Weights | 13e9 × 2 B | 26.0 GB |
| KV cache | 2 × 40 × 8,000 × 5,120 × 2 B × 16 | 104.9 GB |
| Activations and buffers | ~10 % | 13.1 GB |
| Total | 144.0 GB |
That does not fit an 80 GB card. Three ways out, in order of effect:
- Grouped queries in the model, factor 8 on the cache: from 104.9 down to 13.1 GB.
- Cache in int8, factor 2: halved again.
- Cap context at 4,000 tokens, factor 2 on the cache.
- Weights in int8, factor 2: from 26.0 down to 13.0 GB.
With the first two measures the total lands around 46 GB and fits comfortably. That order is typical: the cache is nearly always a bigger lever than the weights.
Related courses and sources
FlashAttention
Computing attention without ever materialising the quadratic matrix. The precondition for long contexts.
For anyone running long contexts who needs to know what memory hangs on.
vLLM
A high-throughput server with continuous batching and paged attention cache. The reason self-hosting becomes economical at volume.
For self-hosting at volume; the reason it can pay off at all.