Context windows in depth
Why long context is expensive, what gets lost in the middle, and how the limit can be moved without retraining the model.
The idea
The context window is everything the model can see at once for one request: system instruction, history, attached documents and the answer so far. It is measured in tokens, not characters or pages.
Orders of magnitude
| Window | Roughly equivalent to |
|---|---|
| 8,000 tokens | 20 to 25 pages of prose |
| 32,000 tokens | 80 to 100 pages |
| 128,000 tokens | a medium-length book |
| 1,000,000 tokens | several thousand pages |
The cost
| Phase | Cost | What that means |
|---|---|---|
| Processing the prompt | quadratic in length | Double the input, four times the time to first token |
| Generating tokens | linear in length | Every new token reads the whole cache |
| Memory | linear in length | The cache grows with every token |
- Invariant parts at the front, so prompt caching engages.
- The most important material at the start and the end, not in the middle.
- Retrieve selectively rather than pasting everything. Ten relevant passages beat two hundred arbitrary ones.
- Cap the maximum input length in production, or capacity cannot be planned.
Extending context
Plain interpolation degrades resolution at short distances, because adjacent tokens are compressed too. NTK-aware methods therefore stretch unevenly, high frequencies less than low ones, preserving near-range precision.
Measuring the middle problem
The standard test places an unambiguously findable fact at various positions in a long context and plots hit rate against position. A typical profile for a model with a 128,000-token window:
| Position of the fact | Hit rate |
|---|---|
| first 10 % | 95 to 99 % |
| 25 % | 85 to 92 % |
| 50 % | 60 to 80 % |
| 75 % | 80 to 90 % |
| last 10 % | 93 to 98 % |
The U shape is robust across models. It is why retrieval with a few relevant passages is practically superior to a very long context.
Cost and memory calculation
For a model with 32 layers, h_kv = 1024 (grouped queries), float16:
| Context | KV cache per request | Time to first token, relative |
|---|---|---|
| 4,000 | 0.5 GB | 1 |
| 16,000 | 2.1 GB | 16 |
| 64,000 | 8.4 GB | 256 |
| 128,000 | 16.8 GB | 1,024 |
The last column is why very long prompts take seconds to minutes before the first output in practice. If that cannot be accepted, you retrieve rather than append. See RAG in depth.