Sampling and temperature
How a probability distribution becomes a concrete word: temperature, top-k, top-p, and which value is right when.
The idea
After each step the model has a probability for every possible next token. One has to be chosen. Always taking the most probable gives correct but monotonous text. Rolling dice gives lively but unreliable text.
Temperature controls how much dice-rolling happens.
The dials
| Dial | Effect |
|---|---|
| Temperature 0 | Always the most probable token |
| Temperature 1 | Exactly the model distribution |
| Temperature above 1 | The improbable becomes more probable |
| Top-k | Only the k most probable are eligible |
| Top-p | Only as many as needed for their sum to reach p |
What the dials do
import numpy as np
def softmax(z, T=1.0):
z = np.asarray(z, float) / T
z = z - z.max()
e = np.exp(z)
return e / e.sum()
logits = [4.0, 3.5, 2.0, 1.0, 0.5]
for T in (0.2, 0.7, 1.0, 1.5):
p = softmax(logits, T)
print(f"T={T:.1f} {np.round(p, 3)} entropy {-(p*np.log2(p)).sum():.2f}")
# Low temperature -> one option dominates, entropy small.
# High temperature -> everything levels out, entropy large.Recommendations by task
| Task | Temperature | Top-p |
|---|---|---|
| Extracting fields from a document | 0 | 1.0 |
| Classifying | 0 | 1.0 |
| Generating code | 0 to 0.2 | 0.95 |
| Summarising | 0.3 | 0.9 |
| Drafting a customer reply | 0.5 | 0.9 |
| Brainstorming | 0.9 | 0.95 |
The formulas
Top-p beats top-k because the candidate set adapts to the situation. Where a
continuation is practically certain, V_p contains exactly one token; where the
situation is open, it contains many. A fixed k would be wrong in both cases.
Why temperature 0 is not exactly reproducible
Three causes, in order:
- Floating-point addition is not associative; different thread partitions give minimally different logits.
- Under batching the partitioning changes with batch composition, and with it the last bit.
- With two nearly equal logits the choice flips on that last bit.
For evidence purposes that means: reproducibility is not what gets logged, the result is, together with model version, parameters and timestamp. See Logging.
The link to hallucination
A high temperature raises the chance of picking a token the model itself gave low probability. For factual statements that is exactly the mechanism by which wrong claims arise.
Conversely, the entropy of the output distribution serves as a warning signal: high entropy across several consecutive tokens means the model is guessing. That quantity is cheap to collect and a usable trigger for review. See Why models hallucinate.