Why models hallucinate
Not a bug but a consequence of the design: where invented statements come from, which signals precede them, and what actually helps.
The idea
A model picks every word so that it fits the text so far. It has no register of what it knows and does not know. When no matching information is available, a plausible continuation still emerges: only an invented one.
That also explains why invented statements sound especially convincing: that is exactly what the method optimises for.
Where it happens most
- With case numbers, sections, standards and citations.
- With figures, years and statistics.
- With quotations and source attributions.
- With very specialised questions on which little text exists.
- With questions containing a false premise.
What actually helps
| Measure | Effect | Effort |
|---|---|---|
| Retrieval with a citation duty | very large | medium |
| Verifying citations against the source | large | medium |
| "If not in the sources, say so" | large | small |
| Low temperature for factual questions | medium | none |
| Self-consistency across several runs | medium | high |
| Tools for figures and dates | large, within scope | small |
| A polite instruction in the prompt | small | none |
The last row is the most common measure and the weakest. "Do not invent anything" in the prompt lowers the rate measurably, but far less than grounding in real sources.
A signal from the distribution
import numpy as np
def uncertainty(logprobs_per_token):
"""Mean negative log probability of the chosen tokens.
High values suggest guessing."""
return float(-np.mean(logprobs_per_token))
# In practice: fix a threshold on your own evaluation set and route
# answers above it to review.Why it is structural
Truth does not appear in that objective. A model that reproduces the text distribution perfectly also reproduces the frequency of false statements in the training material. On top of that: for a question the data says nothing about, every continuation is equally probable: the model rolls dice, and the result sounds certain.
Two kinds of uncertainty
| Kind | Cause | Recognised by |
|---|---|---|
| Aleatoric | The question is ambiguous | High entropy across all runs |
| Epistemic | The model does not know | High variance between several runs |
The second is the relevant one. It can be measured by asking the same question several times at temperature above zero and evaluating agreement. Low agreement on a factual question is a strong warning sign, however confident each individual answer sounds.
Measuring grounding
This can be estimated automatically: split the answer into individual claims and check each against the supplied passages. Faithfulness below 0.9 means every tenth claim is unsupported: not tenable for legal or medical information.
The right not to answer
The single most effective measure is to explicitly permit and reward non-answers. A system that says "that is not in the documents available to me" in 8 percent of cases is more useful than one that always answers and is wrong 8 percent of the time. The refusal rate therefore belongs tracked as a metric, not treated as a defect. See Troubleshooting.