Models that compute before they answer

The idea#

A language model has a fixed amount of computation available per token. Answering a hard question in a single token is therefore structurally hard, however good the model is.

Let it write twenty intermediate steps instead and it gets twenty times the computation. The intermediate steps are also a notepad on which partial results stay put.

When it helps#

The methods#

The last row is often overlooked and is the best answer for arithmetic: a model that delegates a calculation to a calculator is more reliable and cheaper than one that works it out.

Self-consistency#

from collections import Counter

def self_consistent(question, model, n=8):
    # Compute several times at temperature > 0 and take the most common answer.
    answers = [model(question, temperature=0.7)["final"] for _ in range(n)]
    counts = Counter(answers)
    best, freq = counts.most_common(1)[0]
    return best, freq / n          # the second value is a confidence measure

The second return value is the real gain: it is an empirical confidence measure. Below about 60 percent agreement, the case belongs in human review.

Test-time compute#

At 2,000 thinking tokens and a 200-token answer, cost is eleven times that of a direct answer. Empirically, accuracy on multi-step tasks improves roughly logarithmically in T_think: doubling thinking time yields a roughly constant increment until saturation.

The chain of thought is not a log#

This is the most important caveat for any evidence purpose. The printed chain is itself generated text and need not match the computation that produced the answer. Studies show cases where a model demonstrably uses a hint in the prompt and does not mention it in its chain.

For practice that means:

See Duties by role and Logging.

When the cost pays#

Worked through: extra cost 0.02 EUR per request, accuracy improvement 6 percentage points, error cost 15 EUR. Then 0.06 × 15 = 0.90 EUR > 0.02 EUR: the surcharge pays back 45 times over. On a task with no error cost, such as rephrasing a text, it never pays.