Reading benchmarks
Why leaderboards say little about your own use case, which questions to put to a figure, and what an in-house evaluation set looks like.
The idea
A benchmark is an exam question. Whoever answers it well can answer that question. Whether they can do the work you care about is a separate matter.
The questions to put to any figure
- Which task is measured, and how close is it to mine?
- How many cases, and how wide is the confidence interval?
- Could the test data have been in training?
- How many configurations were tried before this figure was reported?
- Who measured, and who has an interest in the result?
The in-house evaluation set
- 01
Collect real cases
50 to 300 transactions from actual operations, not invented ones.
- 02
Record the expected result
What would the right answer be? For open tasks, a criteria list rather than a wording.
- 03
Include hard cases
Edge cases, ambiguous cases, and ones where the right answer is "no information".
- 04
Version and freeze
The set becomes the reference across model changes.
What it costs and returns
| Effort | Result |
|---|---|
| Two to five person-days once | Comparability across years |
| An hour per model version | A defensible decision instead of a feeling |
| Reusable at every vendor change | Negotiating position |
A model as judge
For open tasks, automatic evaluation by a model is the only practical way to scale. The known biases and their remedies:
| Bias | Remedy |
|---|---|
| Longer answers preferred | Exclude length explicitly from the rubric |
| The first option wins | Swap the order and average both runs |
| Own style preferred | Use a different model as judge |
| Confidence read as correctness | Judge against a reference answer rather than freely |
| Agreement bias | Ask explicitly for faults rather than a grade |
In addition, the judge model's agreement with human ratings must be measured on a sample of 50 to 100 cases before relying on it. Agreement below about 70 percent makes automatic evaluation unusable.
Detecting contamination
A drop from 82 to 61 percent on the same tasks with different numbers and different wording is not chance but the usual finding on contaminated benchmarks.
What belongs in a report
- Model name and exact version, not just the family.
- Date of measurement. Providers update models behind the same label.
- All parameters: temperature, system prompt, context length, tool access.
- Number of cases and confidence interval.
- Number of variants tried before this one was reported.
- The cases where the model failed, with examples.
The last point is the most valuable and almost always missing. A list of failures says more about suitability than any average.
Related courses and sources
Hugging Face datasets
Open datasets with description, licence and preview. Useful for evaluation sets, risky as training data without checking provenance.
Good for evaluation sets; as training data only with a provenance and licence check.
ImageNet
The dataset image processing measured itself against for a decade. Historically important and well documented in its biases.
For anyone reading benchmarks: almost every image recognition figure refers back to it.
Model Cards for Model Reporting
The proposal to document purpose, limits and tested groups for every model. Today effectively a precondition for any audit.
For anyone preparing an audit; model cards have effectively become a precondition.