Metrics
Which numbers carry a decision, which only look good, and how to collect a baseline before it is too late.
·1 min read·By Redaktion KI-Kompass
DETAIL3 sections
Metrics that carry
| Metric | What it measures | How collected |
|---|---|---|
| Time per task | The direct value | Sample before and after |
| Correction rate | Actual quality | Arises at every review |
| Share of unchanged acceptances | Practical usefulness | From the system |
| Refusal rate | Whether it guesses | From the system |
| End-to-end cycle time | Whether the bottleneck merely moved | From the business system |
| Cost per completed task | The only figure the business compares | Invoice plus review effort |
Metrics that do not carry
- Number of texts or requests produced. Measures activity.
- Number of users. Measures curiosity, highest in week one.
- Automation rate. Can rise while the error rate rises with it.
- Satisfaction surveys with no baseline. Measures mood.
- An accuracy figure with no case count or interval. Measures nothing.
The full economics
| Item | Before | After |
|---|---|---|
| Handling time per task | measured | measured |
| Review time per task | 0 | measured |
| Rework on errors, times the error rate | measured | measured |
| Request cost | 0 | measured |
| Introduction, spread over the useful life | 0 | estimated |
| Ongoing upkeep: templates, evaluation sets, checks | 0 | estimated |
The last two rows are missing from most calculations and are why a project with a narrow advantage has none in practice.
Reporting defensibly
- Always with a case count. A rate without a denominator is not information.
- With a confidence interval. At 200 cases it is roughly plus or minus four points.
- Compare two processes paired, on the same cases.
- Report the number of variants tried, or the winner is accidental.
See Statistics for AI.
The correction rate as a data asset
Where a person reviews the result anyway, their correction is a free label. Collected systematically it yields, within six months, a dataset no vendor can supply: real cases from your own operation with the correct answer beside them.
- Capture corrections in a structured way, not as free text in a comment box.
- Record per correction: what was wrong, what is right, which category.
- Evaluate monthly: which error type is rising, which falling.
- Reuse the store as an evaluation set, with a frozen portion.
The metrics sheet
FREE ACCOUNT
Metrics sheet per task
A sheet for baselines, ongoing collection and the full economics, including the items that are usually missing.
Worksheet3 items
Was this page helpful?