Statistics for AI

The idea#

Have ten invoices checked and nine come back right: 90 percent. Check ten different ones and it might be 70. Both figures are correctly measured and both say little, because the sample is too small.

Statistics is the tool that makes that uncertainty visible instead of rounding it away.

What it is good for#

Every decision about an AI system rests on a comparison: model A against model B, with AI against without, prompt v1 against prompt v2. Without an error figure, those comparisons are coin flips with decimal places.

The calculation you always need#

For a measured rate p from n cases the standard error is √(p(1−p)/n), and the 95 percent interval is roughly twice that.

import math

def ci95(hits, n):
    p = hits / n
    se = math.sqrt(p * (1 - p) / n)
    return round(p - 1.96 * se, 4), round(p + 1.96 * se, 4)

print(ci95(180, 200))    # (0.8584, 0.9416)
print(ci95(900, 1000))   # (0.8814, 0.9186)

Comparing two models properly#

1. 01

   Same cases

   Run both models on identical inputs. Separate samples throw away half the statistical power.

2. 02

   Count only the disagreements

   Cases where both perform the same contribute nothing. What counts is the cases where exactly one was right.

3. 03

   Run McNemar's test

   With 30 cases only A solves and 15 only B solves, the statistic is (30−15)²/(30+15) = 5.0, which corresponds to p ≈ 0.025 and is just about defensible.

4. 04

   State the effect size

   Always report the absolute difference and its interval alongside significance, so the decision is made on substance rather than on p-values.

The formulas#

The √n factor is the most commercially significant number in statistics: to double measurement precision you need four times the data.

Bootstrap instead of a formula#

For metrics with no closed-form error, such as F1, mAP, BLEU or mean handling time, bootstrapping is the pragmatic route.

import numpy as np
rng = np.random.default_rng(0)

def bootstrap_ci(scores, reps=10_000):
    scores = np.asarray(scores)
    means = [rng.choice(scores, size=scores.size, replace=True).mean()
             for _ in range(reps)]
    # Percentile method: simple, robust, no distributional assumption.
    return np.percentile(means, [2.5, 97.5])

print(bootstrap_ci(rng.normal(0.83, 0.12, 300)).round(3))

The mistake that ruins benchmarks#

Multiple testing. Try 20 prompt variants on the same test set and report the best, and at a 5 percent significance level you have a 1 − 0.95²⁰ ≈ 64 % chance of finding a purely accidental "significant" winner. The remedy is a held-out test set touched exactly once, plus a Bonferroni or Benjamini-Hochberg correction for everything before it. See also Data leakage.