What is the difference between P(A|B) and P(B|A)?

A very large one. How often a test fires when someone is ill has little to do with how likely illness is when the test fires.

How often something occurs at all, before any test is run. Ignoring it is the single most common mistake when judging AI results.

Why does a language model output probabilities?

Because it estimates a distribution over all possible next tokens. Which token is then chosen is decided later, by sampling.

Probability

The idea

Probability measures ignorance, not randomness. When a model says a document is "87 percent an invoice", it means: among all documents that look like this one, 87 percent were invoices in the training data.

What it is good for

Every language-model output is a probability distribution over the whole vocabulary. Every classification is a distribution over classes. Being able to read those numbers is how you notice when a model is guessing.

The example that explains everything

A checking system detects fraudulent invoices with 99 percent sensitivity and a 1 percent false-alarm rate. Out of 10,000 invoices, 10 are fraudulent.

	actually fraud	actually fine	total
System raises alarm	9.9	99.9	109.8
System stays silent	0.1	9,890.1	9,890.2
Total	10	9,990	10,000

Of 109.8 alarms, 9.9 are real. That is 9 percent. Nine out of ten alarms are false, even though the system is 99 percent "correct".

def posterior(prevalence, sensitivity, specificity):
    tp = prevalence * sensitivity
    fp = (1 - prevalence) * (1 - specificity)
    return tp / (tp + fp)

print(round(posterior(0.001, 0.99, 0.99), 4))   # 0.0902
print(round(posterior(0.100, 0.99, 0.99), 4))   # 0.9167

Same model quality, two orders of magnitude difference in usefulness.

The formulas

Conditional probability and Bayes' theorem

P(A|B) = P(B|A) · P(A) / P(B) P(B) = P(B|A) · P(A) + P(B|¬A) · P(¬A)

The probability you want is the reversed one, multiplied by the base rate and divided by the overall frequency of the observation.

P(A): the base rate of A before anything is observed
P(B|A): how often B occurs when A holds
P(A|B): what you want: how often A holds when B was observed

Where this sits inside the model

An autoregressive language model factorises the probability of a whole sequence using the chain rule of probability:

Autoregressive factorisation

P(x₁…xₙ | θ) = Πᵢ P(xᵢ | x₁…xᵢ₋₁, θ)

The probability of a text is the product of the probability of each token given all the ones before it.

x₁…xₙ: the tokens of the sequence
θ: the model parameters

In practice this is computed with logarithms, because a product of thousands of numbers below one underflows to zero in float32 immediately. The product becomes a sum of log probabilities, and the mean negative log probability becomes perplexity, see Information and entropy.

Measuring calibration

Expected Calibration Error buckets predictions by confidence and compares the mean stated confidence per bucket with the actual hit rate in it. An ECE above 0.1 means the model's confidence figures are unusable as a basis for decisions and need recalibrating, for example by temperature scaling on a validation set.

The idea#

What it is good for#

The example that explains everything#

The formulas#

Where this sits inside the model#

Measuring calibration#