Why is accuracy usually unsuitable?

Because with rare events it is dominated by the majority class. At one percent fraud, 'never raise an alarm' already scores 99 percent accuracy.

Precision when false alarms are expensive, recall when missed cases are. Maximising both at once is a trade-off, not an optimisation problem.

What is wrong with ROC-AUC?

With heavily imbalanced classes it looks optimistic because it averages over all thresholds. The precision-recall curve is more informative there.

Reading metrics properly

The confusion matrix

Everything starts with four numbers.

	Model says yes	Model says no
Truly yes	True positive (TP)	False negative (FN)
Truly no	False positive (FP)	True negative (TN)

Every common metric is a ratio of those four cells. With the matrix in front of you, the definitions do not need memorising.

The four key numbers

Metric	Question	Formula
Precision	How many alarms are right?	TP / (TP + FP)
Recall	How many cases do I find?	TP / (TP + FN)
F1	A compromise between the two	2·P·R / (P + R)
Specificity	How quiet does the system stay?	TN / (TN + FP)

A complete picture

import numpy as np
from sklearn.metrics import (confusion_matrix, precision_recall_fscore_support,
                             roc_auc_score, average_precision_score)

rng = np.random.default_rng(0)
y = (rng.random(2000) < 0.03).astype(int)                 # 3 % positives
s = np.clip(rng.normal(0.2 + 0.5 * y, 0.2), 0, 1)         # model score

for thr in (0.3, 0.5, 0.7):
    pred = (s >= thr).astype(int)
    tn, fp, fn, tp = confusion_matrix(y, pred).ravel()
    p, r, f1, _ = precision_recall_fscore_support(y, pred, average="binary",
                                                  zero_division=0)
    print(f"threshold {thr}  TP {tp:3d} FP {fp:3d} FN {fn:3d}  "
          f"P {p:.3f} R {r:.3f} F1 {f1:.3f}")

print("ROC-AUC", round(roc_auc_score(y, s), 3))
print("PR-AUC ", round(average_precision_score(y, s), 3))
# At 3 % positives, chance PR-AUC is 0.03 and chance ROC-AUC is 0.5.
# Only the first number is informative here.

Metrics for other tasks

Task	Usual metric	Trap
Regression	RMSE, MAE, R²	R² can go negative on non-linear relationships
Object detection	mAP at IoU 0.5 to 0.95	The IoU threshold decides half the number
Segmentation	Dice, IoU	Very unstable for small objects
Text generation	Human rating, LLM judge	Automatic scores such as BLEU correlate weakly
Search and RAG	Recall@k, nDCG, MRR	Not comparable without a fixed evaluation set

The generalised metric

F-beta

F_β = (1 + β²) · P · R / (β² · P + R)

At beta equal to one this is F1; at beta equal to two, recall counts four times as much as precision.

β: how much more important recall is than precision
P: precision
R: recall

Better than any F-variant is an explicit cost calculation:

Expected cost per case

K(t) = ( c_FP · FP(t) + c_FN · FN(t) ) / n

The threshold is chosen so that expected total cost is minimal.

c_FP: cost of a false alarm
c_FN: cost of a missed case
n: number of evaluated cases

Plotting that curve over t and reading off its minimum is the only threshold choice that can be justified to a business owner.

Macro, micro, weighted

Across multiple classes the averaging methods differ substantially:

Macro averages the per-class metric with equal weight. Small classes count as much as large ones. Usually the fair choice.
Micro pools all TP, FP and FN. With one label per case, micro F1 is identical to accuracy and inherits its weakness.
Weighted averages by class frequency and thereby hides exactly the problem you were investigating.

If only one number may be reported, report macro F1 and put the confusion matrix next to it.

The confusion matrix#

The four key numbers#

A complete picture#

Metrics for other tasks#

The generalised metric#

Macro, micro, weighted#