What is the difference between loss and metric?

The loss is optimised and must be differentiable. The metric is reported and may be as awkward as you like, accuracy for instance.

When MAE instead of MSE?

When outliers exist in the target and should not dominate. MSE punishes an error of 10 a hundred times as hard as an error of 1; MAE only ten times.

Why not optimise accuracy directly?

Because it is piecewise constant and its gradient is zero everywhere. You optimise a smooth surrogate and report accuracy afterwards.

Loss functions

The idea

Before a model can learn, somebody has to define what "off" means. Is an estimate of 90 instead of 100 as bad as 110 instead of 100? Is a missed fraud case as expensive as a false alarm?

The loss function is those answers, cast into numbers.

What it is good for

It is where business priorities enter the model. Choose it unconsciously and you have handed that decision to a library default.

The selection

Task	Loss	Property
Regression, clean data	MSE	Prefers the mean, sensitive to outliers
Regression, outliers	MAE or Huber	Prefers the median, robust
Binary classification	Binary cross-entropy	Yields calibratable probabilities
Very rare class	Focal loss	Damps easy cases, emphasises hard ones
Multi-class	Cross-entropy	Standard, steadier with label smoothing
Learning similarity	Triplet, InfoNCE	Basis of most embeddings
Ranking	Pairwise logistic loss	Optimises order rather than values

import numpy as np

y  = np.array([100.0, 100.0, 100.0])
p1 = np.array([ 90.0, 100.0, 110.0])   # evenly off
p2 = np.array([100.0, 100.0,  70.0])   # one gross outlier

for name, p in [("evenly off", p1), ("one outlier", p2)]:
    mse = float(((y - p) ** 2).mean())
    mae = float(np.abs(y - p).mean())
    print(f"{name:14s} MSE {mse:8.1f}   MAE {mae:6.1f}")
# MSE punishes the single gross error harder than two moderate ones.

The formulas

The four most important losses

MSE = (1/n) Σ (yᵢ − ŷᵢ)² MAE = (1/n) Σ |yᵢ − ŷᵢ| Huber = ½(y−ŷ)² for |y−ŷ| ≤ δ δ|y−ŷ| − ½δ² otherwise BCE = −[ y·log p + (1−y)·log(1−p) ] Focal = −(1−p_t)^γ · log p_t

Squared, absolute, a mix of the two, and for classification the logarithm of the probability assigned to the correct label.

y: the true value or label
ŷ, p: the prediction
δ: the Huber threshold above which the penalty turns linear
γ: the focal parameter, usually 2

Why MSE estimates the mean

Set the derivative of Σ(yᵢ − c)² with respect to c to zero and you get c = (1/n)Σyᵢ, the mean. For Σ|yᵢ − c| the derivative is the count above minus the count below; it hits zero at the median.

That has a practical consequence that is often missed: model handling times with MSE and you predict the mean duration. Handling times are always right-skewed, and there that value sits well above the typical case. Right for capacity planning, wrong for a customer promise.

Class weights, worked through

At 2 percent positives with unweighted cross-entropy, 98 percent of the gradient signal comes from the majority class. A weight of w₊ = n/(2·n₊) = 25 balances that. In practice it usually decalibrates the probabilities: the threshold has to be re-derived afterwards, see Reading metrics properly.

The idea#

What it is good for#

The selection#

The formulas#

Why MSE estimates the mean#

Class weights, worked through#