Supervised learning
Classification and regression: how a model learns from examples with known answers, and how to spot that the labels are the real problem.
The idea
You hand a trainee two thousand completed cases with the outcome written beside each one. After a while they see the pattern and can judge new cases themselves. That is supervised learning: examples with solutions, from which a rule emerges.
What it is good for
By a wide margin the most common kind of AI in operation: filing documents, routing tickets, estimating prices, predicting failures, detecting intent.
Task types
| Task | Output | Usual loss | Usual metric |
|---|---|---|---|
| Binary classification | yes / no | Binary cross-entropy | Precision, recall, ROC-AUC |
| Multi-class | one of k | Cross-entropy | Macro F1, confusion matrix |
| Multi-label | several of k | Binary cross-entropy per label | Micro F1 |
| Regression | number | MSE or MAE | RMSE, MAE, R² |
| Ranking | order | Pairwise loss | NDCG, MRR |
The threshold is a business decision
A classifier outputs a probability. Where that becomes a yes is not set by the model but by what a false alarm costs relative to a missed case.
import numpy as np
from sklearn.metrics import precision_recall_curve
y_true = np.array([0,0,1,0,1,1,0,1,0,0])
y_score = np.array([.1,.3,.8,.4,.6,.9,.2,.55,.35,.05])
p, r, thr = precision_recall_curve(y_true, y_score)
for pi, ri, ti in zip(p[:-1], r[:-1], thr):
print(f"threshold {ti:.2f} precision {pi:.2f} recall {ri:.2f}")
# Pick the row with the recall operations can live with,
# not the row with the highest F1.- Write down the cost of each error type before setting the threshold.
- Choose the threshold on validation, never on the test set.
- If the class balance shifts in production, re-set the threshold rather than retraining.
The objective
The ceiling imposed by label noise
With a label error rate ε and a model that learns the true function perfectly,
measured accuracy against the noisy test set is at most 1 − ε. At ε = 0.05
you stop at 95 percent no matter how good the model gets. Measuring above that
means measuring overfit to the noise.
That implies an order of work that is rarely followed: check the labels first, change the model second. A double-blind re-annotation of 200 cases with a second annotator costs a day and answers whether further modelling is worth anything at all. See Annotation and labelling.
Do not forget calibration
Tree methods and boosting produce scores, not probabilities. If the output feeds a cost calculation, it must be calibrated, Platt scaling or isotonic regression on a separate slice of the validation data. Without it, any expected-value calculation on top has no basis.
Related courses and sources
Machine Learning Specialization
Andrew Ng's course in its reworked form. Regression, classification, neural networks and the mistakes that actually happen in practice. The maths is included.
Free to audit; the certificate costs. If you only want to understand it, you do not need one.