When is machine learning worth it and when is it not?

It is worth it when the rule is too complex to write down but enough examples exist. With clear rules and few cases, ordinary programming is cheaper and more reliable.

How many examples do I need?

Often a few hundred per class for simple classification, considerably more for language and images. What matters is not volume alone but how well the examples cover the reality that follows.

Is a language model machine learning?

Yes. It is supervised learning with a trick: the target is already in the text, namely the next word. That is why it needs no hand-assigned labels.

What machine learning is

What this is about

To teach a program to tell an invoice from a delivery note there are two routes. The classical one: somebody writes rules. "If the phrase invoice number appears, then …". The second: show the program ten thousand invoices and ten thousand delivery notes and let it work out for itself where the difference sits.

The second route is machine learning. It pays off whenever the rules have too many exceptions to write down.

The three kinds

Kind	What you have	Example
Supervised	Examples with the right answer	Invoice or delivery note, labelled
Unsupervised	Examples without answers	Finding customer groups without knowing which
Reinforcement	Reward for good decisions	Robot arm, game strategy, control systems

The sequence that never changes

01
Define the question
What exactly is to be predicted, and how will you know the answer was right? Settling this before the data saves the most time.
02
Split the data
Training, validation, test. The test set is touched exactly once, at the very end. See Cross-validation.
03
Establish a baseline
How good is the simplest conceivable rule? Without that reference no progress can later be claimed.
04
Fit and evaluate
Only now come model choice and hyperparameters, in that order and not the other way round.

When ordinary programming is better

The rule is known and stable, such as a statutory deadline.
There are fewer than a few hundred examples.
Every individual case must be justifiable and an approximation will not do.
The failure case is expensive and rare. Then you check everything instead.

What a model formally does

The learning problem

f* = argmin_f R(f), R(f) = E₍ₓ,ᵧ₎∼𝓓 [ L(f(x), y) ]

You are looking for the function with the smallest average error on data you have never seen.

𝓓: the unknown distribution the data comes from
f: the function sought, from a hypothesis class
L: the loss function
R: expected risk: the average error on unseen data

The problem: 𝓓 is unknown. You minimise empirical risk on the sample instead, and the rest of the discipline is about the gap between the two, see Overfitting, bias and variance.

A complete example

from sklearn.datasets import load_breast_cancer
from sklearn.model_selection import train_test_split
from sklearn.linear_model import LogisticRegression
from sklearn.metrics import classification_report

X, y = load_breast_cancer(return_X_y=True)
Xtr, Xte, ytr, yte = train_test_split(X, y, test_size=0.25,
                                      stratify=y, random_state=0)

# stratify=y keeps the class balance identical in both parts; without it
# the measured quality moves purely because of the split.
model = LogisticRegression(max_iter=5000).fit(Xtr, ytr)
print(classification_report(yte, model.predict(Xte), digits=3))

The mistake that costs most

Building a model before the success measure is fixed. With no metric and no baseline agreed in advance, what gets searched for at the end is the metric on which the model looks good. That is no longer an evaluation but a narrative.

What this is about#

The three kinds#

The sequence that never changes#

When ordinary programming is better#

What a model formally does#

A complete example#

The mistake that costs most#