MLOps
What lies between a working notebook and dependable operation: versioning, monitoring, drift, rollback and evidence.
The idea
Training a model is an experiment. Running one is a commitment: that it will do the same thing tomorrow, that someone is responsible, and that you can show what it did and when.
The four questions
| Question | Answer in operation |
|---|---|
| Which model is running right now? | Model registry with version and checksum |
| What was it built from? | Data, code and configuration state |
| How good is it today? | Continuous measurement against a fixed set |
| What do we do when it worsens? | Trigger, rollback, ownership |
Detecting drift
| Type | What changes | How measurable |
|---|---|---|
| Data drift | The inputs | Distribution comparison per feature |
| Concept drift | The relationship | Only with delayed labels |
| Label drift | The class balance | Proportions over time |
| Prediction drift | The outputs | Distribution of model outputs |
The fourth row is the only one available immediately and therefore the practical entry point: a shift in the output distribution is an early warning long before labels arrive.
import numpy as np
def psi(expected, observed, bins=10):
"""Population stability index. Above 0.2 counts as a clear shift."""
edges = np.percentile(expected, np.linspace(0, 100, bins + 1))
edges[0], edges[-1] = -np.inf, np.inf
e = np.histogram(expected, bins=edges)[0] / len(expected)
o = np.histogram(observed, bins=edges)[0] / len(observed)
e, o = np.clip(e, 1e-6, None), np.clip(o, 1e-6, None)
return float(((o - e) * np.log(o / e)).sum())- PSI below 0.1: unremarkable. 0.1 to 0.2: watch. Above 0.2: investigate.
- Monitor not only features but the share of missing values per column.
- Run a shadow deployment before a new model goes live.
What makes a run reproducible
- Data: a snapshot with a checksum, not just a path. A directory changes.
- Code: a commit identifier, not a branch name.
- Environment: the container image digest, not a package list.
- Configuration: complete, including seeds.
- Result: model checksum and metrics on a fixed evaluation set.
Those five items are simultaneously what Annex IV of the AI Act requires as technical documentation for high-risk systems. Keeping them anyway discharges most of that obligation. See Preparing for audit.
The rollback
A rollback is only an option once three conditions hold:
- 01
The previous version is available
Not only the weights but the complete serving environment.
- 02
The interface is compatible
Rollbacks routinely fail because the output format changed and downstream systems expect the new one.
- 03
It has been rehearsed
With a measured duration. An unrehearsed rollback takes about three times the estimate when it matters.
The metrics that actually get read
| Metric | Why |
|---|---|
| Share of refused or escalated cases | Early warning, available immediately |
| Time to first token, 95th percentile | User experience, not the mean |
| Share of answers with a citation | For RAG the decisive figure |
| Human correction rate | The most honest quality figure there is |
| Cost per completed task | The only number the business compares |
The fourth row is the most valuable and the least often collected: if a human is reviewing the result anyway, their correction is a free label. Systematically collecting those corrections yields, within six months, a dataset no vendor can supply.