Data leakage in training
When information from the future or from the test set reaches training: the common forms, their signs, and how to rule them out.
The idea
A leak is any information available to the model during training but not in real use. The model exploits it, looks superb, and fails later.
The three forms
| Form | Example |
|---|---|
| Target leak | A feature created only after the decision, such as a cancellation date in a churn model |
| Time leak | A value from the future, such as a monthly average in a daily forecast |
| Group leak | The same customer in training and test, with nearly identical cases |
The checklist
- Ask of every feature: was this value available at prediction time?
- Remove identifiers, timestamps and sequence numbers from features; they often encode order and thereby the target.
- Put scaling, imputation and encoding into the pipeline so they learn per fold from the training part only.
- With several cases per person, split by group.
- For time series, validate strictly forward, with a gap the size of the lead time.
- Deduplicate before splitting, not after.
Tracking down a leak
import numpy as np
from sklearn.ensemble import HistGradientBoostingClassifier
from sklearn.model_selection import cross_val_score
def suspect_features(X, y, names, cv=5):
"""Test each feature alone as the sole predictor.
A single feature with a very high AUC is nearly always a leak."""
for i, name in enumerate(names):
s = cross_val_score(HistGradientBoostingClassifier(max_iter=60),
X[:, [i]], y, cv=cv, scoring="roc_auc").mean()
if s > 0.90:
print(f"SUSPECT {name}: AUC {s:.3f} alone")A single feature reaching an AUC above 0.9 on its own is, for a real business problem, almost always a disguised copy of the target.
The time leak, precisely
The error almost never lies in the formula but in the collection: a database field often carries only the last-modified timestamp, not the creation time. A status field reading "closed" today says nothing about what it contained six months ago. Without historisation a time leak is practically unavoidable.
Benchmark contamination
With large language models a particular form appears: public test data was in pre-training. The usual checks:
- N-gram overlap. Check whether longer word sequences from the test set appear in the training material.
- Canary strings. Unique markers in test data that must not appear in the model.
- Date cut. Use only tasks created after the training cut-off.
- Rephrased versions. Pose the same task reworded. A marked drop indicates memorisation.
The last point is the most informative and the cheapest. A model that solves a known task and fails the same task with different numbers has not understood it but seen it. See Reading benchmarks.
What to do when a leak is found
- 01
Do not fix it quietly
Every number reported so far is invalid. That has to be said.
- 02
Document the cause
Do not just remove the feature; record how it got in.
- 03
Add the check to the pipeline
So the same form does not recur.
- 04
Re-evaluate
With the original test set, unless it is affected too.