Cross-validation and data splits
How to split data without fooling yourself: k-fold, grouped and time-based splits, and the test set you open exactly once.
The idea
Sitting an exam made of exactly the questions you practised tells you nothing about your ability. So you hold back part of the data and do not touch it for the whole of development.
The three parts
| Part | Purpose | How often touched |
|---|---|---|
| Training | Fit the model | constantly |
| Validation | Choose hyperparameters and thresholds | often |
| Test | Evidence the final number | exactly once |
Which split for which data
| Data situation | Method | Why |
|---|---|---|
| Independent cases | KFold, StratifiedKFold | Standard |
| Several cases per person or customer | GroupKFold | Otherwise the same person is in both parts |
| Time series | TimeSeriesSplit | The future must not explain the past |
| Very little data | RepeatedStratifiedKFold | Steadier estimate |
| Very rare class | Stratified, plus bootstrap | Otherwise some folds contain zero positives |
import numpy as np
from sklearn.model_selection import GroupKFold, cross_val_score
from sklearn.linear_model import LogisticRegression
from sklearn.datasets import make_classification
X, y = make_classification(n_samples=900, n_features=12, random_state=0)
customer = np.repeat(np.arange(90), 10) # 10 cases per customer
random_cv = cross_val_score(LogisticRegression(max_iter=2000), X, y, cv=5)
grouped_cv = cross_val_score(LogisticRegression(max_iter=2000), X, y,
cv=GroupKFold(5), groups=customer)
print("random ", random_cv.mean().round(3))
print("grouped", grouped_cv.mean().round(3))
# The second number is the honest one. The gap is the extent of the leakage.- Scaling, imputation and feature selection belong inside the pipeline, not before the split.
- For time series, always validate forward, never shuffle.
- Report the spread across folds, not just the mean.
- Separate the test set before the first line of modelling code and lock it away.
The estimator
The trade-off: small k means smaller training sets and therefore an upward-
biased error estimate; large k means less bias but strongly correlated models
and therefore an understated spread. Five to ten folds is the usual choice
because both effects are small there.
Nested cross-validation
Choosing hyperparameters with the same cross-validation that estimates quality
gives an optimistic number. The clean construction has two loops: the inner one
picks hyperparameters, the outer one estimates the quality of the whole selection
procedure. The cost is k_outer × k_inner training runs, 25 instead of 5 at
5 × 5.
For large models that is unaffordable. There you substitute a fixed three-way split with a held-out test set and document how many configurations were tried on validation. That number belongs in the report, because it quantifies the extent of multiple testing.
Time splits and lead time
For forecasts with a lead time, a gap of exactly that length belongs between the training and validation parts. Predicting demand 14 days ahead while training up to one day before the target measures a quality that is unreachable in production.
Related courses and sources
Model evaluation in scikit-learn
The complete overview of scoring metrics with their pitfalls. The shortest answer to why accuracy is usually the wrong number.
The shortest answer to why accuracy is usually the wrong number.
The Elements of Statistical Learning
The statistical view of machine learning, the reference on bias, variance and model selection for twenty years. Demanding, free as a PDF.
For readers with a statistics background; without one the entry is hard going.