Definition
Why trust one train-test split when a different random split could change the score?
Cross-validation estimates performance by repeatedly splitting the data into training and validation parts and averaging the scores. k-fold uses k equal folds; stratified k-fold keeps class proportions in each fold; group k-fold keeps all rows of a group (such as a patient) in the same fold; time-series split always validates on data later than the training data; nested cross-validation separates tuning from evaluation.
In simple words
With 5-fold cross-validation, every row is used for validation exactly once and for training four times. The average of five scores is more reliable than one, and their spread tells you how much luck was involved. The right way to split depends on how the model will be used.
Why it matters
The wrong strategy leaks information and overstates performance: random splits on time series, or patients split across folds. Choosing the strategy is part of designing an honest experiment.
Key terms
- k-fold cross-validation
- Splitting data into k folds, training on k − 1 and validating on the remaining one, k times.
- Stratified k-fold
- Folds that keep the same class proportions as the whole data set.
- Group k-fold
- Folds that keep all rows of one group, such as a patient, together.
- Time-series split
- Expanding or rolling windows that always validate on later data than training.
- Leave-one-out
- k equal to the number of rows; nearly unbiased but expensive and high-variance.
- Repeated cross-validation
- Running k-fold several times with different shuffles to reduce the noise in the estimate.
Explained in detail
Why one split is not enough
A single train-validation split gives a noisy estimate: with a small data set, a lucky or unlucky split can change accuracy by several percentage points. k-fold cross-validation uses every row for validation exactly once. With k = 5, the data is divided into five folds; the model is trained five times, each time on four folds and evaluated on the fifth, and the five scores are averaged. The spread of the five scores is as informative as the mean: scores of 0.81, 0.82, 0.80, 0.83 and 0.81 suggest a stable model, while 0.70, 0.88, 0.75, 0.91 and 0.79 warn that results depend heavily on which rows are used. Typical choices are k = 5 or 10; larger k reduces bias but costs more computation.
Matching the strategy to the data
| Data | Strategy | scikit-learn |
|---|---|---|
| Independent rows, balanced classes | Shuffled k-fold | KFold(shuffle=True) |
| Imbalanced classes | Stratified k-fold | StratifiedKFold |
| Repeated measurements per person or site | Group k-fold | GroupKFold, StratifiedGroupKFold |
| Ordered in time | Forward-chaining time split, often with a gap | TimeSeriesSplit(gap=…) |
| Very small data set | Repeated stratified k-fold or leave-one-out | RepeatedStratifiedKFold, LeaveOneOut |
Cross-validation is used for two different purposes that should not be confused. Model selection uses it to compare settings and pick the best. Performance estimation uses it to predict how well the chosen model will do on new data. If you use the same folds for both, the reported score is optimistically biased, because the best configuration was chosen partly by luck on those folds. Nested cross-validation solves this with an outer loop for estimation and an inner loop for selection. In time series, add a gap between training and validation periods when features use recent history, so information near the boundary cannot leak.
Remember to place all preprocessing inside a pipeline so it is refitted on each training fold, to fix the random seed for reproducibility, and to report both the mean and the standard deviation of the fold scores.
How it works
| Strategy | Use when | scikit-learn |
|---|---|---|
| k-fold | Independent rows, regression | KFold |
| Stratified k-fold | Classification, especially imbalanced | StratifiedKFold |
| Group k-fold | Several rows per person, device or store | GroupKFold |
| Time-series split | Forecasting; order matters | TimeSeriesSplit |
| Nested CV | Honest estimate after tuning | GridSearchCV inside cross_val_score |
The maths
In code
Which rows each strategy puts in validation, on a tiny dataset, and a nested cross-validation estimate.
import numpy as np
from sklearn.datasets import load_breast_cancer
from sklearn.model_selection import GridSearchCV, GroupKFold, KFold, StratifiedKFold, TimeSeriesSplit, cross_val_score
from sklearn.pipeline import make_pipeline
from sklearn.preprocessing import StandardScaler
from sklearn.svm import SVC
X = np.arange(12).reshape(-1, 1)
y = np.array([0, 0, 0, 0, 0, 0, 0, 0, 1, 1, 1, 1])
groups = np.repeat(["A", "B", "C", "D"], 3)
for name, splitter in [("KFold", KFold(3)), ("Stratified", StratifiedKFold(3)), ("Group", GroupKFold(4)), ("TimeSeries", TimeSeriesSplit(3))]:
folds = [val.tolist() for _, val in splitter.split(X, y, groups)]
print(f"{name:>10} validation rows: {folds}")
Xb, yb = load_breast_cancer(return_X_y=True)
inner = GridSearchCV(make_pipeline(StandardScaler(), SVC()), {"svc__C": [0.1, 1, 10]}, cv=3)
scores = cross_val_score(inner, Xb, yb, cv=StratifiedKFold(5, shuffle=True, random_state=0))
print(f"Nested CV accuracy: {scores.mean():.3f} +/- {scores.std() / np.sqrt(len(scores)):.3f}")Output
KFold validation rows: [[0, 1, 2, 3], [4, 5, 6, 7], [8, 9, 10, 11]]
Stratified validation rows: [[0, 1, 2, 8], [3, 4, 5, 9], [6, 7, 10, 11]]
Group validation rows: [[9, 10, 11], [6, 7, 8], [3, 4, 5], [0, 1, 2]]
TimeSeries validation rows: [[3, 4, 5], [6, 7, 8], [9, 10, 11]]
Nested CV accuracy: 0.977 +/- 0.003Try it yourself
Which strategy?
Match each problem to the cross-validation strategy it needs.
- Forecast next week's sales from daily history
- Detect a disease with 5 per cent positives
- Ten scans per patient, predicting new patients
- Report an honest score after tuning hyperparameters
Common mistakes
- Shuffling time-series data before splitting.
- Reporting the best inner-loop score of a grid search as the final estimate.
- Ignoring the spread of fold scores.
Check yourself
Summary
- Cross-validation averages scores over several splits.
- Match the strategy to the data: stratified, grouped or time-ordered.
- Use nested CV for an honest score after tuning.
Read it. Now try it yourself.
Open the interactive activityFinished reading?
Sources: scikit-learn: Cross-validation · James et al., An Introduction to Statistical Learning