A1 · Supervised Machine Learning in Depth

Cross-Validation Strategies

Cross-validation estimates performance by repeatedly splitting the data into training and validation parts and averaging the scores. k-fold uses k equal folds; stratified k-fold keeps class proportions in each fold; group k-fold keeps all rows of a group (such as a patient) in the same fold; time-series split always validates on data later than the training data; nested cross-validation separates tuning from evaluation.

10 min Advanced Jump to the activity

What you need: The earlier topics of this module, and its prerequisites.

Definition

Why trust one train-test split when a different random split could change the score?

Cross-validation estimates performance by repeatedly splitting the data into training and validation parts and averaging the scores. k-fold uses k equal folds; stratified k-fold keeps class proportions in each fold; group k-fold keeps all rows of a group (such as a patient) in the same fold; time-series split always validates on data later than the training data; nested cross-validation separates tuning from evaluation.

In simple words

With 5-fold cross-validation, every row is used for validation exactly once and for training four times. The average of five scores is more reliable than one, and their spread tells you how much luck was involved. The right way to split depends on how the model will be used.

Why it matters

The wrong strategy leaks information and overstates performance: random splits on time series, or patients split across folds. Choosing the strategy is part of designing an honest experiment.

Key terms

k-fold cross-validation
Splitting data into k folds, training on k − 1 and validating on the remaining one, k times.
Stratified k-fold
Folds that keep the same class proportions as the whole data set.
Group k-fold
Folds that keep all rows of one group, such as a patient, together.
Time-series split
Expanding or rolling windows that always validate on later data than training.
Leave-one-out
k equal to the number of rows; nearly unbiased but expensive and high-variance.
Repeated cross-validation
Running k-fold several times with different shuffles to reduce the noise in the estimate.

Explained in detail

Why one split is not enough

A single train-validation split gives a noisy estimate: with a small data set, a lucky or unlucky split can change accuracy by several percentage points. k-fold cross-validation uses every row for validation exactly once. With k = 5, the data is divided into five folds; the model is trained five times, each time on four folds and evaluated on the fifth, and the five scores are averaged. The spread of the five scores is as informative as the mean: scores of 0.81, 0.82, 0.80, 0.83 and 0.81 suggest a stable model, while 0.70, 0.88, 0.75, 0.91 and 0.79 warn that results depend heavily on which rows are used. Typical choices are k = 5 or 10; larger k reduces bias but costs more computation.

Matching the strategy to the data

DataStrategyscikit-learn
Independent rows, balanced classesShuffled k-foldKFold(shuffle=True)
Imbalanced classesStratified k-foldStratifiedKFold
Repeated measurements per person or siteGroup k-foldGroupKFold, StratifiedGroupKFold
Ordered in timeForward-chaining time split, often with a gapTimeSeriesSplit(gap=…)
Very small data setRepeated stratified k-fold or leave-one-outRepeatedStratifiedKFold, LeaveOneOut

Cross-validation is used for two different purposes that should not be confused. Model selection uses it to compare settings and pick the best. Performance estimation uses it to predict how well the chosen model will do on new data. If you use the same folds for both, the reported score is optimistically biased, because the best configuration was chosen partly by luck on those folds. Nested cross-validation solves this with an outer loop for estimation and an inner loop for selection. In time series, add a gap between training and validation periods when features use recent history, so information near the boundary cannot leak.

Remember to place all preprocessing inside a pipeline so it is refitted on each training fold, to fix the random seed for reproducibility, and to report both the mean and the standard deviation of the fold scores.

How it works

StrategyUse whenscikit-learn
k-foldIndependent rows, regressionKFold
Stratified k-foldClassification, especially imbalancedStratifiedKFold
Group k-foldSeveral rows per person, device or storeGroupKFold
Time-series splitForecasting; order mattersTimeSeriesSplit
Nested CVHonest estimate after tuningGridSearchCV inside cross_val_score

The maths

The cross-validation estimate and its spread

In code

Which rows each strategy puts in validation, on a tiny dataset, and a nested cross-validation estimate.

cv_strategies.py
import numpy as np
from sklearn.datasets import load_breast_cancer
from sklearn.model_selection import GridSearchCV, GroupKFold, KFold, StratifiedKFold, TimeSeriesSplit, cross_val_score
from sklearn.pipeline import make_pipeline
from sklearn.preprocessing import StandardScaler
from sklearn.svm import SVC

X = np.arange(12).reshape(-1, 1)
y = np.array([0, 0, 0, 0, 0, 0, 0, 0, 1, 1, 1, 1])
groups = np.repeat(["A", "B", "C", "D"], 3)
for name, splitter in [("KFold", KFold(3)), ("Stratified", StratifiedKFold(3)), ("Group", GroupKFold(4)), ("TimeSeries", TimeSeriesSplit(3))]:
    folds = [val.tolist() for _, val in splitter.split(X, y, groups)]
    print(f"{name:>10} validation rows: {folds}")

Xb, yb = load_breast_cancer(return_X_y=True)
inner = GridSearchCV(make_pipeline(StandardScaler(), SVC()), {"svc__C": [0.1, 1, 10]}, cv=3)
scores = cross_val_score(inner, Xb, yb, cv=StratifiedKFold(5, shuffle=True, random_state=0))
print(f"Nested CV accuracy: {scores.mean():.3f} +/- {scores.std() / np.sqrt(len(scores)):.3f}")

Output

     KFold validation rows: [[0, 1, 2, 3], [4, 5, 6, 7], [8, 9, 10, 11]]
Stratified validation rows: [[0, 1, 2, 8], [3, 4, 5, 9], [6, 7, 10, 11]]
     Group validation rows: [[9, 10, 11], [6, 7, 8], [3, 4, 5], [0, 1, 2]]
TimeSeries validation rows: [[3, 4, 5], [6, 7, 8], [9, 10, 11]]
Nested CV accuracy: 0.977 +/- 0.003

Try it yourself

Which strategy?

Match each problem to the cross-validation strategy it needs.

  • Forecast next week's sales from daily history
  • Detect a disease with 5 per cent positives
  • Ten scans per patient, predicting new patients
  • Report an honest score after tuning hyperparameters

Common mistakes

  • Shuffling time-series data before splitting.
  • Reporting the best inner-loop score of a grid search as the final estimate.
  • Ignoring the spread of fold scores.

Check yourself

Question 1

1. In 5-fold CV, how often is each row used for validation?

Question 2

2. Why use GroupKFold for patients?

Question 3

3. What does nested cross-validation protect against?

Summary

  • Cross-validation averages scores over several splits.
  • Match the strategy to the data: stratified, grouped or time-ordered.
  • Use nested CV for an honest score after tuning.

Read it. Now try it yourself.

Open the interactive activity

Finished reading?

Sources: scikit-learn: Cross-validation · James et al., An Introduction to Statistical Learning