I5 · Machine Learning Foundations

Cross-Validation

Cross-validation estimates how well a model will generalise by training and testing it several times on different splits of the data. In k-fold cross-validation the data is divided into k roughly equal folds in ordinary KFold; each fold takes one turn as the validation set while the other k − 1 folds train the model, and the k scores are averaged.

8 min Intermediate · Sublevel Intermediate Jump to the activity

What you need: The earlier topics of this module.

Definition

You split your data once and your model scores 91 per cent. Split it differently and it scores 84. Which number is true?

Cross-validation estimates how well a model will generalise by training and testing it several times on different splits of the data. In k-fold cross-validation the data is divided into k roughly equal folds in ordinary KFold; each fold takes one turn as the validation set while the other k − 1 folds train the model, and the k scores are averaged.

In simple words

Instead of judging a cricketer on one match, you average their score over five matches against different teams. Cross-validation gives a model five "matches" with different test data and averages the results, so one lucky or unlucky split does not decide everything.

Overview

  • Cross-validation estimates performance by training and validating on several different splits of the training data.
  • In k-fold cross-validation, each of k folds serves once as the validation set.
  • The mean score is more reliable than a single split, and the spread shows how stable it is.
  • Preprocessing must happen inside each fold, using a Pipeline, to avoid leakage.
  • A separate test set is still kept for the final check.
  • Typical values are k = 5 or 10.

Why it matters

A single split, especially on small data, gives a noisy estimate. Cross-validation gives a more reliable score and its spread, uses all the data for both training and validation, and is the standard way to compare models and choose hyperparameters.

Key terms

Fold
One of the k equal parts the data is divided into.
k-fold cross-validation
Training k times, each time validating on a different fold.
Stratified k-fold
k-fold that keeps class proportions the same in every fold.
Leave-one-out
Cross-validation with k equal to the number of examples.
Grid search
Trying every combination of hyperparameter values, scored by cross-validation.
Nested cross-validation
An outer loop to estimate performance and an inner loop to tune hyperparameters.

Types

There are several types of cross-validation, each suited to a different data structure.

1. k-fold cross-validation

Splitting data into k equal folds and rotating which one is used for validation.

  • How it works: Train on k − 1 folds and validate on the remaining one, k times; cross_val_score returns the k scores.
  • Example: 5-fold cross-validation on 1,000 rows trains five models on 800 rows each.
  • When to use it: Use it as the default for regression and balanced classification.

2. Stratified k-fold

k-fold that keeps class proportions the same in every fold.

  • How it works: StratifiedKFold samples within each class; scikit-learn uses it by default for classifiers.
  • Example: Each fold keeps about 2% fraud cases.
  • When to use it: Use it for classification, especially imbalanced or small data. Without it, a fold could contain no fraud at all, making its recall impossible to compute and the overall estimate unreliable.

3. Leave-one-out and repeated cross-validation

Leave-one-out uses one example as validation each time; repeated CV runs k-fold several times with different shuffles.

  • How it works: Leave-one-out trains n models and is costly; repeated k-fold averages out the luck of one particular split.
  • Example: Leave-one-out on 40 patient records; 5 × 10-fold repeated CV for a reliable comparison.
  • When to use it: Use leave-one-out for very small data sets and repeated CV when comparing close models.

4. Group k-fold

Cross-validation that keeps all rows of a group in the same fold.

  • How it works: GroupKFold uses a group label, such as patient ID, so validation groups are never seen in training.
  • Example: Several photos of the same plant must not be split between training and validation.
  • When to use it: Use it whenever rows from one source are related.

5. Time-series cross-validation

Rolling-origin validation that always trains on the past and validates on the future.

  • How it works: TimeSeriesSplit grows the training window forward fold by fold; no future rows ever enter training.
  • Example: Training on 2021–2022 and validating on 2023, then training on 2021–2023 and validating on 2024.
  • When to use it: Use it for forecasting and any data where order matters.

6. Nested cross-validation

An outer loop that estimates performance and an inner loop that tunes hyperparameters.

  • How it works: Tuning inside each outer fold prevents the optimistic bias of reporting the best tuned score.
  • Example: GridSearchCV inside cross_val_score.
  • When to use it: Use it for honest performance estimates when tuning on small data sets. It costs more training but gives a fairer number.

Explained in detail

How 5-fold works

RoundTraining foldsValidation foldScore
12, 3, 4, 510.88
21, 3, 4, 520.91
31, 2, 4, 530.85
41, 2, 3, 540.89
51, 2, 3, 450.87

Every example is used for validation exactly once and for training four times. The mean score here is 0.88 with a standard deviation of about 0.02. Reporting "0.88 ± 0.02" says much more than a single number: if another model scores 0.89 ± 0.03, the two are effectively tied. Five or ten folds are the usual choice; more folds give more training data per round but take longer.

Choosing the right kind

  • Stratified k-fold for classification, so a rare class appears in every fold; scikit-learn uses it by default for classifiers.
  • Group k-fold when several rows come from one person, patient or device, so they never appear in both training and validation.
  • Time-series split for forecasting: always train on the past and validate on the future, never shuffle.
  • Leave-one-out for very small data sets, at the cost of many training runs.

Tuning hyperparameters honestly

Grid search tries every combination of chosen hyperparameter values, for example k in 3, 5, 7 and weights uniform or distance for k-NN, and scores each by cross-validation. Randomised search samples combinations instead, which is faster when there are many settings. Once the best settings are chosen, the cross-validation score used to choose them is slightly optimistic, because you picked the winner. For a fair final number, evaluate the chosen model on a separate test set, or use nested cross-validation, where an inner loop tunes and an outer loop estimates performance.

Cross-validation costs time: 10-fold means training ten models. For large data sets or slow models, a single well-sized validation set is often enough, while for small data sets, where every example is precious, cross-validation is almost always worth it. A practical compromise is to use 5 folds while exploring and reserve heavier methods for the final comparison.

How it works

Choose the split that matches the prediction you want to make. If the same reader borrowed ten books and your claim concerns completely new readers, keep all loans from each reader in one fold. For next month's demand, validate on later dates and train on earlier ones. In both designs learn the imputation value and scaling parameters inside each training fold. Fitting them once to the whole table lets the held-out rows influence their own evaluation.

  1. Keep a final test set aside.
  2. Build a Pipeline of preprocessing and model.
  3. Run cross_val_score with 5 or 10 folds on the remaining data.
  4. Report the mean and standard deviation.
  5. Tune with GridSearchCV, then check once on the test set.

In code

cross_validation.py
from sklearn.datasets import load_breast_cancer
from sklearn.linear_model import LogisticRegression
from sklearn.model_selection import GridSearchCV, cross_val_score, train_test_split
from sklearn.pipeline import make_pipeline
from sklearn.preprocessing import StandardScaler

X, y = load_breast_cancer(return_X_y=True)
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, stratify=y, random_state=0)

pipe = make_pipeline(StandardScaler(), LogisticRegression(max_iter=1000))
scores = cross_val_score(pipe, X_train, y_train, cv=5)
print("Fold scores:", scores.round(3))
print(f"Mean {scores.mean():.3f} +/- {scores.std():.3f}")

grid = GridSearchCV(pipe, {"logisticregression__C": [0.01, 0.1, 1, 10]}, cv=5).fit(X_train, y_train)
print("Best C:", grid.best_params_["logisticregression__C"], " CV score:", round(grid.best_score_, 3))
print("Test score (used once):", round(grid.score(X_test, y_test), 3))

Output

Fold scores: [0.956 0.967 0.989 0.956 0.978]
Mean 0.969 +/- 0.013
Best C: 0.1  CV score: 0.971
Test score (used once): 0.965

Try it yourself

The lucky split

Two students compare a k-NN model and a decision tree on 300 rows of data.

Which split strategy?

Choose the right cross-validation for each problem.

  • Forecasting next month's electricity demand

  • Detecting a rare disease that affects 3 per cent of patients

  • Recordings from many users of a voice app

  • Several X-rays from each patient

Common mistakes

  • Trusting one train-test split on a small data set.
  • Shuffling time-series data before splitting.
  • Reporting the tuned cross-validation score as the final, unbiased result.

Check yourself

Question 1

1. In 5-fold cross-validation, how many times is each example used for validation?

Question 2

2. Why use stratified folds for classification?

Question 3

3. What does a large spread in fold scores suggest?

Summary

  • k-fold cross-validation trains and validates k times and averages the scores.
  • Use stratified, group or time-series splits to match the data.
  • Tune with cross-validation, then confirm once on an untouched test set.

Finished reading?

Sources: scikit-learn: Cross-validation · An Introduction to Statistical Learning (James, Witten, Hastie, Tibshirani)