Definition
You split your data once and your model scores 91 per cent. Split it differently and it scores 84. Which number is true?
Cross-validation estimates how well a model will generalise by training and testing it several times on different splits of the data. In k-fold cross-validation the data is divided into k roughly equal folds in ordinary KFold; each fold takes one turn as the validation set while the other k − 1 folds train the model, and the k scores are averaged.
In simple words
Instead of judging a cricketer on one match, you average their score over five matches against different teams. Cross-validation gives a model five "matches" with different test data and averages the results, so one lucky or unlucky split does not decide everything.
Overview
- Cross-validation estimates performance by training and validating on several different splits of the training data.
- In k-fold cross-validation, each of k folds serves once as the validation set.
- The mean score is more reliable than a single split, and the spread shows how stable it is.
- Preprocessing must happen inside each fold, using a Pipeline, to avoid leakage.
- A separate test set is still kept for the final check.
- Typical values are k = 5 or 10.
Why it matters
A single split, especially on small data, gives a noisy estimate. Cross-validation gives a more reliable score and its spread, uses all the data for both training and validation, and is the standard way to compare models and choose hyperparameters.
Key terms
- Fold
- One of the k equal parts the data is divided into.
- k-fold cross-validation
- Training k times, each time validating on a different fold.
- Stratified k-fold
- k-fold that keeps class proportions the same in every fold.
- Leave-one-out
- Cross-validation with k equal to the number of examples.
- Grid search
- Trying every combination of hyperparameter values, scored by cross-validation.
- Nested cross-validation
- An outer loop to estimate performance and an inner loop to tune hyperparameters.
Types
There are several types of cross-validation, each suited to a different data structure.
1. k-fold cross-validation
Splitting data into k equal folds and rotating which one is used for validation.
- How it works: Train on k − 1 folds and validate on the remaining one, k times; cross_val_score returns the k scores.
- Example: 5-fold cross-validation on 1,000 rows trains five models on 800 rows each.
- When to use it: Use it as the default for regression and balanced classification.
2. Stratified k-fold
k-fold that keeps class proportions the same in every fold.
- How it works: StratifiedKFold samples within each class; scikit-learn uses it by default for classifiers.
- Example: Each fold keeps about 2% fraud cases.
- When to use it: Use it for classification, especially imbalanced or small data. Without it, a fold could contain no fraud at all, making its recall impossible to compute and the overall estimate unreliable.
3. Leave-one-out and repeated cross-validation
Leave-one-out uses one example as validation each time; repeated CV runs k-fold several times with different shuffles.
- How it works: Leave-one-out trains n models and is costly; repeated k-fold averages out the luck of one particular split.
- Example: Leave-one-out on 40 patient records; 5 × 10-fold repeated CV for a reliable comparison.
- When to use it: Use leave-one-out for very small data sets and repeated CV when comparing close models.
4. Group k-fold
Cross-validation that keeps all rows of a group in the same fold.
- How it works: GroupKFold uses a group label, such as patient ID, so validation groups are never seen in training.
- Example: Several photos of the same plant must not be split between training and validation.
- When to use it: Use it whenever rows from one source are related.
5. Time-series cross-validation
Rolling-origin validation that always trains on the past and validates on the future.
- How it works: TimeSeriesSplit grows the training window forward fold by fold; no future rows ever enter training.
- Example: Training on 2021–2022 and validating on 2023, then training on 2021–2023 and validating on 2024.
- When to use it: Use it for forecasting and any data where order matters.
6. Nested cross-validation
An outer loop that estimates performance and an inner loop that tunes hyperparameters.
- How it works: Tuning inside each outer fold prevents the optimistic bias of reporting the best tuned score.
- Example: GridSearchCV inside cross_val_score.
- When to use it: Use it for honest performance estimates when tuning on small data sets. It costs more training but gives a fairer number.
Explained in detail
How 5-fold works
| Round | Training folds | Validation fold | Score |
|---|---|---|---|
| 1 | 2, 3, 4, 5 | 1 | 0.88 |
| 2 | 1, 3, 4, 5 | 2 | 0.91 |
| 3 | 1, 2, 4, 5 | 3 | 0.85 |
| 4 | 1, 2, 3, 5 | 4 | 0.89 |
| 5 | 1, 2, 3, 4 | 5 | 0.87 |
Every example is used for validation exactly once and for training four times. The mean score here is 0.88 with a standard deviation of about 0.02. Reporting "0.88 ± 0.02" says much more than a single number: if another model scores 0.89 ± 0.03, the two are effectively tied. Five or ten folds are the usual choice; more folds give more training data per round but take longer.
Choosing the right kind
- Stratified k-fold for classification, so a rare class appears in every fold; scikit-learn uses it by default for classifiers.
- Group k-fold when several rows come from one person, patient or device, so they never appear in both training and validation.
- Time-series split for forecasting: always train on the past and validate on the future, never shuffle.
- Leave-one-out for very small data sets, at the cost of many training runs.
Tuning hyperparameters honestly
Grid search tries every combination of chosen hyperparameter values, for example k in 3, 5, 7 and weights uniform or distance for k-NN, and scores each by cross-validation. Randomised search samples combinations instead, which is faster when there are many settings. Once the best settings are chosen, the cross-validation score used to choose them is slightly optimistic, because you picked the winner. For a fair final number, evaluate the chosen model on a separate test set, or use nested cross-validation, where an inner loop tunes and an outer loop estimates performance.
Cross-validation costs time: 10-fold means training ten models. For large data sets or slow models, a single well-sized validation set is often enough, while for small data sets, where every example is precious, cross-validation is almost always worth it. A practical compromise is to use 5 folds while exploring and reserve heavier methods for the final comparison.
How it works
Choose the split that matches the prediction you want to make. If the same reader borrowed ten books and your claim concerns completely new readers, keep all loans from each reader in one fold. For next month's demand, validate on later dates and train on earlier ones. In both designs learn the imputation value and scaling parameters inside each training fold. Fitting them once to the whole table lets the held-out rows influence their own evaluation.
- Keep a final test set aside.
- Build a Pipeline of preprocessing and model.
- Run cross_val_score with 5 or 10 folds on the remaining data.
- Report the mean and standard deviation.
- Tune with GridSearchCV, then check once on the test set.
In code
from sklearn.datasets import load_breast_cancer
from sklearn.linear_model import LogisticRegression
from sklearn.model_selection import GridSearchCV, cross_val_score, train_test_split
from sklearn.pipeline import make_pipeline
from sklearn.preprocessing import StandardScaler
X, y = load_breast_cancer(return_X_y=True)
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, stratify=y, random_state=0)
pipe = make_pipeline(StandardScaler(), LogisticRegression(max_iter=1000))
scores = cross_val_score(pipe, X_train, y_train, cv=5)
print("Fold scores:", scores.round(3))
print(f"Mean {scores.mean():.3f} +/- {scores.std():.3f}")
grid = GridSearchCV(pipe, {"logisticregression__C": [0.01, 0.1, 1, 10]}, cv=5).fit(X_train, y_train)
print("Best C:", grid.best_params_["logisticregression__C"], " CV score:", round(grid.best_score_, 3))
print("Test score (used once):", round(grid.score(X_test, y_test), 3))Output
Fold scores: [0.956 0.967 0.989 0.956 0.978] Mean 0.969 +/- 0.013 Best C: 0.1 CV score: 0.971 Test score (used once): 0.965
Try it yourself
The lucky split
Two students compare a k-NN model and a decision tree on 300 rows of data.
Which split strategy?
Choose the right cross-validation for each problem.
Forecasting next month's electricity demand
Detecting a rare disease that affects 3 per cent of patients
Recordings from many users of a voice app
Several X-rays from each patient
Common mistakes
- Trusting one train-test split on a small data set.
- Shuffling time-series data before splitting.
- Reporting the tuned cross-validation score as the final, unbiased result.
Check yourself
Summary
- k-fold cross-validation trains and validates k times and averages the scores.
- Use stratified, group or time-series splits to match the data.
- Tune with cross-validation, then confirm once on an untouched test set.
Finished reading?
Sources: scikit-learn: Cross-validation · An Introduction to Statistical Learning (James, Witten, Hastie, Tibshirani)