I5 · Machine Learning Foundations

Training, Validation and Test Sets

Data is split into three sets. The training set teaches the model. The validation set is used to compare models and tune settings. The test set gives a final, honest estimate, and is used once at the end.

10 min Intermediate · Sublevel Intermediate

What you need: The earlier topics of this module.

Definition

Data is split into three sets. The training set teaches the model. The validation set is used to compare models and tune settings. The test set gives a final, honest estimate, and is used once at the end.

In simple words

Textbook problems are the training set. Mock tests are the validation set: you use them to change how you study. The board exam is the test set. If you saw the board paper in advance, your score would mean nothing, and the same holds for a model.

Overview

  • Data is split so that a model is judged on examples it did not learn from.
  • The training set fits parameters; the validation set chooses models and hyperparameters; the test set gives the final estimate.
  • Common splits are 70/15/15 or 80/10/10; large data sets can use smaller fractions for testing.
  • Stratified splits keep class proportions equal across sets.
  • Time series must be split by time, and related rows (such as one patient's records) must stay in the same set.
  • Using the test set repeatedly for decisions slowly turns it into another validation set.

Why it matters

A score on data the model has already seen tells you almost nothing. Without a held-out set, you can ship a model that looks perfect and fails on real users.

Key terms

Training set
The data the model learns its parameters from.
Validation set
Data used to compare models and tune hyperparameters.
Test set
Data held back until the end for a single, honest estimate of performance.
Stratified split
A split that keeps class proportions the same in each set.
Random seed
A number fixing the randomness so a split can be repeated exactly.
Time-based split
Training on earlier data and testing on later data, for problems that involve time.

Types

There are several ways to split data. The right type depends on the data's structure.

1. Random hold-out split

Shuffling the rows and dividing them into training, validation and test sets.

  • How it works: train_test_split(X, y, test_size=0.2, random_state=0) shuffles and splits; calling it twice makes three sets.
  • Example: Splitting 10,000 independent customer records 8,000 / 1,000 / 1,000.
  • When to use it: Use it when rows are independent and the data set is reasonably large.

2. Stratified split

A random split that keeps each class's share the same in every set.

  • How it works: Passing stratify=y samples within each class separately.
  • Example: With 2% fraud, each set keeps about 2% fraud instead of some sets having almost none.
  • When to use it: Use it for classification, especially with imbalanced or small data.

3. Time-based split

Training on earlier data and testing on later data.

  • How it works: Sort by time, cut at a date, and never let future rows enter training; TimeSeriesSplit makes several such folds.
  • Example: Training a sales forecast on 2021–2023 and testing on 2024.
  • When to use it: Use it for any data where order matters: forecasting, finance and logs.

4. Group split

Keeping all rows from the same group, such as a patient, user or school, in one set.

  • How it works: GroupShuffleSplit or GroupKFold use a group column so the model is tested on unseen groups.
  • Example: Several X-rays of one patient must not appear in both training and test sets.
  • When to use it: Use it whenever rows from one source are similar, or scores will be falsely high.

5. Cross-validation instead of a single validation set

Rotating which part of the training data acts as validation.

  • How it works: With k-fold cross-validation, each of k folds serves once as validation and the scores are averaged; the test set is still kept separate.
  • Example: 5-fold cross-validation on 800 training rows uses 640 for training and 160 for validation each time.
  • When to use it: Use it for small and medium data sets, where one validation split would be noisy.

Explained in detail

Why three sets?

If you test a model on the data it trained on, you measure its memory, not its ability to generalise. So you hold some data back. But if you then try twenty models and pick the one that does best on that held-back data, you have used it to make choices, and its score is now optimistically biased. The solution is three sets: train on the training set, choose between models and settings using the validation set, and only at the very end, once, measure the chosen model on the test set. That final number is your honest estimate of real-world performance.

Common split sizes

Data sizeTypical splitNote
A few thousand rows60 / 20 / 20 or 70 / 15 / 15Consider cross-validation instead of a single validation set
Hundreds of thousands80 / 10 / 10Each set is still large
Millions98 / 1 / 1One per cent can be tens of thousands of examples

Splitting correctly

  • Shuffle before splitting, unless order matters, so each set is representative.
  • Stratify for classification, especially with rare classes: if 2 per cent of transactions are fraud, each set should contain about 2 per cent fraud.
  • Split by time for forecasting: train on January to October and test on November and December, never the reverse.
  • Split by group when records are related: all scans from one patient go in the same set, or the model may recognise the patient rather than the disease.
  • Fit preprocessing, such as scalers, on the training set only, then apply them to validation and test data.

Protecting the test set

The test set is valuable precisely because it has influenced nothing. Every time you look at test results and change something in response, a little of that independence is lost. Professional competitions such as those on Kaggle keep a private test set hidden until the end for exactly this reason. If you have looked at the test set many times, the honest response is to collect fresh test data. In scikit-learn, train_test_split(X, y, test_size=0.2, stratify=y, random_state=42) performs a stratified, repeatable split in one line; call it twice to carve out a validation set too.

How it works

SetTypical shareUsed forModel updates from it?
Training70-80%Learning the parametersYes
Validation10-15%Choosing the model and settingsNo, but your choices do
Test10-15%The final reportNever

Choosing how to split

  • Stratified split: keeps class shares equal across sets. Use it for imbalanced classes.
  • Time split: for time series, train on the past and test on the future. Never shuffle.
  • Group split: keep one person's records in only one set, or the model can recognise that person.

In code

split.py
from sklearn.datasets import load_iris
from sklearn.model_selection import train_test_split
import numpy as np

X, y = load_iris(return_X_y=True)

X_rest, X_test, y_rest, y_test = train_test_split(X, y, test_size=0.2, random_state=0, stratify=y)
X_train, X_val, y_train, y_val = train_test_split(X_rest, y_rest, test_size=0.25, random_state=0, stratify=y_rest)

print("Sizes:", len(X_train), len(X_val), len(X_test))
print("Class counts in test:", np.bincount(y_test).tolist())

Output

Sizes: 90 30 30
Class counts in test: [10, 10, 10]

Common mistakes

  • Tuning on the test set. If you check it repeatedly, it stops being a test.
  • Shuffling time-series data, which lets the model see the future.
  • Scaling before splitting.

Check yourself

Question 1

1. Which set helps you choose the learning rate?

Question 2

2. You're predicting stock prices. How should you split?

Summary

  • Train to learn, validate to choose, test once to report.
  • Split the right way: stratified, by time or by group.
  • Fit all preprocessing on training data only.

Read it. Now try it yourself.

Open the interactive activity

Finished reading?

Sources: scikit-learn user guide