A1 · Supervised Machine Learning in Depth

Data Leakage and Validation Design

Data leakage is any way in which information that would not be available at prediction time reaches the model during training or evaluation, making scores look better than they will be in real use. Validation design is choosing splits that mimic real use, so that the evaluation is honest.

12 min Advanced Jump to the activity

What you need: The earlier topics of this module, and its prerequisites.

Definition

Why did a model with 99 per cent validation accuracy fail in real use?

Data leakage is any way in which information that would not be available at prediction time reaches the model during training or evaluation, making scores look better than they will be in real use. Validation design is choosing splits that mimic real use, so that the evaluation is honest.

In simple words

Leakage is the model seeing the answers before the exam. A feature recorded after the outcome, such as 'refund issued' when predicting fraud, gives the answer away. Preprocessing or feature selection done on all the data lets test information slip into training. Future data in the training set lets a forecast cheat.

Why it matters

Leakage is the most common reason for models that shine in notebooks and fail in production. It's invisible in the score; you only find it by reasoning about how and when each feature is created.

Key terms

Data leakage
Information in training that would not be available at prediction time, inflating scores.
Target leakage
A feature that is caused by, or recorded after, the target, such as "refund issued" when predicting complaints.
Train-test contamination
Test information influencing training, for example scaling with statistics from all rows.
Group split
Keeping all rows from one patient, customer or school in the same fold.
Temporal split
Training on earlier periods and validating on later ones.
Nested cross-validation
An outer loop for evaluation and an inner loop for tuning, so tuning never sees the evaluation folds.

Explained in detail

How leakage happens

Leakage is the most common reason a model that scored brilliantly in development fails in use. It comes in several forms. Target leakage occurs when a feature encodes the answer: a model predicting whether a patient has pneumonia may learn from "antibiotics prescribed", which is recorded after diagnosis. Preprocessing leakage occurs when scaling, imputation, feature selection or target encoding is fitted on the whole data set before splitting, so test statistics shape the training features. Duplicate or near-duplicate leakage occurs when the same person, photo or document appears in both training and test data. Temporal leakage occurs when future information, such as next month's prices, is used to predict the present. In each case the validation score measures something easier than the real task.

Designing the split to match reality

SituationSplitReason
Independent rows, such as one-off customersRandom or stratified k-foldRows are exchangeable
Several rows per patient or studentGroupKFold by patient or studentThe model must work on new people
Data ordered in timeTimeSeriesSplit or a date cut-offThe model only ever sees the past
Rare positive classStratified foldsEach fold keeps the class ratio
Model must work in a new hospital or regionLeave-one-site-outMeasures transfer to unseen sites

A practical defence is to ask, for every feature, "would I know this value at the exact moment I need the prediction?" and to wrap all preprocessing inside a pipeline that cross-validation refits on each training fold. Be suspicious of results that are too good, such as 99 per cent accuracy on a hard problem, and of single features with enormous importance. When tuning many hyperparameters, use nested cross-validation or a separate validation set, because the best score among many tries is optimistically biased. Finally, test on data collected later or elsewhere whenever possible, since that is the closest preview of deployment.

A famous example comes from medical imaging: models trained to detect pneumonia on chest X-rays from several hospitals partly learned which hospital an image came from, because hospitals differed in both disease rates and scanner markings. Splitting by patient rather than by image, and testing on a hospital not used in training, revealed much lower true accuracy. Kaggle competitions have also been won by exploiting leaked identifiers, which shows how easily leakage inflates scores without anyone intending it.

How it works

LeakExamplePrevention
Target leakageA 'treatment given' feature when predicting diseaseUse only features known before the prediction time
Preprocessing leakageScaling or selecting features on all data, then splittingPut every step inside a Pipeline
Group leakageScans of the same patient in train and testGroupKFold by patient
Temporal leakageRandom split of time-series dataTrain on the past, test on the future
Duplicate leakageThe same row in train and testDeduplicate before splitting

In code

Pure noise has nothing to learn, so honest accuracy should be about 50 per cent. Selecting features on all the data first makes noise look predictive; doing it inside the pipeline does not.

leakage.py
import numpy as np
from sklearn.feature_selection import SelectKBest, f_classif
from sklearn.linear_model import LogisticRegression
from sklearn.model_selection import cross_val_score
from sklearn.pipeline import make_pipeline

rng = np.random.default_rng(0)
X = rng.normal(size=(300, 2000))          # 2,000 random features
y = rng.integers(0, 2, size=300)          # random labels: nothing to learn

# Leaky: choose the 20 'best' features using ALL rows, then cross-validate
X_leaky = SelectKBest(f_classif, k=20).fit_transform(X, y)
leaky = cross_val_score(LogisticRegression(max_iter=1000), X_leaky, y, cv=5).mean()

# Honest: feature selection happens inside each training fold
honest_model = make_pipeline(SelectKBest(f_classif, k=20), LogisticRegression(max_iter=1000))
honest = cross_val_score(honest_model, X, y, cv=5).mean()

print(f"Leaky accuracy on pure noise:  {leaky:.2f}")
print(f"Honest accuracy on pure noise: {honest:.2f}")

Output

Leaky accuracy on pure noise:  0.75
Honest accuracy on pure noise: 0.51

Try it yourself

Is there a leak?

For each setup, decide whether information leaks into training or evaluation.

  • Several X-rays per patient, split by patient with GroupKFold

  • Filling missing values with the mean of the whole dataset before splitting

  • Random split of daily sales to forecast next month

  • StandardScaler inside a Pipeline evaluated with cross_val_score

  • Predicting loan default using 'number of collection calls made'

Common mistakes

  • Trusting a score that seems too good to be true.
  • Doing feature selection or oversampling before the split.
  • Using random splits for time series or grouped data.

Check yourself

Question 1

1. What best prevents preprocessing leakage?

Question 2

2. For time-series forecasting, the test set should be…

Question 3

3. In the code, why does the leaky version beat 50 per cent on noise?

Summary

  • Leakage gives the model information it won't have in real use.
  • Target, preprocessing, group, temporal and duplicate leaks are the common kinds.
  • Design validation to mimic real use, and put preprocessing in a Pipeline.

Finished reading?

Sources: scikit-learn: Common pitfalls and data leakage · Hastie et al., The Elements of Statistical Learning