Definition
Why did a model with 99 per cent validation accuracy fail in real use?
Data leakage is any way in which information that would not be available at prediction time reaches the model during training or evaluation, making scores look better than they will be in real use. Validation design is choosing splits that mimic real use, so that the evaluation is honest.
In simple words
Leakage is the model seeing the answers before the exam. A feature recorded after the outcome, such as 'refund issued' when predicting fraud, gives the answer away. Preprocessing or feature selection done on all the data lets test information slip into training. Future data in the training set lets a forecast cheat.
Why it matters
Leakage is the most common reason for models that shine in notebooks and fail in production. It's invisible in the score; you only find it by reasoning about how and when each feature is created.
Key terms
- Data leakage
- Information in training that would not be available at prediction time, inflating scores.
- Target leakage
- A feature that is caused by, or recorded after, the target, such as "refund issued" when predicting complaints.
- Train-test contamination
- Test information influencing training, for example scaling with statistics from all rows.
- Group split
- Keeping all rows from one patient, customer or school in the same fold.
- Temporal split
- Training on earlier periods and validating on later ones.
- Nested cross-validation
- An outer loop for evaluation and an inner loop for tuning, so tuning never sees the evaluation folds.
Explained in detail
How leakage happens
Leakage is the most common reason a model that scored brilliantly in development fails in use. It comes in several forms. Target leakage occurs when a feature encodes the answer: a model predicting whether a patient has pneumonia may learn from "antibiotics prescribed", which is recorded after diagnosis. Preprocessing leakage occurs when scaling, imputation, feature selection or target encoding is fitted on the whole data set before splitting, so test statistics shape the training features. Duplicate or near-duplicate leakage occurs when the same person, photo or document appears in both training and test data. Temporal leakage occurs when future information, such as next month's prices, is used to predict the present. In each case the validation score measures something easier than the real task.
Designing the split to match reality
| Situation | Split | Reason |
|---|---|---|
| Independent rows, such as one-off customers | Random or stratified k-fold | Rows are exchangeable |
| Several rows per patient or student | GroupKFold by patient or student | The model must work on new people |
| Data ordered in time | TimeSeriesSplit or a date cut-off | The model only ever sees the past |
| Rare positive class | Stratified folds | Each fold keeps the class ratio |
| Model must work in a new hospital or region | Leave-one-site-out | Measures transfer to unseen sites |
A practical defence is to ask, for every feature, "would I know this value at the exact moment I need the prediction?" and to wrap all preprocessing inside a pipeline that cross-validation refits on each training fold. Be suspicious of results that are too good, such as 99 per cent accuracy on a hard problem, and of single features with enormous importance. When tuning many hyperparameters, use nested cross-validation or a separate validation set, because the best score among many tries is optimistically biased. Finally, test on data collected later or elsewhere whenever possible, since that is the closest preview of deployment.
A famous example comes from medical imaging: models trained to detect pneumonia on chest X-rays from several hospitals partly learned which hospital an image came from, because hospitals differed in both disease rates and scanner markings. Splitting by patient rather than by image, and testing on a hospital not used in training, revealed much lower true accuracy. Kaggle competitions have also been won by exploiting leaked identifiers, which shows how easily leakage inflates scores without anyone intending it.
How it works
| Leak | Example | Prevention |
|---|---|---|
| Target leakage | A 'treatment given' feature when predicting disease | Use only features known before the prediction time |
| Preprocessing leakage | Scaling or selecting features on all data, then splitting | Put every step inside a Pipeline |
| Group leakage | Scans of the same patient in train and test | GroupKFold by patient |
| Temporal leakage | Random split of time-series data | Train on the past, test on the future |
| Duplicate leakage | The same row in train and test | Deduplicate before splitting |
In code
Pure noise has nothing to learn, so honest accuracy should be about 50 per cent. Selecting features on all the data first makes noise look predictive; doing it inside the pipeline does not.
import numpy as np
from sklearn.feature_selection import SelectKBest, f_classif
from sklearn.linear_model import LogisticRegression
from sklearn.model_selection import cross_val_score
from sklearn.pipeline import make_pipeline
rng = np.random.default_rng(0)
X = rng.normal(size=(300, 2000)) # 2,000 random features
y = rng.integers(0, 2, size=300) # random labels: nothing to learn
# Leaky: choose the 20 'best' features using ALL rows, then cross-validate
X_leaky = SelectKBest(f_classif, k=20).fit_transform(X, y)
leaky = cross_val_score(LogisticRegression(max_iter=1000), X_leaky, y, cv=5).mean()
# Honest: feature selection happens inside each training fold
honest_model = make_pipeline(SelectKBest(f_classif, k=20), LogisticRegression(max_iter=1000))
honest = cross_val_score(honest_model, X, y, cv=5).mean()
print(f"Leaky accuracy on pure noise: {leaky:.2f}")
print(f"Honest accuracy on pure noise: {honest:.2f}")Output
Leaky accuracy on pure noise: 0.75 Honest accuracy on pure noise: 0.51
Try it yourself
Is there a leak?
For each setup, decide whether information leaks into training or evaluation.
Several X-rays per patient, split by patient with GroupKFold
Filling missing values with the mean of the whole dataset before splitting
Random split of daily sales to forecast next month
StandardScaler inside a Pipeline evaluated with cross_val_score
Predicting loan default using 'number of collection calls made'
Common mistakes
- Trusting a score that seems too good to be true.
- Doing feature selection or oversampling before the split.
- Using random splits for time series or grouped data.
Check yourself
Summary
- Leakage gives the model information it won't have in real use.
- Target, preprocessing, group, temporal and duplicate leaks are the common kinds.
- Design validation to mimic real use, and put preprocessing in a Pipeline.
Finished reading?
Sources: scikit-learn: Common pitfalls and data leakage · Hastie et al., The Elements of Statistical Learning