Definition
A model scores 99% in training and 60% on new data. What went wrong?
Overfitting is when a model learns the training data too closely, including its noise, so it does poorly on new data. Underfitting is when a model is too simple to capture the real pattern, so it does poorly even on the training data.
In simple words
An overfit student memorised last year's answers, and fails when the questions change. An underfit student barely studied, and fails everything. A good student understood the ideas, and handles new questions.
Overview
- Underfitting means a model is too simple to capture the pattern: errors are high on training and test data.
- Overfitting means a model memorises training noise: training error is low but test error high.
- The bias-variance trade-off explains both: simple models have high bias, complex ones high variance.
- Learning curves and validation curves diagnose which problem a model has.
- More data, regularisation and simpler models fight overfitting; richer features and models fight underfitting.
- The goal is a model that generalises: it performs well on new, unseen data.
Why it matters
Generalising to new data is the whole point of ML, so this is the most common problem in the field. Spotting which one you have tells you exactly what to do next.
Key terms
- Overfitting
- Fitting the training data so closely, noise included, that performance on new data suffers.
- Underfitting
- A model too simple to capture the real pattern, doing poorly even on training data.
- Bias
- Error from overly simple assumptions; high bias causes underfitting.
- Variance
- Error from sensitivity to the particular training sample; high variance causes overfitting.
- Learning curve
- A plot of training and validation error as training data or training time grows.
- Regularisation
- Techniques that discourage needless complexity, such as weight penalties or dropout.
Types
There are a few types of fit and several types of remedy. Knowing which applies tells you what to try next.
1. Underfitting (high bias)
A model too simple for the data.
- How it works: It misses real patterns, so training and validation errors are both high and close together.
- Example: A straight line fitted to yields that rise and then fall with fertiliser.
- When to use it: Fix it with more informative features, a more flexible model or less regularisation.
2. Good fit
A model that captures the real pattern without memorising noise.
- How it works: Training error is reasonably low and validation error is close to it.
- Example: A degree-2 polynomial with cross-validated error 0.095 on the fertiliser data.
- When to use it: Aim for it, and confirm it once on the test set. Expect a small gap between the two errors.
3. Overfitting (high variance)
A model so flexible that it fits the training data's noise.
- How it works: Training error is very low but validation error much higher; small changes in the training data change the model a lot.
- Example: A single deep decision tree scoring 1.00 on training data and 0.83 on test data.
- When to use it: Fix it with more data, regularisation, simpler models, ensembles, early stopping or data augmentation.
4. Learning curves
Plots of training and validation scores as the training set grows.
- How it works: If both curves plateau at a poor score, the model underfits; if a large gap remains, it overfits and more data may help.
- Example: learning_curve in scikit-learn showing validation accuracy still rising at 5,000 examples.
- When to use it: Use them to decide whether collecting more data is worthwhile.
5. Validation curves and double descent
Plots of scores against a complexity setting, such as tree depth or polynomial degree.
- How it works: Validation error usually falls then rises as complexity grows (a U-shape); very large neural networks can show a second fall, called double descent.
- Example: Validation error for boosting lowest at 48 trees and higher by 500.
- When to use it: Use them to choose the complexity setting with the lowest validation error.
Explained in detail
Two ways to fail
Imagine fitting a curve to 20 noisy points that really follow a gentle curve. A straight line misses the bend: it is wrong on the training points and on new points alike. That is underfitting. A wiggly curve through every single point gets the training error to zero, but it has followed the noise, so it swings wildly between the points and predicts new data badly. That is overfitting. A smooth curve that follows the general trend and ignores the noise is the goal. The same pattern appears with students: one who learns nothing fails every exam, and one who memorises last year's answers fails when the questions change.
Diagnosing with training and validation scores
| Training error | Validation error | Diagnosis | What to try |
|---|---|---|---|
| High | High | Underfitting (high bias) | A more flexible model, better features, less regularisation, train longer |
| Low | Much higher | Overfitting (high variance) | More data, simpler model, regularisation, early stopping |
| Low | Low and close | A good fit | Confirm on the test set |
| Low | Suspiciously low too | Possible leakage | Check splits and features |
The bias-variance trade-off
Prediction error on new data can be thought of as bias, plus variance, plus irreducible noise. Simple models have high bias but low variance: they would give similar predictions whichever training sample they saw. Flexible models have low bias but high variance: their predictions change a lot with the training sample. Increasing complexity first reduces total error, as bias falls, then increases it, as variance takes over. The best model sits near the bottom of this U-shaped curve, and the way to find it is to compare validation performance across levels of complexity.
Remedies for overfitting
- Get more and more varied training data; this is often the most effective fix.
- Simplify the model: fewer features, a shallower tree, fewer layers.
- Regularise: L1 or L2 penalties on weights, dropout in neural networks.
- Stop early: halt training when validation error starts to rise.
- Augment data: create varied copies of images by flipping, cropping or changing brightness.
- Use ensembles such as random forests, which average away much of the variance.
How it works
| Training score | Validation score | Fix | |
|---|---|---|---|
| Underfitting | Low | Low | More features, a more complex model, longer training |
| Good fit | High | High, and close to training | You're done. Test it. |
| Overfitting | Very high | Much lower | More data, regularisation, a simpler model, early stopping |
In code
Watch the training score climb as the tree grows deeper, while the validation score stops improving.
from sklearn.datasets import load_breast_cancer
from sklearn.model_selection import train_test_split
from sklearn.tree import DecisionTreeClassifier
X, y = load_breast_cancer(return_X_y=True)
X_train, X_val, y_train, y_val = train_test_split(X, y, random_state=0)
for depth in (1, 2, 4, 8, None):
tree = DecisionTreeClassifier(max_depth=depth, random_state=0).fit(X_train, y_train)
print(f"depth {str(depth):>4}: train {tree.score(X_train, y_train):.3f} validation {tree.score(X_val, y_val):.3f}")Output
depth 1: train 0.923 validation 0.881 depth 2: train 0.941 validation 0.937 depth 4: train 0.993 validation 0.902 depth 8: train 1.000 validation 0.881 depth None: train 1.000 validation 0.881
Common mistakes
- Celebrating a high training score. It says little by itself.
- Adding complexity when the model already overfits.
- Collecting more data for an underfit model. It usually won't help, because the model can't capture the pattern anyway.
Check yourself
Summary
- Overfitting: great on training, poor on new data.
- Underfitting: poor on both.
- Compare training and validation scores to tell which.
Finished reading?
Sources: Google Machine Learning Crash Course