I5 · Machine Learning Foundations

Overfitting and Underfitting

Overfitting is when a model learns the training data too closely, including its noise, so it does poorly on new data. Underfitting is when a model is too simple to capture the real pattern, so it does poorly even on the training data.

10 min Intermediate · Sublevel Intermediate

What you need: The earlier topics of this module.

Definition

A model scores 99% in training and 60% on new data. What went wrong?

Overfitting is when a model learns the training data too closely, including its noise, so it does poorly on new data. Underfitting is when a model is too simple to capture the real pattern, so it does poorly even on the training data.

In simple words

An overfit student memorised last year's answers, and fails when the questions change. An underfit student barely studied, and fails everything. A good student understood the ideas, and handles new questions.

Overview

  • Underfitting means a model is too simple to capture the pattern: errors are high on training and test data.
  • Overfitting means a model memorises training noise: training error is low but test error high.
  • The bias-variance trade-off explains both: simple models have high bias, complex ones high variance.
  • Learning curves and validation curves diagnose which problem a model has.
  • More data, regularisation and simpler models fight overfitting; richer features and models fight underfitting.
  • The goal is a model that generalises: it performs well on new, unseen data.

Why it matters

Generalising to new data is the whole point of ML, so this is the most common problem in the field. Spotting which one you have tells you exactly what to do next.

Key terms

Overfitting
Fitting the training data so closely, noise included, that performance on new data suffers.
Underfitting
A model too simple to capture the real pattern, doing poorly even on training data.
Bias
Error from overly simple assumptions; high bias causes underfitting.
Variance
Error from sensitivity to the particular training sample; high variance causes overfitting.
Learning curve
A plot of training and validation error as training data or training time grows.
Regularisation
Techniques that discourage needless complexity, such as weight penalties or dropout.

Types

There are a few types of fit and several types of remedy. Knowing which applies tells you what to try next.

1. Underfitting (high bias)

A model too simple for the data.

  • How it works: It misses real patterns, so training and validation errors are both high and close together.
  • Example: A straight line fitted to yields that rise and then fall with fertiliser.
  • When to use it: Fix it with more informative features, a more flexible model or less regularisation.

2. Good fit

A model that captures the real pattern without memorising noise.

  • How it works: Training error is reasonably low and validation error is close to it.
  • Example: A degree-2 polynomial with cross-validated error 0.095 on the fertiliser data.
  • When to use it: Aim for it, and confirm it once on the test set. Expect a small gap between the two errors.

3. Overfitting (high variance)

A model so flexible that it fits the training data's noise.

  • How it works: Training error is very low but validation error much higher; small changes in the training data change the model a lot.
  • Example: A single deep decision tree scoring 1.00 on training data and 0.83 on test data.
  • When to use it: Fix it with more data, regularisation, simpler models, ensembles, early stopping or data augmentation.

4. Learning curves

Plots of training and validation scores as the training set grows.

  • How it works: If both curves plateau at a poor score, the model underfits; if a large gap remains, it overfits and more data may help.
  • Example: learning_curve in scikit-learn showing validation accuracy still rising at 5,000 examples.
  • When to use it: Use them to decide whether collecting more data is worthwhile.

5. Validation curves and double descent

Plots of scores against a complexity setting, such as tree depth or polynomial degree.

  • How it works: Validation error usually falls then rises as complexity grows (a U-shape); very large neural networks can show a second fall, called double descent.
  • Example: Validation error for boosting lowest at 48 trees and higher by 500.
  • When to use it: Use them to choose the complexity setting with the lowest validation error.

Explained in detail

Two ways to fail

Imagine fitting a curve to 20 noisy points that really follow a gentle curve. A straight line misses the bend: it is wrong on the training points and on new points alike. That is underfitting. A wiggly curve through every single point gets the training error to zero, but it has followed the noise, so it swings wildly between the points and predicts new data badly. That is overfitting. A smooth curve that follows the general trend and ignores the noise is the goal. The same pattern appears with students: one who learns nothing fails every exam, and one who memorises last year's answers fails when the questions change.

Diagnosing with training and validation scores

Training errorValidation errorDiagnosisWhat to try
HighHighUnderfitting (high bias)A more flexible model, better features, less regularisation, train longer
LowMuch higherOverfitting (high variance)More data, simpler model, regularisation, early stopping
LowLow and closeA good fitConfirm on the test set
LowSuspiciously low tooPossible leakageCheck splits and features

The bias-variance trade-off

Prediction error on new data can be thought of as bias, plus variance, plus irreducible noise. Simple models have high bias but low variance: they would give similar predictions whichever training sample they saw. Flexible models have low bias but high variance: their predictions change a lot with the training sample. Increasing complexity first reduces total error, as bias falls, then increases it, as variance takes over. The best model sits near the bottom of this U-shaped curve, and the way to find it is to compare validation performance across levels of complexity.

Remedies for overfitting

  1. Get more and more varied training data; this is often the most effective fix.
  2. Simplify the model: fewer features, a shallower tree, fewer layers.
  3. Regularise: L1 or L2 penalties on weights, dropout in neural networks.
  4. Stop early: halt training when validation error starts to rise.
  5. Augment data: create varied copies of images by flipping, cropping or changing brightness.
  6. Use ensembles such as random forests, which average away much of the variance.

How it works

Training scoreValidation scoreFix
UnderfittingLowLowMore features, a more complex model, longer training
Good fitHighHigh, and close to trainingYou're done. Test it.
OverfittingVery highMuch lowerMore data, regularisation, a simpler model, early stopping

In code

Watch the training score climb as the tree grows deeper, while the validation score stops improving.

overfit.py
from sklearn.datasets import load_breast_cancer
from sklearn.model_selection import train_test_split
from sklearn.tree import DecisionTreeClassifier

X, y = load_breast_cancer(return_X_y=True)
X_train, X_val, y_train, y_val = train_test_split(X, y, random_state=0)

for depth in (1, 2, 4, 8, None):
    tree = DecisionTreeClassifier(max_depth=depth, random_state=0).fit(X_train, y_train)
    print(f"depth {str(depth):>4}: train {tree.score(X_train, y_train):.3f}   validation {tree.score(X_val, y_val):.3f}")

Output

depth    1: train 0.923   validation 0.881
depth    2: train 0.941   validation 0.937
depth    4: train 0.993   validation 0.902
depth    8: train 1.000   validation 0.881
depth None: train 1.000   validation 0.881

Common mistakes

  • Celebrating a high training score. It says little by itself.
  • Adding complexity when the model already overfits.
  • Collecting more data for an underfit model. It usually won't help, because the model can't capture the pattern anyway.

Check yourself

Question 1

1. Training 98%, validation 72%. What is this?

Question 2

2. Training 62%, validation 60%. What should you try?

Question 3

3. Which helps reduce overfitting?

Summary

  • Overfitting: great on training, poor on new data.
  • Underfitting: poor on both.
  • Compare training and validation scores to tell which.

Finished reading?

Sources: Google Machine Learning Crash Course