A1 · Supervised Machine Learning in Depth

The Bias-Variance Trade-off

The bias-variance trade-off describes two sources of a model's error on new data. Bias is error from wrong or too-simple assumptions. Variance is error from being too sensitive to the particular training data. As a model gets more complex, bias tends to fall and variance tends to rise.

12 min Advanced Jump to the activity

What you need: The earlier topics of this module, and its prerequisites.

Definition

The bias-variance trade-off describes two sources of a model's error on new data. Bias is error from wrong or too-simple assumptions. Variance is error from being too sensitive to the particular training data. As a model gets more complex, bias tends to fall and variance tends to rise.

In simple words

Imagine throwing darts. High bias means your throws land close together but away from the centre: you're consistently wrong. High variance means your throws are scattered everywhere: sometimes close, but unreliable. You want low both: tight and on target.

Why it matters

It tells you how to improve a model. High bias means make it more complex. High variance means simplify it, regularise it or get more data. It explains why ensembles work, and it's among the most common interview questions.

Key terms

Bias
Error from overly simple assumptions; the model misses real patterns (underfitting).
Variance
Error from sensitivity to the particular training sample; the model fits noise (overfitting).
Irreducible error
Noise in the data itself that no model can remove.
Model complexity
How flexible a model is, such as polynomial degree, tree depth or number of parameters.
Learning curve
Training and validation error plotted against the amount of training data.
Double descent
An observation that test error can fall again when models grow far beyond the interpolation point.

Explained in detail

Decomposing prediction error

For squared error, the expected error of a model on new data can be split into three parts: bias squared, variance and irreducible noise. Imagine training the same kind of model many times on different random samples of the same size. Bias measures how far the average of those models' predictions is from the truth; variance measures how much the individual models' predictions scatter around their own average. A straight line fitted to a curved relationship has high bias, because even its average is wrong, but low variance, because every sample gives a similar line. A degree-15 polynomial has low bias but high variance: each sample produces a wildly different curve.

Diagnosing with training and validation error

PatternDiagnosisRemedies
High training error, similar validation errorHigh bias: underfittingMore features, a more flexible model, less regularisation, longer training
Low training error, much higher validation errorHigh variance: overfittingMore data, regularisation, simpler model, bagging, early stopping, augmentation
Both low and closeA good fitCheck on truly new data and monitor after deployment

Learning curves add another view. If both errors are high and flat as data grows, more data will not help; the model needs more capacity. If there is a large gap that narrows as data grows, collecting more data is the best investment. Ensembles exploit the trade-off directly: bagging averages high-variance trees to reduce variance, while boosting combines high-bias stumps to reduce bias. Regularisation strength, tree depth, number of neighbours in k-NN and the number of training epochs are all knobs that move a model along the bias-variance spectrum, and cross-validation is how we find the sweet spot.

Modern deep learning complicates the classic U-shaped picture. Very large neural networks can fit training data perfectly yet still generalise well, and research since 2019 on "double descent" has shown test error falling again as models grow beyond the point where they first fit the training data exactly. Implicit regularisation from stochastic gradient descent, architecture choices and large data sets help explain this. The practical lesson is unchanged: judge models by validation performance, not training fit.

How it works

High biasHigh variance
Also calledUnderfittingOverfitting
Training errorHighLow
Validation errorHigh, close to trainingMuch higher than training
ExampleA line fit to a curveA very deep tree
FixMore complex model, better featuresMore data, regularisation, simpler model, ensembles

The maths

The decomposition for squared error
The true function
What the model learned, from one training set
Noise in the data, which no model can remove

In code

bias_variance.py
import numpy as np
from sklearn.linear_model import LinearRegression
from sklearn.pipeline import make_pipeline
from sklearn.preprocessing import PolynomialFeatures

rng = np.random.default_rng(0)
true = lambda x: np.sin(x)
test_x = np.array([[1.5]])

for degree in (1, 3, 15):
    predictions = []
    for _ in range(200):                       # many different training sets
        x = rng.uniform(0, 6, 20).reshape(-1, 1)
        y = true(x).ravel() + rng.normal(0, 0.3, 20)
        model = make_pipeline(PolynomialFeatures(degree), LinearRegression()).fit(x, y)
        predictions.append(model.predict(test_x)[0])
    predictions = np.array(predictions)
    bias_squared = (predictions.mean() - true(1.5)) ** 2
    print(f"degree {degree:2}: bias^2 = {bias_squared:.4f}, variance = {predictions.var():.4f}")

Output

degree  1: bias^2 = 0.1976, variance = 0.0322
degree  3: bias^2 = 0.0003, variance = 0.0192
degree 15: bias^2 = 0.0003, variance = 0.5668

Try it yourself

Which problem is shown?

Read each pair of training and validation results and diagnose it.

  • Training error 5 per cent, validation error 6 per cent

  • Training error 1 per cent, validation error 25 per cent

  • Validation error falls as you add training data, but stays above training error

  • Training error 30 per cent, validation error 31 per cent

Common mistakes

  • Confusing this statistical bias with social bias. They're different things that share a word.
  • Adding data to fix high bias. It mainly helps variance.

Check yourself

Question 1

1. Training error is low, and validation error is high. Which is it?

Question 2

2. Which part of the error can't be reduced by any model?

Summary

  • Bias: too simple. Variance: too sensitive to the data.
  • Error = Bias² + Variance + Noise.
  • Diagnose which one you have, then apply the right fix.

Finished reading?

Sources: Hastie, Tibshirani & Friedman, The Elements of Statistical Learning