Definition
The bias-variance trade-off describes two sources of a model's error on new data. Bias is error from wrong or too-simple assumptions. Variance is error from being too sensitive to the particular training data. As a model gets more complex, bias tends to fall and variance tends to rise.
In simple words
Imagine throwing darts. High bias means your throws land close together but away from the centre: you're consistently wrong. High variance means your throws are scattered everywhere: sometimes close, but unreliable. You want low both: tight and on target.
Why it matters
It tells you how to improve a model. High bias means make it more complex. High variance means simplify it, regularise it or get more data. It explains why ensembles work, and it's among the most common interview questions.
Key terms
- Bias
- Error from overly simple assumptions; the model misses real patterns (underfitting).
- Variance
- Error from sensitivity to the particular training sample; the model fits noise (overfitting).
- Irreducible error
- Noise in the data itself that no model can remove.
- Model complexity
- How flexible a model is, such as polynomial degree, tree depth or number of parameters.
- Learning curve
- Training and validation error plotted against the amount of training data.
- Double descent
- An observation that test error can fall again when models grow far beyond the interpolation point.
Explained in detail
Decomposing prediction error
For squared error, the expected error of a model on new data can be split into three parts: bias squared, variance and irreducible noise. Imagine training the same kind of model many times on different random samples of the same size. Bias measures how far the average of those models' predictions is from the truth; variance measures how much the individual models' predictions scatter around their own average. A straight line fitted to a curved relationship has high bias, because even its average is wrong, but low variance, because every sample gives a similar line. A degree-15 polynomial has low bias but high variance: each sample produces a wildly different curve.
Diagnosing with training and validation error
| Pattern | Diagnosis | Remedies |
|---|---|---|
| High training error, similar validation error | High bias: underfitting | More features, a more flexible model, less regularisation, longer training |
| Low training error, much higher validation error | High variance: overfitting | More data, regularisation, simpler model, bagging, early stopping, augmentation |
| Both low and close | A good fit | Check on truly new data and monitor after deployment |
Learning curves add another view. If both errors are high and flat as data grows, more data will not help; the model needs more capacity. If there is a large gap that narrows as data grows, collecting more data is the best investment. Ensembles exploit the trade-off directly: bagging averages high-variance trees to reduce variance, while boosting combines high-bias stumps to reduce bias. Regularisation strength, tree depth, number of neighbours in k-NN and the number of training epochs are all knobs that move a model along the bias-variance spectrum, and cross-validation is how we find the sweet spot.
Modern deep learning complicates the classic U-shaped picture. Very large neural networks can fit training data perfectly yet still generalise well, and research since 2019 on "double descent" has shown test error falling again as models grow beyond the point where they first fit the training data exactly. Implicit regularisation from stochastic gradient descent, architecture choices and large data sets help explain this. The practical lesson is unchanged: judge models by validation performance, not training fit.
How it works
| High bias | High variance | |
|---|---|---|
| Also called | Underfitting | Overfitting |
| Training error | High | Low |
| Validation error | High, close to training | Much higher than training |
| Example | A line fit to a curve | A very deep tree |
| Fix | More complex model, better features | More data, regularisation, simpler model, ensembles |
The maths
- The true function
- What the model learned, from one training set
- Noise in the data, which no model can remove
In code
import numpy as np
from sklearn.linear_model import LinearRegression
from sklearn.pipeline import make_pipeline
from sklearn.preprocessing import PolynomialFeatures
rng = np.random.default_rng(0)
true = lambda x: np.sin(x)
test_x = np.array([[1.5]])
for degree in (1, 3, 15):
predictions = []
for _ in range(200): # many different training sets
x = rng.uniform(0, 6, 20).reshape(-1, 1)
y = true(x).ravel() + rng.normal(0, 0.3, 20)
model = make_pipeline(PolynomialFeatures(degree), LinearRegression()).fit(x, y)
predictions.append(model.predict(test_x)[0])
predictions = np.array(predictions)
bias_squared = (predictions.mean() - true(1.5)) ** 2
print(f"degree {degree:2}: bias^2 = {bias_squared:.4f}, variance = {predictions.var():.4f}")Output
degree 1: bias^2 = 0.1976, variance = 0.0322 degree 3: bias^2 = 0.0003, variance = 0.0192 degree 15: bias^2 = 0.0003, variance = 0.5668
Try it yourself
Which problem is shown?
Read each pair of training and validation results and diagnose it.
Training error 5 per cent, validation error 6 per cent
Training error 1 per cent, validation error 25 per cent
Validation error falls as you add training data, but stays above training error
Training error 30 per cent, validation error 31 per cent
Common mistakes
- Confusing this statistical bias with social bias. They're different things that share a word.
- Adding data to fix high bias. It mainly helps variance.
Check yourself
Summary
- Bias: too simple. Variance: too sensitive to the data.
- Error = Bias² + Variance + Noise.
- Diagnose which one you have, then apply the right fix.
Finished reading?
Sources: Hastie, Tibshirani & Friedman, The Elements of Statistical Learning