Definition
When is a straight line not enough?
Multiple regression uses several features at once. Polynomial regression models curved relationships by adding powers and products of features, such as x² and x₁x₂, and then fitting an ordinary linear model on the expanded features. The degree controls flexibility, and is chosen by validation error.
In simple words
The model is still 'linear' in its weights, which is why the same least-squares maths works, but the inputs are bent. Degree 1 is a line, degree 2 a parabola, degree 9 a wiggly curve that can pass through every training point and still predict badly between them.
Why it matters
Real relationships are rarely straight. Polynomial features are the simplest way to add flexibility, and choosing the degree is the clearest example of the bias-variance trade-off you'll meet in this module.
Key terms
- Multiple regression
- Linear regression with several features at once.
- Polynomial features
- New features such as x², x³ and x₁x₂ that let a linear model fit curves.
- Interaction term
- A product of two features, capturing an effect of one that depends on the other.
- Degree
- The highest power used; higher degree means more flexible curves.
- Partial effect
- The change in prediction from one feature while others are held constant.
- Spline
- A smooth curve made of pieces joined at knots, often more stable than high-degree polynomials.
Explained in detail
Several features at once
Multiple regression predicts with many features: price = w₀ + w₁·area + w₂·bedrooms + w₃·age + …. Each coefficient is a partial effect, the change in price for one more square metre while bedrooms and age stay the same. This can differ sharply from a simple one-feature slope. Bedrooms alone may correlate positively with price, yet in a model that already includes area, the bedrooms coefficient can be negative, because for a fixed area more bedrooms means smaller rooms. Such sign changes are a common source of confusion, and they are why coefficients must always be interpreted in the context of the other features in the model.
Bending the line with polynomial features
A linear model is linear in its weights, not necessarily in the original inputs. By adding x² as a new column, the model ŷ = w₀ + w₁x + w₂x² can fit a parabola, for example crop yield rising with fertiliser up to a point and then falling as excess fertiliser damages plants. Adding x³ allows an S-shape, and interaction terms such as rainfall × fertiliser let the effect of fertiliser depend on rainfall. PolynomialFeatures in scikit-learn generates these columns automatically. The number of columns grows quickly: degree 3 with 10 features gives 286 columns, so polynomial models need regularisation and careful validation.
| Degree | Typical behaviour |
|---|---|
| 1 | A straight line; underfits curved patterns |
| 2 or 3 | Captures simple curves and turning points |
| 10 or more | Wiggles through training points; extreme values at the edges; overfits |
Choose the degree by cross-validation, not by training error, which always falls as degree rises. Scale features before creating polynomial terms, because x¹⁰ of a value like 1,000 is astronomically large and causes numerical problems. Beware of extrapolation: a polynomial fitted on fertiliser doses between 50 and 150 kilograms per hectare can predict absurd yields at 300. When you need flexible curves, splines or tree-based models are often more stable, and generalised additive models combine the interpretability of separate per-feature curves with good accuracy.
When reporting a multiple regression, show each coefficient with its units and a confidence interval, and state which other features were held constant.
How it works
- PolynomialFeatures(degree=d) creates every power and product up to degree d.
- The number of features grows fast: with 10 features, degree 3 gives 286 columns.
- Scale features before high degrees, or powers become enormous.
- Choose the degree with the lowest validation error, not training error.
The maths
In code
Fitting degrees 1 to 12 to a curved signal, and comparing training and validation error.
import numpy as np
from sklearn.linear_model import LinearRegression
from sklearn.metrics import mean_squared_error
from sklearn.model_selection import train_test_split
from sklearn.pipeline import make_pipeline
from sklearn.preprocessing import PolynomialFeatures, StandardScaler
rng = np.random.default_rng(0)
x = rng.uniform(-3, 3, 60).reshape(-1, 1)
y = np.sin(x).ravel() + rng.normal(scale=0.2, size=60)
x_tr, x_va, y_tr, y_va = train_test_split(x, y, test_size=0.4, random_state=0)
print("degree train MSE validation MSE")
for d in [1, 2, 3, 5, 8, 12]:
model = make_pipeline(PolynomialFeatures(d), StandardScaler(), LinearRegression()).fit(x_tr, y_tr)
tr = mean_squared_error(y_tr, model.predict(x_tr))
va = mean_squared_error(y_va, model.predict(x_va))
print(f"{d:>6} {tr:9.3f} {va:14.3f}")
print("Columns for 10 features at degree 3:", PolynomialFeatures(3).fit(np.zeros((1, 10))).n_output_features_)Output
degree train MSE validation MSE
1 0.188 0.329
2 0.188 0.329
3 0.039 0.053
5 0.032 0.046
8 0.022 0.072
12 0.020 0.087
Columns for 10 features at degree 3: 286Try it yourself
Best degree?
Training error always falls as degree rises. Classify each situation.
Degree 3: low training error, lowest validation error
Degree 12: near-zero training error, validation error rising sharply
Degree 1 on a sine wave: high training and validation error
Common mistakes
- Choosing the degree by training error.
- Using high degrees without scaling, causing numerical problems.
- Extrapolating a polynomial far outside the training range, where it shoots off.
Check yourself
Summary
- Polynomial features let a linear model fit curves.
- Degree controls flexibility; feature count grows fast.
- Pick the degree by validation error and scale first.
Read it. Now try it yourself.
Open the interactive activityFinished reading?
Sources: James et al., An Introduction to Statistical Learning · scikit-learn: Polynomial features