Definition
Crop yield rises as you add fertiliser, but only up to a point; then too much fertiliser burns the plants and yield falls. A straight line cannot show that. A curve can, but a curve that is too wiggly will chase noise. This page shows how to bend a line just enough.
Polynomial regression fits a curve by adding powers of a feature, such as x² and x³, as extra columns and then using ordinary linear regression on them: ŷ = w₀ + w₁x + w₂x² + … + w_dx^d. The degree d controls flexibility. Regularisation adds a penalty on the size of the weights to the loss: ridge (L2) adds λΣw², shrinking all weights; lasso (L1) adds λΣ|w|, which can set some weights exactly to zero. The penalty strength λ (called alpha in scikit-learn) is chosen by validation.
In simple words
Think of a flexible ruler. A stiff ruler (degree 1) can only make a straight line. A bendy ruler (high degree) can follow every bump, including random ones. Regularisation is like adding a spring that pulls the ruler back towards straight, so it bends only where the data really demands.
Overview
- Polynomial regression adds powers of features so that a linear model can fit curves.
- The degree controls flexibility: too low underfits, too high overfits.
- Regularisation adds a penalty on weight size to the loss, discouraging extreme weights.
- The strength λ (alpha in scikit-learn) is chosen by cross-validation on a log scale.
- Features must be scaled before regularising.
- Polynomials should not be used far outside the range of the training data.
Why it matters
Many real relationships are curved: dose and response, speed and fuel use, age and income. Polynomial features and regularisation let simple, explainable linear models capture curves without overfitting, and the same penalty idea appears everywhere in machine learning, from logistic regression to neural networks.
Key terms
- Polynomial feature
- A new column such as x², x³ or x₁x₂ created from existing features.
- Degree
- The highest power used; it sets how flexible the curve is.
- Ridge regression (L2)
- Least squares plus λ times the sum of squared weights.
- Lasso regression (L1)
- Least squares plus λ times the sum of absolute weights; performs feature selection.
- Elastic net
- A blend of L1 and L2 penalties.
- Regularisation strength (λ, alpha)
- How strongly large weights are penalised; larger means simpler models.
Types
There are several types of regularisation, and polynomial features come in a few forms. These are the main types.
1. Polynomial and interaction features
Extra columns built from powers and products of features.
- How it works: PolynomialFeatures(degree=2) turns (x₁, x₂) into (1, x₁, x₂, x₁², x₁x₂, x₂²); interaction_only=True keeps only products.
- Example: Fertiliser × rainfall capturing that fertiliser helps more in a good monsoon.
- When to use it: Use them for curved effects and interactions, keeping the degree low (2 or 3).
2. Ridge regression (L2)
Least squares plus λ times the sum of squared weights.
- How it works: Large weights are shrunk smoothly towards zero but rarely reach it; correlated features share weight.
- Example: Ridge keeping a degree-12 polynomial's cross-validated error at 0.47 instead of millions.
- When to use it: Use it as the default regulariser when many features each matter a little.
3. Lasso regression (L1)
Least squares plus λ times the sum of absolute weights.
- How it works: The absolute penalty pushes weak weights exactly to zero, so lasso selects features.
- Example: Reducing 300 gene measurements to the 15 that predict a disease.
- When to use it: Use it when you suspect only a few features matter and want a simpler model.
4. Elastic net
A mix of L1 and L2 penalties controlled by l1_ratio.
- How it works: It selects features like lasso but handles groups of correlated features more stably, like ridge.
- Example: Predicting house prices from many overlapping location features.
- When to use it: Use it when features are many and correlated and you also want sparsity.
5. Early stopping and other regularisers
Other ways to limit model complexity: stopping training early, dropout and limiting tree depth.
- How it works: Early stopping halts when validation error stops improving; dropout randomly disables neurons; max_depth caps tree size.
- Example: Gradient boosting stopped at 48 trees when validation error began to rise.
- When to use it: Use the regulariser that fits the model family: penalties for linear models, early stopping for boosting and networks.
Explained in detail
Fitting curves with a linear model
The model ŷ = w₀ + w₁x + w₂x² is still called linear regression because it is linear in the weights; only the inputs have been transformed. In scikit-learn, PolynomialFeatures(degree=2) turns a column x into [1, x, x²], and LinearRegression then fits the weights as usual. With two features, degree 2 also creates the interaction x₁x₂, which lets the effect of one feature depend on the other, for example fertiliser helping more when rainfall is good. The number of new columns grows fast: 10 features at degree 3 produce 286 columns. Always scale features before raising them to powers, because a value like 1,000 cubed is a billion.
Choosing the degree
| Degree | Training error | Validation error | Diagnosis |
|---|---|---|---|
| 1 | High | High | Underfits the curve |
| 2 or 3 | Lower | Lowest | A good fit |
| 12 | Very low | High | Overfits: wiggles between points |
How regularisation tames a flexible model
A high-degree polynomial overfits because its weights become huge and of opposite signs, cancelling each other at training points and swinging wildly between them. Ridge regression adds λΣw² to the squared error, so the model must justify every large weight with a real reduction in error. As λ grows, weights shrink and the curve smooths out; at λ = 0 we get ordinary least squares. Lasso's penalty λΣ|w| behaves differently: as λ grows, weights hit exactly zero one by one, leaving a simpler model that uses only some features, which helps when many features are irrelevant. Elastic net mixes the two and works well when features are correlated.
Choose λ, and the degree, by cross-validation: try values on a logarithmic scale such as 0.001, 0.01, 0.1, 1, 10 and 100, and keep the one with the lowest validation error. Scikit-learn's RidgeCV and LassoCV do this automatically. Features must be standardised before regularising, otherwise a feature measured in rupees and one measured in lakhs would be penalised very differently. Finally, never extrapolate a polynomial far outside the range of the training data: a curve fitted to fertiliser doses between 50 and 150 kilograms can predict nonsense at 300. In the code below, plain least squares does best at degree 2 (cross-validated error 0.095) and collapses at degree 12 (an error in the millions), while ridge with λ chosen by cross-validation keeps the degree-12 error at 0.47.
How it works
- Plot the data to see whether the relationship is curved.
- Scale the features, then add polynomial features of a chosen degree.
- Fit ridge or lasso instead of plain least squares.
- Choose the degree and λ by cross-validation.
- Check the final curve on test data and avoid extrapolating.
The maths
- Regularisation strength; larger means smaller weights
Fertiliser doses of 0, 50, 100 and 150 kg per hectare give yields of 2.0, 3.2, 3.8 and 3.6 tonnes. A straight line fitted to these points has slope about 0.011 tonnes per kg and predicts yield always rising, missing the fall after 100. A degree-2 fit gives approximately ŷ = 1.99 + 0.0318x − 0.000140x². Its peak is where the slope 0.0318 − 2 × 0.000140x equals zero, at x ≈ 114 kg, with predicted yield about 3.8 tonnes. That turning point is useful advice that the straight line could never give.
In code
import numpy as np
from sklearn.linear_model import LinearRegression, RidgeCV
from sklearn.model_selection import cross_val_score
from sklearn.pipeline import make_pipeline
from sklearn.preprocessing import PolynomialFeatures, StandardScaler
doses = np.array([0, 50, 100, 150])
yields = np.array([2.0, 3.2, 3.8, 3.6])
quad = np.polyfit(doses, yields, 2) # coefficients for x^2, x, 1
peak = -quad[1] / (2 * quad[0])
print(f"Quadratic fit: y = {quad[2]:.2f} + {quad[1]:.4f}x - {abs(quad[0]):.6f}x^2, best dose about {peak:.0f} kg")
rng = np.random.default_rng(0)
x = np.sort(rng.uniform(0, 200, 40)).reshape(-1, 1)
y = 2 + 0.03 * x[:, 0] - 0.00012 * x[:, 0] ** 2 + rng.normal(0, 0.25, 40)
print("Degree plain-CV-MSE ridge-CV-MSE")
for degree in [1, 2, 3, 12]:
plain = make_pipeline(PolynomialFeatures(degree), StandardScaler(), LinearRegression())
ridge = make_pipeline(PolynomialFeatures(degree), StandardScaler(), RidgeCV(alphas=np.logspace(-4, 2, 13)))
s1 = -cross_val_score(plain, x, y, cv=5, scoring="neg_mean_squared_error").mean()
s2 = -cross_val_score(ridge, x, y, cv=5, scoring="neg_mean_squared_error").mean()
print(f"{degree:>6} {s1:12.3f} {s2:12.3f}")Output
Quadratic fit: y = 1.99 + 0.0318x - 0.000140x^2, best dose about 114 kg
Degree plain-CV-MSE ridge-CV-MSE
1 0.613 0.525
2 0.095 0.126
3 0.397 0.190
12 3842990.362 0.472Try it yourself
Finding the best fertiliser dose
An agricultural extension officer analyses trial plots from twenty farms.
Ridge or lasso?
Which penalty suits each situation better?
You suspect only a few of 200 features matter
You want a short list of features to explain to managers
You only want to stabilise a polynomial fit
Many small, real effects from correlated features
Common mistakes
- Choosing a high degree because training error is lowest.
- Regularising without scaling the features first.
- Extrapolating a polynomial far outside the data range.
Check yourself
Summary
- Polynomial features let linear models fit curves; the degree sets flexibility.
- Ridge shrinks weights, lasso zeroes some, elastic net mixes both.
- Scale first, choose degree and λ by cross-validation, and do not extrapolate.
Finished reading?
Sources: James, Witten, Hastie and Tibshirani, An Introduction to Statistical Learning (free book) · scikit-learn user guide