I5 · Machine Learning Foundations

Polynomial Regression and Regularisation

Polynomial regression fits a curve by adding powers of a feature, such as x² and x³, as extra columns and then using ordinary linear regression on them: ŷ = w₀ + w₁x + w₂x² + … + w_dx^d. The degree d controls flexibility. Regularisation adds a penalty on the size of the weights to the loss: ridge (L2) adds λΣw², shrinking all weights; lasso (L1) adds λΣ|w|, which can set some weights exactly to zero. The penalty strength λ (called alpha in scikit-learn) is chosen by validation.

12 min Intermediate · Sublevel Intermediate Jump to the activity

What you need: The earlier topics of this module.

Definition

Crop yield rises as you add fertiliser, but only up to a point; then too much fertiliser burns the plants and yield falls. A straight line cannot show that. A curve can, but a curve that is too wiggly will chase noise. This page shows how to bend a line just enough.

Polynomial regression fits a curve by adding powers of a feature, such as x² and x³, as extra columns and then using ordinary linear regression on them: ŷ = w₀ + w₁x + w₂x² + … + w_dx^d. The degree d controls flexibility. Regularisation adds a penalty on the size of the weights to the loss: ridge (L2) adds λΣw², shrinking all weights; lasso (L1) adds λΣ|w|, which can set some weights exactly to zero. The penalty strength λ (called alpha in scikit-learn) is chosen by validation.

In simple words

Think of a flexible ruler. A stiff ruler (degree 1) can only make a straight line. A bendy ruler (high degree) can follow every bump, including random ones. Regularisation is like adding a spring that pulls the ruler back towards straight, so it bends only where the data really demands.

Overview

  • Polynomial regression adds powers of features so that a linear model can fit curves.
  • The degree controls flexibility: too low underfits, too high overfits.
  • Regularisation adds a penalty on weight size to the loss, discouraging extreme weights.
  • The strength λ (alpha in scikit-learn) is chosen by cross-validation on a log scale.
  • Features must be scaled before regularising.
  • Polynomials should not be used far outside the range of the training data.

Why it matters

Many real relationships are curved: dose and response, speed and fuel use, age and income. Polynomial features and regularisation let simple, explainable linear models capture curves without overfitting, and the same penalty idea appears everywhere in machine learning, from logistic regression to neural networks.

Key terms

Polynomial feature
A new column such as x², x³ or x₁x₂ created from existing features.
Degree
The highest power used; it sets how flexible the curve is.
Ridge regression (L2)
Least squares plus λ times the sum of squared weights.
Lasso regression (L1)
Least squares plus λ times the sum of absolute weights; performs feature selection.
Elastic net
A blend of L1 and L2 penalties.
Regularisation strength (λ, alpha)
How strongly large weights are penalised; larger means simpler models.

Types

There are several types of regularisation, and polynomial features come in a few forms. These are the main types.

1. Polynomial and interaction features

Extra columns built from powers and products of features.

  • How it works: PolynomialFeatures(degree=2) turns (x₁, x₂) into (1, x₁, x₂, x₁², x₁x₂, x₂²); interaction_only=True keeps only products.
  • Example: Fertiliser × rainfall capturing that fertiliser helps more in a good monsoon.
  • When to use it: Use them for curved effects and interactions, keeping the degree low (2 or 3).

2. Ridge regression (L2)

Least squares plus λ times the sum of squared weights.

  • How it works: Large weights are shrunk smoothly towards zero but rarely reach it; correlated features share weight.
  • Example: Ridge keeping a degree-12 polynomial's cross-validated error at 0.47 instead of millions.
  • When to use it: Use it as the default regulariser when many features each matter a little.

3. Lasso regression (L1)

Least squares plus λ times the sum of absolute weights.

  • How it works: The absolute penalty pushes weak weights exactly to zero, so lasso selects features.
  • Example: Reducing 300 gene measurements to the 15 that predict a disease.
  • When to use it: Use it when you suspect only a few features matter and want a simpler model.

4. Elastic net

A mix of L1 and L2 penalties controlled by l1_ratio.

  • How it works: It selects features like lasso but handles groups of correlated features more stably, like ridge.
  • Example: Predicting house prices from many overlapping location features.
  • When to use it: Use it when features are many and correlated and you also want sparsity.

5. Early stopping and other regularisers

Other ways to limit model complexity: stopping training early, dropout and limiting tree depth.

  • How it works: Early stopping halts when validation error stops improving; dropout randomly disables neurons; max_depth caps tree size.
  • Example: Gradient boosting stopped at 48 trees when validation error began to rise.
  • When to use it: Use the regulariser that fits the model family: penalties for linear models, early stopping for boosting and networks.

Explained in detail

Fitting curves with a linear model

The model ŷ = w₀ + w₁x + w₂x² is still called linear regression because it is linear in the weights; only the inputs have been transformed. In scikit-learn, PolynomialFeatures(degree=2) turns a column x into [1, x, x²], and LinearRegression then fits the weights as usual. With two features, degree 2 also creates the interaction x₁x₂, which lets the effect of one feature depend on the other, for example fertiliser helping more when rainfall is good. The number of new columns grows fast: 10 features at degree 3 produce 286 columns. Always scale features before raising them to powers, because a value like 1,000 cubed is a billion.

Choosing the degree

DegreeTraining errorValidation errorDiagnosis
1HighHighUnderfits the curve
2 or 3LowerLowestA good fit
12Very lowHighOverfits: wiggles between points

How regularisation tames a flexible model

A high-degree polynomial overfits because its weights become huge and of opposite signs, cancelling each other at training points and swinging wildly between them. Ridge regression adds λΣw² to the squared error, so the model must justify every large weight with a real reduction in error. As λ grows, weights shrink and the curve smooths out; at λ = 0 we get ordinary least squares. Lasso's penalty λΣ|w| behaves differently: as λ grows, weights hit exactly zero one by one, leaving a simpler model that uses only some features, which helps when many features are irrelevant. Elastic net mixes the two and works well when features are correlated.

Choose λ, and the degree, by cross-validation: try values on a logarithmic scale such as 0.001, 0.01, 0.1, 1, 10 and 100, and keep the one with the lowest validation error. Scikit-learn's RidgeCV and LassoCV do this automatically. Features must be standardised before regularising, otherwise a feature measured in rupees and one measured in lakhs would be penalised very differently. Finally, never extrapolate a polynomial far outside the range of the training data: a curve fitted to fertiliser doses between 50 and 150 kilograms can predict nonsense at 300. In the code below, plain least squares does best at degree 2 (cross-validated error 0.095) and collapses at degree 12 (an error in the millions), while ridge with λ chosen by cross-validation keeps the degree-12 error at 0.47.

How it works

  1. Plot the data to see whether the relationship is curved.
  2. Scale the features, then add polynomial features of a chosen degree.
  3. Fit ridge or lasso instead of plain least squares.
  4. Choose the degree and λ by cross-validation.
  5. Check the final curve on test data and avoid extrapolating.

The maths

Polynomial model
Ridge loss
Regularisation strength; larger means smaller weights
Lasso loss

Fertiliser doses of 0, 50, 100 and 150 kg per hectare give yields of 2.0, 3.2, 3.8 and 3.6 tonnes. A straight line fitted to these points has slope about 0.011 tonnes per kg and predicts yield always rising, missing the fall after 100. A degree-2 fit gives approximately ŷ = 1.99 + 0.0318x − 0.000140x². Its peak is where the slope 0.0318 − 2 × 0.000140x equals zero, at x ≈ 114 kg, with predicted yield about 3.8 tonnes. That turning point is useful advice that the straight line could never give.

In code

poly_ridge.py
import numpy as np
from sklearn.linear_model import LinearRegression, RidgeCV
from sklearn.model_selection import cross_val_score
from sklearn.pipeline import make_pipeline
from sklearn.preprocessing import PolynomialFeatures, StandardScaler

doses = np.array([0, 50, 100, 150])
yields = np.array([2.0, 3.2, 3.8, 3.6])
quad = np.polyfit(doses, yields, 2)                      # coefficients for x^2, x, 1
peak = -quad[1] / (2 * quad[0])
print(f"Quadratic fit: y = {quad[2]:.2f} + {quad[1]:.4f}x - {abs(quad[0]):.6f}x^2, best dose about {peak:.0f} kg")

rng = np.random.default_rng(0)
x = np.sort(rng.uniform(0, 200, 40)).reshape(-1, 1)
y = 2 + 0.03 * x[:, 0] - 0.00012 * x[:, 0] ** 2 + rng.normal(0, 0.25, 40)
print("Degree  plain-CV-MSE  ridge-CV-MSE")
for degree in [1, 2, 3, 12]:
    plain = make_pipeline(PolynomialFeatures(degree), StandardScaler(), LinearRegression())
    ridge = make_pipeline(PolynomialFeatures(degree), StandardScaler(), RidgeCV(alphas=np.logspace(-4, 2, 13)))
    s1 = -cross_val_score(plain, x, y, cv=5, scoring="neg_mean_squared_error").mean()
    s2 = -cross_val_score(ridge, x, y, cv=5, scoring="neg_mean_squared_error").mean()
    print(f"{degree:>6}  {s1:12.3f}  {s2:12.3f}")

Output

Quadratic fit: y = 1.99 + 0.0318x - 0.000140x^2, best dose about 114 kg
Degree  plain-CV-MSE  ridge-CV-MSE
     1         0.613         0.525
     2         0.095         0.126
     3         0.397         0.190
    12   3842990.362         0.472

Try it yourself

Finding the best fertiliser dose

An agricultural extension officer analyses trial plots from twenty farms.

Ridge or lasso?

Which penalty suits each situation better?

  • You suspect only a few of 200 features matter

  • You want a short list of features to explain to managers

  • You only want to stabilise a polynomial fit

  • Many small, real effects from correlated features

Common mistakes

  • Choosing a high degree because training error is lowest.
  • Regularising without scaling the features first.
  • Extrapolating a polynomial far outside the data range.

Check yourself

Question 1

1. Why is ŷ = w₀ + w₁x + w₂x² still called linear regression?

Question 2

2. What does increasing λ in ridge regression do?

Question 3

3. Which penalty can set some weights exactly to zero?

Summary

  • Polynomial features let linear models fit curves; the degree sets flexibility.
  • Ridge shrinks weights, lasso zeroes some, elastic net mixes both.
  • Scale first, choose degree and λ by cross-validation, and do not extrapolate.

Finished reading?

Sources: James, Witten, Hastie and Tibshirani, An Introduction to Statistical Learning (free book) · scikit-learn user guide