A1 · Supervised Machine Learning in Depth

Regularisation: Ridge, Lasso and Elastic Net

Regularisation adds a penalty on the size of the weights to the loss, trading a little bias for a large reduction in variance. Ridge (L2) penalises the sum of squared weights and shrinks them smoothly. Lasso (L1) penalises the sum of absolute weights and drives some exactly to zero, performing feature selection. Elastic Net combines both. The strength λ (alpha in scikit-learn) is tuned by validation.

15 min Advanced Jump to the activity

What you need: The earlier topics of this module, and its prerequisites.

Definition

How do you stop a flexible model from memorising the noise?

Regularisation adds a penalty on the size of the weights to the loss, trading a little bias for a large reduction in variance. Ridge (L2) penalises the sum of squared weights and shrinks them smoothly. Lasso (L1) penalises the sum of absolute weights and drives some exactly to zero, performing feature selection. Elastic Net combines both. The strength λ (alpha in scikit-learn) is tuned by validation.

In simple words

An overfitted model uses huge positive and negative weights that cancel out on the training data. The penalty says 'every unit of weight costs you something', so the model keeps only weights that genuinely reduce error. Lasso is stricter: it would rather switch a feature off completely.

Why it matters

Regularisation is how almost every modern model, from linear regression to large neural networks with weight decay, stays general. Lasso also gives sparse, interpretable models when you have many candidate features.

Key terms

Regularisation
Adding a penalty on model complexity to the training objective to reduce overfitting.
Ridge (L2)
Penalises the sum of squared coefficients, shrinking them smoothly towards zero.
Lasso (L1)
Penalises the sum of absolute coefficients, setting some exactly to zero.
Elastic net
A mix of L1 and L2 penalties controlled by l1_ratio.
Alpha (λ)
The penalty strength; larger values mean simpler models.
Shrinkage
Pulling coefficients towards zero to trade a little bias for lower variance.

Explained in detail

Why add a penalty?

With many features, few rows or correlated features, ordinary least squares can fit noise: coefficients become very large and swing wildly from one sample to another. Regularisation adds a cost for large coefficients. Ridge minimises the squared errors plus α times the sum of squared weights; lasso adds α times the sum of absolute weights. The penalty trades a small increase in bias for a large decrease in variance, which usually improves performance on new data. The intercept is not penalised, and features must be standardised first, otherwise the penalty treats a coefficient on rupees very differently from one on lakhs.

Ridge, lasso or elastic net?

MethodBehaviourUse when
RidgeShrinks all coefficients; keeps every feature; handles correlated features by sharing weightMany small effects; multicollinearity
LassoSets many coefficients exactly to zero, performing feature selectionYou expect only a few features to matter and want a sparse, readable model
Elastic netSelects features like lasso but keeps groups of correlated features togetherMany correlated features, such as gene or sensor data

Geometrically, ridge constrains the weights to lie inside a circle and lasso inside a diamond; the corners of the diamond touch the axes, which is why lasso solutions often have exact zeros. Choose α by cross-validation, for example with RidgeCV, LassoCV or ElasticNetCV, which try a range of values on a logarithmic scale such as 0.001 to 1,000. Plot the coefficients against α, the regularisation path, to see which features survive as the penalty grows. The same idea appears throughout machine learning: weight decay in neural networks is L2 regularisation, and the C parameter in logistic regression and SVMs is the inverse of the penalty strength.

A worked scenario shows the value. Suppose a hospital wants to predict length of stay from 200 measurements on only 400 patients. Plain least squares fits the training data almost perfectly but its validation error is poor and its coefficients are huge. Ridge with α chosen by cross-validation reduces validation error substantially and keeps all 200 features with small coefficients. Lasso reaches similar accuracy using only about 25 non-zero features, which doctors find much easier to review. Elastic net is chosen in the end, because several laboratory measurements are strongly correlated and lasso kept arbitrarily switching between them from one fold to the next.

How it works

Ridge (L2)Lasso (L1)Elastic Net
Penaltyλ Σ wⱼ²λ Σ |wⱼ|A mix of both
EffectShrinks all weightsSets some weights to exactly 0Sparse, but stable with correlated features
Use whenMany small effects, correlated featuresYou expect few relevant featuresMany correlated features, some irrelevant

The maths

Ridge objective
Ridge closed form
Lasso objective
Regularisation strength: larger means smaller weights

In code

Ridge and Lasso on the diabetes dataset: as alpha grows, ridge shrinks every weight while lasso switches features off.

regularisation.py
import numpy as np
from sklearn.datasets import load_diabetes
from sklearn.linear_model import Lasso, Ridge, RidgeCV
from sklearn.preprocessing import StandardScaler

X, y = load_diabetes(return_X_y=True)
X = StandardScaler().fit_transform(X)

print("alpha   ridge |w| sum   lasso |w| sum   lasso zero weights")
for alpha in [0.01, 1, 10, 50]:
    ridge = Ridge(alpha=alpha).fit(X, y)
    lasso = Lasso(alpha=alpha, max_iter=10000).fit(X, y)
    print(f"{alpha:>5}   {np.abs(ridge.coef_).sum():13.1f}   {np.abs(lasso.coef_).sum():13.1f}   {np.sum(lasso.coef_ == 0):>18}")

best = RidgeCV(alphas=np.logspace(-3, 3, 13)).fit(X, y)
print("Alpha chosen by cross-validation:", best.alpha_)

Output

alpha   ridge |w| sum   lasso |w| sum   lasso zero weights
 0.01           164.3           160.0                    0
    1           143.7            90.7                    3
   10           105.0            52.1                    6
   50            93.8             0.0                   10
Alpha chosen by cross-validation: 1.0

Try it yourself

Which penalty gives zeros?

Assign each property to Ridge or Lasso.

  • Spreads weight across correlated features

  • Has a closed-form solution

  • Sets some weights exactly to zero

  • Performs automatic feature selection

Common mistakes

  • Regularising unscaled features, so the penalty hits some features harder than others.
  • Penalising the intercept. Libraries leave it out by default; your own code should too.
  • Picking alpha by hand instead of cross-validation.

Check yourself

Question 1

1. Which regularisation can remove features entirely?

Question 2

2. As λ increases, the weights…

Question 3

3. Regularisation mainly reduces…

Summary

  • Ridge adds λΣw² and shrinks; Lasso adds λΣ|w| and zeroes.
  • Elastic Net mixes both for correlated features.
  • Standardise first and tune λ by cross-validation.

Read it. Now try it yourself.

Open the interactive activity

Finished reading?

Sources: James et al., An Introduction to Statistical Learning · scikit-learn: Linear models