Definition
How do you stop a flexible model from memorising the noise?
Regularisation adds a penalty on the size of the weights to the loss, trading a little bias for a large reduction in variance. Ridge (L2) penalises the sum of squared weights and shrinks them smoothly. Lasso (L1) penalises the sum of absolute weights and drives some exactly to zero, performing feature selection. Elastic Net combines both. The strength λ (alpha in scikit-learn) is tuned by validation.
In simple words
An overfitted model uses huge positive and negative weights that cancel out on the training data. The penalty says 'every unit of weight costs you something', so the model keeps only weights that genuinely reduce error. Lasso is stricter: it would rather switch a feature off completely.
Why it matters
Regularisation is how almost every modern model, from linear regression to large neural networks with weight decay, stays general. Lasso also gives sparse, interpretable models when you have many candidate features.
Key terms
- Regularisation
- Adding a penalty on model complexity to the training objective to reduce overfitting.
- Ridge (L2)
- Penalises the sum of squared coefficients, shrinking them smoothly towards zero.
- Lasso (L1)
- Penalises the sum of absolute coefficients, setting some exactly to zero.
- Elastic net
- A mix of L1 and L2 penalties controlled by l1_ratio.
- Alpha (λ)
- The penalty strength; larger values mean simpler models.
- Shrinkage
- Pulling coefficients towards zero to trade a little bias for lower variance.
Explained in detail
Why add a penalty?
With many features, few rows or correlated features, ordinary least squares can fit noise: coefficients become very large and swing wildly from one sample to another. Regularisation adds a cost for large coefficients. Ridge minimises the squared errors plus α times the sum of squared weights; lasso adds α times the sum of absolute weights. The penalty trades a small increase in bias for a large decrease in variance, which usually improves performance on new data. The intercept is not penalised, and features must be standardised first, otherwise the penalty treats a coefficient on rupees very differently from one on lakhs.
Ridge, lasso or elastic net?
| Method | Behaviour | Use when |
|---|---|---|
| Ridge | Shrinks all coefficients; keeps every feature; handles correlated features by sharing weight | Many small effects; multicollinearity |
| Lasso | Sets many coefficients exactly to zero, performing feature selection | You expect only a few features to matter and want a sparse, readable model |
| Elastic net | Selects features like lasso but keeps groups of correlated features together | Many correlated features, such as gene or sensor data |
Geometrically, ridge constrains the weights to lie inside a circle and lasso inside a diamond; the corners of the diamond touch the axes, which is why lasso solutions often have exact zeros. Choose α by cross-validation, for example with RidgeCV, LassoCV or ElasticNetCV, which try a range of values on a logarithmic scale such as 0.001 to 1,000. Plot the coefficients against α, the regularisation path, to see which features survive as the penalty grows. The same idea appears throughout machine learning: weight decay in neural networks is L2 regularisation, and the C parameter in logistic regression and SVMs is the inverse of the penalty strength.
A worked scenario shows the value. Suppose a hospital wants to predict length of stay from 200 measurements on only 400 patients. Plain least squares fits the training data almost perfectly but its validation error is poor and its coefficients are huge. Ridge with α chosen by cross-validation reduces validation error substantially and keeps all 200 features with small coefficients. Lasso reaches similar accuracy using only about 25 non-zero features, which doctors find much easier to review. Elastic net is chosen in the end, because several laboratory measurements are strongly correlated and lasso kept arbitrarily switching between them from one fold to the next.
How it works
| Ridge (L2) | Lasso (L1) | Elastic Net | |
|---|---|---|---|
| Penalty | λ Σ wⱼ² | λ Σ |wⱼ| | A mix of both |
| Effect | Shrinks all weights | Sets some weights to exactly 0 | Sparse, but stable with correlated features |
| Use when | Many small effects, correlated features | You expect few relevant features | Many correlated features, some irrelevant |
The maths
- Regularisation strength: larger means smaller weights
In code
Ridge and Lasso on the diabetes dataset: as alpha grows, ridge shrinks every weight while lasso switches features off.
import numpy as np
from sklearn.datasets import load_diabetes
from sklearn.linear_model import Lasso, Ridge, RidgeCV
from sklearn.preprocessing import StandardScaler
X, y = load_diabetes(return_X_y=True)
X = StandardScaler().fit_transform(X)
print("alpha ridge |w| sum lasso |w| sum lasso zero weights")
for alpha in [0.01, 1, 10, 50]:
ridge = Ridge(alpha=alpha).fit(X, y)
lasso = Lasso(alpha=alpha, max_iter=10000).fit(X, y)
print(f"{alpha:>5} {np.abs(ridge.coef_).sum():13.1f} {np.abs(lasso.coef_).sum():13.1f} {np.sum(lasso.coef_ == 0):>18}")
best = RidgeCV(alphas=np.logspace(-3, 3, 13)).fit(X, y)
print("Alpha chosen by cross-validation:", best.alpha_)Output
alpha ridge |w| sum lasso |w| sum lasso zero weights
0.01 164.3 160.0 0
1 143.7 90.7 3
10 105.0 52.1 6
50 93.8 0.0 10
Alpha chosen by cross-validation: 1.0Try it yourself
Which penalty gives zeros?
Assign each property to Ridge or Lasso.
Spreads weight across correlated features
Has a closed-form solution
Sets some weights exactly to zero
Performs automatic feature selection
Common mistakes
- Regularising unscaled features, so the penalty hits some features harder than others.
- Penalising the intercept. Libraries leave it out by default; your own code should too.
- Picking alpha by hand instead of cross-validation.
Check yourself
Summary
- Ridge adds λΣw² and shrinks; Lasso adds λΣ|w| and zeroes.
- Elastic Net mixes both for correlated features.
- Standardise first and tune λ by cross-validation.
Read it. Now try it yourself.
Open the interactive activityFinished reading?
Sources: James et al., An Introduction to Statistical Learning · scikit-learn: Linear models