A1 · Supervised Machine Learning in Depth

Linear Regression and Least Squares

Linear regression models the target as a weighted sum of the features plus an intercept. Ordinary least squares chooses the weights that minimise the sum of squared residuals. With the feature matrix X (including a column of ones), the minimiser satisfies the normal equation XᵀXw = Xᵀy, giving w = (XᵀX)⁻¹Xᵀy when XᵀX is invertible.

20 min Advanced Jump to the activity

What you need: The earlier topics of this module, and its prerequisites.

Definition

With ten features instead of one, what is the 'best' line, and can you find it in one step?

Linear regression models the target as a weighted sum of the features plus an intercept. Ordinary least squares chooses the weights that minimise the sum of squared residuals. With the feature matrix X (including a column of ones), the minimiser satisfies the normal equation XᵀXw = Xᵀy, giving w = (XᵀX)⁻¹Xᵀy when XᵀX is invertible.

In simple words

Each prediction is ŷ = w₀ + w₁x₁ + … + wₚxₚ. The error of the whole fit is the sum of squared gaps between predictions and true values: a bowl-shaped function of the weights. The normal equation finds the bottom of the bowl directly, by setting the gradient to zero.

Why it matters

Least squares is the foundation of regression analysis, the default baseline for numeric prediction, and the starting point for ridge, lasso, logistic regression and neural networks. Its coefficients are directly interpretable.

Key terms

Linear model
A prediction that is a weighted sum of features plus an intercept: ŷ = w₀ + w₁x₁ + … + wₚxₚ.
Residual
The difference between an actual value and the prediction, y − ŷ.
Ordinary least squares (OLS)
Choosing weights that minimise the sum of squared residuals.
Normal equation
The closed-form OLS solution w = (XᵀX)⁻¹Xᵀy.
R²
The share of variance in y explained by the model; 1 is perfect, 0 is no better than the mean.
Multicollinearity
Strong correlation between features, which makes individual coefficients unstable.

Explained in detail

Fitting a line by least squares

Linear regression assumes the target is approximately a straight-line function of the features. For one feature, the model ŷ = w₀ + w₁x has a slope w₁, the change in y for a one-unit increase in x, and an intercept w₀. Least squares chooses w₀ and w₁ to make the squared residuals as small as possible in total. For one feature the answer is w₁ = Σ(x − x̄)(y − ȳ) ÷ Σ(x − x̄)² and w₀ = ȳ − w₁x̄. For example, with flat areas of 50, 60, 80 and 100 square metres and prices of 30, 35, 46 and 58 lakh rupees, the mean area is 72.5 and the mean price 42.25; the slope works out to about 0.56 lakh per square metre and the intercept to about 1.6 lakh. Squaring residuals makes the problem smooth to solve, but also means a single extreme point pulls the line strongly.

Assumptions and diagnostics

AssumptionWhat breaks itHow to check or fix
LinearityCurved relationshipsPlot residuals against predictions; add polynomial or log features
Independent errorsTime series, repeated measuresCheck autocorrelation; use time-aware models
Constant varianceErrors grow with the size of y, as with pricesLog-transform the target; robust standard errors
No strong outliersData errors or rare extreme casesInspect leverage; robust regression such as Huber
Low multicollinearityArea in square feet and square metres togetherVariance inflation factor; drop or combine features; ridge

For prediction, mild violations matter less than for interpreting coefficients. A coefficient tells you the expected change in y for a one-unit change in that feature, holding the others fixed, which is not the same as a causal effect: if better schools are near expensive homes, the "school" coefficient mixes school quality with location. Linear regression remains valuable because it is fast, transparent and a strong baseline, and many advanced methods, such as ridge, lasso, generalised linear models and neural networks' final layers, build directly on it.

Linear regression is used widely in practice: economists estimate how demand responds to price, agronomists relate yield to rainfall and fertiliser, and companies forecast sales from advertising spend. Even when a more complex model is deployed, a linear model is often kept alongside it as a sanity check and as a simple explanation for stakeholders.

How it works

  1. Write the loss J(w) = ‖y − Xw‖², the squared distance between targets and predictions.
  2. Its gradient is ∇J = −2Xᵀ(y − Xw).
  3. Setting the gradient to zero gives XᵀXw = Xᵀy.
  4. Solve it with a stable method (np.linalg.lstsq or QR) rather than an explicit inverse.

The maths

The model
The n × (p + 1) feature matrix, with a first column of ones
The weights, including the intercept
The least-squares loss
The normal equation and its solution

In code

The normal equation by hand, a numerically stable solver, and scikit-learn all give the same weights.

least_squares.py
import numpy as np
from sklearn.linear_model import LinearRegression

rng = np.random.default_rng(1)
n = 200
X_raw = rng.normal(size=(n, 2))
y = 4.0 + 3.0 * X_raw[:, 0] - 2.0 * X_raw[:, 1] + rng.normal(scale=0.5, size=n)

X = np.column_stack([np.ones(n), X_raw])           # add the intercept column
w_normal = np.linalg.inv(X.T @ X) @ X.T @ y         # textbook formula
w_lstsq = np.linalg.lstsq(X, y, rcond=None)[0]      # what you should use in practice
model = LinearRegression().fit(X_raw, y)

print("True weights:      [4.0, 3.0, -2.0]")
print("Normal equation:  ", np.round(w_normal, 3))
print("lstsq:            ", np.round(w_lstsq, 3))
print("scikit-learn:     ", np.round([model.intercept_, *model.coef_], 3))
residuals = y - X @ w_lstsq
print("Mean squared residual:", round(float(np.mean(residuals ** 2)), 3))

Output

True weights:      [4.0, 3.0, -2.0]
Normal equation:   [ 4.018  3.013 -1.994]
lstsq:             [ 4.018  3.013 -1.994]
scikit-learn:      [ 4.018  3.013 -1.994]
Mean squared residual: 0.259

Try it yourself

Derive the normal equation

Put the steps of the derivation in order.

  1. 1Solve for w = (XᵀX)⁻¹Xᵀy
  2. 2Differentiate: ∇J = −2Xᵀy + 2XᵀXw
  3. 3Write the loss J(w) = (y − Xw)ᵀ(y − Xw)
  4. 4Set ∇J = 0 to get XᵀXw = Xᵀy
  5. 5Expand: J = yᵀy − 2wᵀXᵀy + wᵀXᵀXw

Common mistakes

  • Forgetting the column of ones, which forces the line through the origin.
  • Computing an explicit inverse for large or ill-conditioned problems.
  • Reading a coefficient as causal. It is an association, holding other features fixed.

Check yourself

Question 1

1. What does least squares minimise?

Question 2

2. When does the normal equation have a unique solution?

Question 3

3. Why prefer np.linalg.lstsq to inv(XᵀX)?

Summary

  • Linear regression predicts ŷ = Xw; least squares minimises ‖y − Xw‖².
  • The normal equation XᵀXw = Xᵀy gives the solution in closed form.
  • Solve it stably, add an intercept column, and watch for collinearity.

Finished reading?

Sources: James et al., An Introduction to Statistical Learning · Hastie et al., The Elements of Statistical Learning