Definition
With ten features instead of one, what is the 'best' line, and can you find it in one step?
Linear regression models the target as a weighted sum of the features plus an intercept. Ordinary least squares chooses the weights that minimise the sum of squared residuals. With the feature matrix X (including a column of ones), the minimiser satisfies the normal equation XᵀXw = Xᵀy, giving w = (XᵀX)⁻¹Xᵀy when XᵀX is invertible.
In simple words
Each prediction is ŷ = w₀ + w₁x₁ + … + wₚxₚ. The error of the whole fit is the sum of squared gaps between predictions and true values: a bowl-shaped function of the weights. The normal equation finds the bottom of the bowl directly, by setting the gradient to zero.
Why it matters
Least squares is the foundation of regression analysis, the default baseline for numeric prediction, and the starting point for ridge, lasso, logistic regression and neural networks. Its coefficients are directly interpretable.
Key terms
- Linear model
- A prediction that is a weighted sum of features plus an intercept: ŷ = w₀ + w₁x₁ + … + wₚxₚ.
- Residual
- The difference between an actual value and the prediction, y − ŷ.
- Ordinary least squares (OLS)
- Choosing weights that minimise the sum of squared residuals.
- Normal equation
- The closed-form OLS solution w = (XᵀX)⁻¹Xᵀy.
- R²
- The share of variance in y explained by the model; 1 is perfect, 0 is no better than the mean.
- Multicollinearity
- Strong correlation between features, which makes individual coefficients unstable.
Explained in detail
Fitting a line by least squares
Linear regression assumes the target is approximately a straight-line function of the features. For one feature, the model ŷ = w₀ + w₁x has a slope w₁, the change in y for a one-unit increase in x, and an intercept w₀. Least squares chooses w₀ and w₁ to make the squared residuals as small as possible in total. For one feature the answer is w₁ = Σ(x − x̄)(y − ȳ) ÷ Σ(x − x̄)² and w₀ = ȳ − w₁x̄. For example, with flat areas of 50, 60, 80 and 100 square metres and prices of 30, 35, 46 and 58 lakh rupees, the mean area is 72.5 and the mean price 42.25; the slope works out to about 0.56 lakh per square metre and the intercept to about 1.6 lakh. Squaring residuals makes the problem smooth to solve, but also means a single extreme point pulls the line strongly.
Assumptions and diagnostics
| Assumption | What breaks it | How to check or fix |
|---|---|---|
| Linearity | Curved relationships | Plot residuals against predictions; add polynomial or log features |
| Independent errors | Time series, repeated measures | Check autocorrelation; use time-aware models |
| Constant variance | Errors grow with the size of y, as with prices | Log-transform the target; robust standard errors |
| No strong outliers | Data errors or rare extreme cases | Inspect leverage; robust regression such as Huber |
| Low multicollinearity | Area in square feet and square metres together | Variance inflation factor; drop or combine features; ridge |
For prediction, mild violations matter less than for interpreting coefficients. A coefficient tells you the expected change in y for a one-unit change in that feature, holding the others fixed, which is not the same as a causal effect: if better schools are near expensive homes, the "school" coefficient mixes school quality with location. Linear regression remains valuable because it is fast, transparent and a strong baseline, and many advanced methods, such as ridge, lasso, generalised linear models and neural networks' final layers, build directly on it.
Linear regression is used widely in practice: economists estimate how demand responds to price, agronomists relate yield to rainfall and fertiliser, and companies forecast sales from advertising spend. Even when a more complex model is deployed, a linear model is often kept alongside it as a sanity check and as a simple explanation for stakeholders.
How it works
- Write the loss J(w) = ‖y − Xw‖², the squared distance between targets and predictions.
- Its gradient is ∇J = −2Xᵀ(y − Xw).
- Setting the gradient to zero gives XᵀXw = Xᵀy.
- Solve it with a stable method (np.linalg.lstsq or QR) rather than an explicit inverse.
The maths
- The n × (p + 1) feature matrix, with a first column of ones
- The weights, including the intercept
In code
The normal equation by hand, a numerically stable solver, and scikit-learn all give the same weights.
import numpy as np
from sklearn.linear_model import LinearRegression
rng = np.random.default_rng(1)
n = 200
X_raw = rng.normal(size=(n, 2))
y = 4.0 + 3.0 * X_raw[:, 0] - 2.0 * X_raw[:, 1] + rng.normal(scale=0.5, size=n)
X = np.column_stack([np.ones(n), X_raw]) # add the intercept column
w_normal = np.linalg.inv(X.T @ X) @ X.T @ y # textbook formula
w_lstsq = np.linalg.lstsq(X, y, rcond=None)[0] # what you should use in practice
model = LinearRegression().fit(X_raw, y)
print("True weights: [4.0, 3.0, -2.0]")
print("Normal equation: ", np.round(w_normal, 3))
print("lstsq: ", np.round(w_lstsq, 3))
print("scikit-learn: ", np.round([model.intercept_, *model.coef_], 3))
residuals = y - X @ w_lstsq
print("Mean squared residual:", round(float(np.mean(residuals ** 2)), 3))Output
True weights: [4.0, 3.0, -2.0] Normal equation: [ 4.018 3.013 -1.994] lstsq: [ 4.018 3.013 -1.994] scikit-learn: [ 4.018 3.013 -1.994] Mean squared residual: 0.259
Try it yourself
Derive the normal equation
Put the steps of the derivation in order.
- 1Solve for w = (XᵀX)⁻¹Xᵀy
- 2Differentiate: ∇J = −2Xᵀy + 2XᵀXw
- 3Write the loss J(w) = (y − Xw)ᵀ(y − Xw)
- 4Set ∇J = 0 to get XᵀXw = Xᵀy
- 5Expand: J = yᵀy − 2wᵀXᵀy + wᵀXᵀXw
Common mistakes
- Forgetting the column of ones, which forces the line through the origin.
- Computing an explicit inverse for large or ill-conditioned problems.
- Reading a coefficient as causal. It is an association, holding other features fixed.
Check yourself
Summary
- Linear regression predicts ŷ = Xw; least squares minimises ‖y − Xw‖².
- The normal equation XᵀXw = Xᵀy gives the solution in closed form.
- Solve it stably, add an intercept column, and watch for collinearity.
Finished reading?
Sources: James et al., An Introduction to Statistical Learning · Hastie et al., The Elements of Statistical Learning