Definition
Linear regression is a supervised learning algorithm that predicts a number by fitting a straight line (or a flat plane, with many features) through the data. It chooses the line that minimises the average squared difference between predictions and true values.
In simple words
Plot flat sizes against prices, and the dots roughly form a rising band. Draw the straight line that passes as close as possible to all the dots. To price a new flat, find its size on that line and read off the price. The whole job of the algorithm is to find the best line.
Overview
- Linear regression predicts a number as a weighted sum of features plus an intercept: ŷ = w₀ + w₁x₁ + … + wₙxₙ.
- Ordinary least squares chooses the weights that minimise the sum of squared errors.
- Each weight is the change in the prediction for a one-unit change in that feature, holding others fixed.
- Assumptions include a linear relationship, independent errors and roughly constant error variance.
- It is fast, explainable and a strong baseline for numeric prediction.
- Least squares was published by Legendre in 1805 and Gauss in 1809.
Why it matters
It's the simplest useful model, it's fast, and you can read exactly what it learned. It's usually the first baseline for any number-prediction problem. It's also the base of logistic regression and of neural networks, since a neuron is linear regression plus an activation.
Key terms
- Linear regression
- Predicting a number as a weighted sum of features plus an intercept.
- Coefficient or weight
- How much the prediction changes for a one-unit increase in a feature, others held fixed.
- Intercept
- The predicted value when every feature is zero.
- Residual
- The difference between the true value and the prediction.
- Mean squared error
- The average of squared residuals, the quantity ordinary least squares minimises.
- R²
- The share of the variation in the target explained by the model, from 0 to 1 on the training data.
Types
Linear regression has several types, from one feature to regularised and generalised forms.
1. Simple linear regression
One feature predicting one target with a straight line.
- How it works: The slope is cov(x, y) ÷ var(x) and the line passes through the means of x and y.
- Example: Predicting marks from hours studied: marks = 32 + 5.5 × hours.
- When to use it: Use it to understand and visualise a single relationship.
2. Multiple linear regression
Several features predicting one target with a flat plane or hyperplane.
- How it works: The weights are found by least squares, usually solved with linear algebra or gradient descent.
- Example: House price from area, rooms, age and distance to the metro.
- When to use it: Use it as the standard baseline for numeric prediction with tabular data.
3. Polynomial regression
Linear regression on powers of features, to fit curves.
- How it works: PolynomialFeatures adds x², x³ and interactions; the model is still linear in its weights.
- Example: Crop yield peaking at about 114 kg of fertiliser per hectare.
- When to use it: Use it for curved relationships, with regularisation and cross-validated degree. Keep the degree low and check validation error.
4. Regularised regression (ridge, lasso, elastic net)
Least squares with a penalty on the size of the weights.
- How it works: Ridge adds λΣw² and shrinks all weights; lasso adds λΣ|w| and sets some to zero; elastic net mixes both.
- Example: Lasso keeping 8 of 200 survey questions that really predict spending.
- When to use it: Use them when there are many or correlated features, to prevent overfitting.
5. Robust and generalised linear models
Variants for outliers (Huber regression) and for non-normal targets such as counts (Poisson regression).
- How it works: Huber loss treats large errors linearly so outliers pull less; generalised linear models link the linear part to the target through a function such as log.
- Example: Predicting the number of hospital visits per year with Poisson regression.
- When to use it: Use them when errors contain outliers or the target is a count or a positive skewed amount.
Explained in detail
The model
Linear regression predicts ŷ = w₁x₁ + w₂x₂ + … + wₙxₙ + b. For house prices, it might learn price = 4,500 × area + 2,00,000 × bedrooms − 30,000 × age + 5,00,000 (in rupees). Each coefficient has a direct meaning: holding the others fixed, each extra square foot adds about ₹4,500 to the predicted price. That readability is linear regression's great strength and the reason it remains widely used in economics, medicine and business, even though more powerful models exist. The method of least squares behind it was published by Adrien-Marie Legendre in 1805 and by Carl Friedrich Gauss soon after.
How the best line is found
For any line, each data point has a residual, the vertical distance between the point and the line. Ordinary least squares chooses the weights that minimise the mean of the squared residuals. Squaring makes all errors positive and punishes large errors heavily. There is an exact formula for the best weights, the normal equation, but for very large data sets the same answer is found iteratively by gradient descent, which is also how neural networks are trained. scikit-learn's LinearRegression uses an efficient exact solver.
Assumptions and how to check them
| Assumption | What it means | How to check |
|---|---|---|
| Linearity | The relationship is roughly a straight line | Plot each feature against the target; plot residuals against predictions |
| Independent errors | One residual does not predict the next | Look for patterns over time |
| Constant spread | Residuals are similar in size across predictions | A funnel-shaped residual plot signals a problem |
| No extreme multicollinearity | Features are not near-copies of each other | Check the correlation matrix |
Extending the model
- Polynomial features, such as area², let a linear model fit curves.
- Log-transforming a skewed target, such as price, often makes the relationship more linear.
- Ridge regression adds an L2 penalty to shrink coefficients and reduce overfitting; lasso adds an L1 penalty that can set some coefficients to zero.
- Coefficients describe association, not causation; a coefficient for "near a metro station" does not prove the station causes higher prices.
How it works
- Pick a line with a slope w and an intercept b.
- Measure the error: how far each point is from the line, squared and averaged.
- Adjust w and b to reduce the error, with gradient descent or the direct formula.
Assumptions
- The relationship is roughly a straight line.
- Errors are independent of each other, with similar spread throughout.
- Features aren't near-duplicates of each other, or coefficients become unstable.
The maths
- The prediction
- The slope: how much y changes when x rises by 1
- The intercept: the prediction when x is 0
- The learning rate
Points: (1, 2), (2, 4), (3, 5). The means are x̄ = 2 and ȳ = 3.67.
Σ(x−x̄)(y−ȳ) = (−1)(−1.67) + (0)(0.33) + (1)(1.33) = 3 Σ(x−x̄)² = 1 + 0 + 1 = 2 w = 3 / 2 = 1.5 b = 3.67 − 1.5 × 2 = 0.67 The line: ŷ = 1.5x + 0.67 Prediction for x = 4: ŷ = 6.67
In code
First from scratch, using the formula above, then with scikit-learn. Both find the same line.
import numpy as np
from sklearn.linear_model import LinearRegression
x = np.array([1, 2, 3])
y = np.array([2, 4, 5])
w = np.sum((x - x.mean()) * (y - y.mean())) / np.sum((x - x.mean()) ** 2)
b = y.mean() - w * x.mean()
print(f"From scratch: y = {w:.2f}x + {b:.2f}")
model = LinearRegression().fit(x.reshape(-1, 1), y)
print(f"scikit-learn: y = {model.coef_[0]:.2f}x + {model.intercept_:.2f}")
print("Prediction for x = 4:", round(model.predict([[4]])[0], 2))Output
From scratch: y = 1.50x + 0.67 scikit-learn: y = 1.50x + 0.67 Prediction for x = 4: 6.67
Common mistakes
- Using it on a clearly curved relationship. Plot the data first.
- Reading a coefficient as cause and effect.
- Predicting far outside the range of the training data.
Check yourself
Summary
- It fits the straight line ŷ = wx + b.
- It minimises the mean squared error.
- Use the formula or gradient descent, and use it as your first baseline.
Read it. Now try it yourself.
Open the interactive activityFinished reading?
Sources: scikit-learn user guide · Google Machine Learning Crash Course