I5 · Machine Learning Foundations

Measuring Regression Error

Regression metrics measure how far numeric predictions are from the true values. Mean absolute error (MAE) is the average size of the errors. Mean squared error (MSE) averages the squared errors and so punishes large errors heavily; its square root, RMSE, is in the target's units. R² is the share of the target's variance the model explains compared with always predicting the mean.

8 min Intermediate · Sublevel Intermediate Jump to the activity

What you need: The earlier topics of this module.

Definition

Two price models both have an average error of ₹50,000. One is off by ₹50,000 on every house; the other is perfect on most and off by ₹10 lakh on a few. Are they equally good?

Regression metrics measure how far numeric predictions are from the true values. Mean absolute error (MAE) is the average size of the errors. Mean squared error (MSE) averages the squared errors and so punishes large errors heavily; its square root, RMSE, is in the target's units. R² is the share of the target's variance the model explains compared with always predicting the mean.

In simple words

MAE asks "on average, how many rupees off are we?" RMSE asks the same but counts big misses extra. R² asks "how much better are we than just guessing the average price every time?"

Overview

  • Regression metrics measure how far predictions are from true values.
  • The residual of each prediction is the true value minus the predicted value.
  • Different metrics weight large errors differently and use different units.
  • Always compare with a baseline such as predicting the mean.
  • Choose the metric that matches the real cost of errors in the application.
  • Residual plots reveal patterns, such as larger errors for expensive houses.

Why it matters

The metric you choose decides which model looks best and what the model is pushed to get right. Choosing a metric that matches the real cost of mistakes is as important as choosing the algorithm.

Key terms

Residual
True value minus prediction for one example.
MAE
Mean of the absolute residuals, in the target's units.
MSE
Mean of the squared residuals; large errors count heavily.
RMSE
The square root of MSE, back in the target's units.
R²
1 − (model's squared error ÷ squared error of predicting the mean).
MAPE
Mean absolute percentage error: errors as a percentage of the true values.

Types

There are several types of regression metric. These are the most used, with what each tells you.

1. Mean absolute error (MAE)

The average size of the errors, ignoring sign.

  • How it works: MAE = (1/n) Σ |y − ŷ|; it is in the target's units and treats all errors equally.
  • Example: Delivery-time predictions off by 3.2 minutes on average.
  • When to use it: Use it when every minute or rupee of error matters equally and you want an easy-to-explain number.

2. Mean squared error (MSE) and RMSE

The average squared error, and its square root.

  • How it works: Squaring punishes large errors heavily; RMSE = √MSE returns to the target's units.
  • Example: Errors of 2, 2 and 10 give MAE 4.7 but RMSE 6.0, because the 10 dominates.
  • When to use it: Use it when large errors are especially harmful, and as the training loss for linear regression.

3. R² (coefficient of determination)

The share of the target's variance explained by the model.

  • How it works: R² = 1 − SS_res ÷ SS_tot; 1 is perfect, 0 is no better than predicting the mean, and it can be negative on test data.
  • Example: An R² of 0.82 means the model explains 82% of the variation in house prices.
  • When to use it: Use it to compare models on the same data; report it with an error metric in real units.

4. Percentage errors (MAPE)

The average absolute error as a percentage of the true value.

  • How it works: MAPE = (100/n) Σ |y − ŷ| ÷ |y|; it is scale-free but explodes when true values are near zero.
  • Example: A sales forecast that is 6% off on average across products of very different sizes.
  • When to use it: Use it to compare accuracy across series of different scales, avoiding it when values can be zero.

5. Residual analysis

Studying the pattern of errors rather than one summary number.

  • How it works: Plot residuals against predictions and features; random scatter is good, while curves or funnels reveal missing features or changing variance.
  • Example: Residuals growing with house size show the model under-predicts large houses.
  • When to use it: Use it after every regression model, to find where and why it fails.

Explained in detail

A worked comparison

HouseTrue price (₹ lakh)Model A errorModel B error
150+20
260−20
370+20
480−2+8

Both models have the same MAE, 2 lakh. But model A's squared errors are 4, 4, 4, 4, giving an MSE of 4 and RMSE of 2, while model B's are 0, 0, 0, 64, giving an MSE of 16 and RMSE of 4. RMSE reveals that model B occasionally makes a big mistake. Which is better depends on the use. If a large miss is a disaster, as when a bank lends against a valuation, prefer the model with lower RMSE. If every rupee of error costs the same, MAE is the fairer judge.

Understanding R²

R² compares your model with the simplest baseline: predicting the mean of the training targets for every example. An R² of 0 means your model is no better than that baseline; 1 means perfect predictions; 0.8 means it removes 80 per cent of the squared error the baseline makes. On test data R² can even be negative, which means the model is worse than guessing the average and something is badly wrong. R² has no units, so it is handy for comparing across problems, but it says nothing about whether the errors are acceptable in rupees, so always report it alongside MAE or RMSE.

Choosing a metric

  • MAE: easy to explain, robust to a few huge errors.
  • RMSE: penalises large errors; the default for many competitions and for models trained with squared loss.
  • MAPE: errors as percentages, useful when targets vary hugely in size, but it breaks down when true values are near zero.
  • R²: a quick unit-free summary, but never the only number.
  • Residual plots: plotting errors against predictions shows patterns no single number reveals, such as larger errors for expensive houses.

Whatever metric you choose, compute it on the same test set for every model, and compare it with the baseline score. A model whose MAE is ₹5 lakh sounds good until you learn that predicting the median gives ₹6 lakh; it sounds excellent if the median gives ₹20 lakh. Numbers only become meaningful through comparison, which is why every serious evaluation report includes a baseline row.

How it works

  1. Predict on the test set.
  2. Compute MAE, RMSE and R² with sklearn.metrics.
  3. Compare with the baseline of predicting the training mean.
  4. Plot residuals to look for patterns.
  5. Report the metric that matches the real cost of errors, in real units.

The maths

Mean absolute error
Root mean squared error
Coefficient of determination
The mean of the true values

In code

regression_metrics.py
import numpy as np
from sklearn.metrics import mean_absolute_error, mean_squared_error, r2_score

true = np.array([50, 60, 70, 80])
model_a = true + np.array([2, -2, 2, -2])
model_b = true + np.array([0, 0, 0, 8])

for name, pred in [("A", model_a), ("B", model_b)]:
    mae = mean_absolute_error(true, pred)
    rmse = np.sqrt(mean_squared_error(true, pred))
    r2 = r2_score(true, pred)
    print(f"Model {name}: MAE={mae:.1f}  RMSE={rmse:.1f}  R2={r2:.3f}")

baseline = np.full_like(true, true.mean(), dtype=float)
print("Baseline R2:", r2_score(true, baseline))

Output

Model A: MAE=2.0  RMSE=2.0  R2=0.968
Model B: MAE=2.0  RMSE=4.0  R2=0.872
Baseline R2: 0.0

Try it yourself

The model that was usually perfect

A property website compares two price models for loan valuations.

Match the metric to what it tells you

What does each metric emphasise?

  • MAE
  • RMSE
  • R²
  • MAPE

Common mistakes

  • Reporting R² alone, with no error in real units.
  • Using MAPE when some true values are zero or tiny.
  • Comparing metrics computed on different test sets.

Check yourself

Question 1

1. Which metric punishes large errors most?

Question 2

2. An R² of 0 on test data means…

Question 3

3. Why is RMSE in the same units as the target?

Summary

  • MAE is the typical error; RMSE weights large errors; R² compares with predicting the mean.
  • Choose the metric that matches the real cost of mistakes.
  • Report errors in real units, compare with a baseline and inspect residuals.

Finished reading?

Sources: scikit-learn: Regression metrics · An Introduction to Statistical Learning (James, Witten, Hastie, Tibshirani)