A1 · Supervised Machine Learning in Depth

Regression Metrics

Regression metrics summarise how far predictions are from true values. MAE is the mean absolute error; MSE the mean squared error; RMSE its square root, in the target's units; and R² the share of the target's variance explained by the model compared with predicting the mean. Residual plots show whether errors have structure the model missed.

10 min Advanced Jump to the activity

What you need: The earlier topics of this module, and its prerequisites.

Definition

Is an error of 5 good or bad?

Regression metrics summarise how far predictions are from true values. MAE is the mean absolute error; MSE the mean squared error; RMSE its square root, in the target's units; and R² the share of the target's variance explained by the model compared with predicting the mean. Residual plots show whether errors have structure the model missed.

In simple words

An error of 5 means nothing until you know the scale: 5 rupees on a phone price is tiny, 5 degrees on body temperature is huge. RMSE keeps the target's units, punishing big misses more. MAE treats every rupee of error equally. R² says how much better you are than always guessing the average.

Why it matters

The metric decides which model wins. A delivery app that hates long delays should look at RMSE; one that cares about typical error, MAE. Reporting R² alone hides whether errors are acceptable in absolute terms.

Key terms

Mean absolute error (MAE)
The average absolute difference between predictions and actual values.
Root mean squared error (RMSE)
The square root of the mean squared error; penalises large errors more.
R² (coefficient of determination)
1 − SS_res ÷ SS_tot: the share of variance explained compared with predicting the mean.
MAPE
Mean absolute percentage error; easy to read but unstable near zero.
Quantile (pinball) loss
A loss for predicting a percentile, used for prediction intervals.
Residual plot
A chart of residuals against predictions to reveal patterns the metric hides.

Explained in detail

Which error measure to use

Suppose a model predicts the prices of four flats as 40, 52, 61 and 75 lakh when the true prices are 42, 50, 60 and 90 lakh. The errors are −2, 2, 1 and −15. MAE is (2 + 2 + 1 + 15) ÷ 4 = 5 lakh: on average we are 5 lakh off. RMSE is √((4 + 4 + 1 + 225) ÷ 4) ≈ 7.6 lakh, higher than MAE because the single 15-lakh miss is squared. The gap between RMSE and MAE is itself informative: a large gap means a few big errors. MAE is easier to explain and less sensitive to outliers; RMSE is better when large errors are disproportionately costly, such as underestimating peak electricity demand. Always report errors in the target's units and alongside a baseline.

R² and its traps

R² compares the model's squared error with that of always predicting the mean. R² = 0.8 means the model removes 80 per cent of that baseline error. It can be negative on test data when the model is worse than the mean. But R² depends on how spread out the target is: the same model can show R² of 0.9 on flats from all over a city and 0.4 on flats from one neighbourhood, with identical rupee errors. Adding features never decreases training R², which is why adjusted R² or, better, validation error should guide feature choice. A high R² also does not prove the relationship is causal or that the model will hold next year.

MetricBest forCaution
MAETypical size of error in plain unitsTreats all errors linearly
RMSEWhen large errors are especially harmfulDominated by a few outliers
MAPEComparing series of different scalesUndefined at zero; penalises over-forecasts and under-forecasts unequally
R²Comparing models on the same dataNot comparable across data sets with different spread

For decisions under uncertainty, a single number is often not enough. Prediction intervals, built with quantile regression or conformal prediction, give a range such as "between 52 and 61 lakh with 90 per cent confidence", which is more useful to a buyer than a point estimate. Evaluate them by coverage: about 90 per cent of true values should fall inside 90 per cent intervals.

How it works

MetricUnitsSensitive to outliersBest value
MAETarget unitsLess0
MSESquared unitsMore0
RMSETarget unitsMore0
R²NoneModerate1 (can be negative)

The maths

Mean absolute error and root mean squared error
Coefficient of determination

In code

All four metrics by hand and from scikit-learn, plus a simple residual check by quartile of the prediction.

regression_metrics.py
import numpy as np
from sklearn.datasets import load_diabetes
from sklearn.linear_model import LinearRegression
from sklearn.metrics import mean_absolute_error, mean_squared_error, r2_score
from sklearn.model_selection import train_test_split

X, y = load_diabetes(return_X_y=True)
X_tr, X_te, y_tr, y_te = train_test_split(X, y, test_size=0.3, random_state=0)
pred = LinearRegression().fit(X_tr, y_tr).predict(X_te)

mae = np.mean(np.abs(y_te - pred))
rmse = np.sqrt(np.mean((y_te - pred) ** 2))
r2 = 1 - np.sum((y_te - pred) ** 2) / np.sum((y_te - y_te.mean()) ** 2)
print(f"By hand:      MAE {mae:.1f}  RMSE {rmse:.1f}  R2 {r2:.3f}")
print(f"scikit-learn: MAE {mean_absolute_error(y_te, pred):.1f}  RMSE {np.sqrt(mean_squared_error(y_te, pred)):.1f}  R2 {r2_score(y_te, pred):.3f}")
print("Target range:", int(y.min()), "to", int(y.max()))

residuals = y_te - pred
for i, chunk in enumerate(np.array_split(np.argsort(pred), 4), start=1):
    print(f"Prediction quartile {i}: mean residual {residuals[chunk].mean():6.1f}")

Output

By hand:      MAE 44.6  RMSE 55.7  R2 0.393
scikit-learn: MAE 44.6  RMSE 55.7  R2 0.393
Target range: 25 to 346
Prediction quartile 1: mean residual    8.6
Prediction quartile 2: mean residual  -13.4
Prediction quartile 3: mean residual    8.5
Prediction quartile 4: mean residual  -25.2

Try it yourself

Choose the metric

Match each requirement to the metric that fits it best.

  • Report error in rupees, treating all errors equally
  • Big misses are much worse than small ones
  • Compare with always predicting the average
  • A smooth loss to optimise during training

Common mistakes

  • Reporting R² without RMSE or MAE, which hides the size of errors.
  • Comparing RMSE across datasets with different scales.
  • Ignoring residual patterns, which reveal a missing feature or curve.

Check yourself

Question 1

1. What does R² = 0 mean?

Question 2

2. Which is in the same units as the target?

Question 3

3. Can R² be negative on a test set?

Summary

  • MAE and RMSE are in target units; RMSE punishes large errors more.
  • R² compares the model with predicting the mean.
  • Always check residuals for patterns.

Finished reading?

Sources: scikit-learn: Regression metrics · James et al., An Introduction to Statistical Learning