Definition
Is an error of 5 good or bad?
Regression metrics summarise how far predictions are from true values. MAE is the mean absolute error; MSE the mean squared error; RMSE its square root, in the target's units; and R² the share of the target's variance explained by the model compared with predicting the mean. Residual plots show whether errors have structure the model missed.
In simple words
An error of 5 means nothing until you know the scale: 5 rupees on a phone price is tiny, 5 degrees on body temperature is huge. RMSE keeps the target's units, punishing big misses more. MAE treats every rupee of error equally. R² says how much better you are than always guessing the average.
Why it matters
The metric decides which model wins. A delivery app that hates long delays should look at RMSE; one that cares about typical error, MAE. Reporting R² alone hides whether errors are acceptable in absolute terms.
Key terms
- Mean absolute error (MAE)
- The average absolute difference between predictions and actual values.
- Root mean squared error (RMSE)
- The square root of the mean squared error; penalises large errors more.
- R² (coefficient of determination)
- 1 − SS_res ÷ SS_tot: the share of variance explained compared with predicting the mean.
- MAPE
- Mean absolute percentage error; easy to read but unstable near zero.
- Quantile (pinball) loss
- A loss for predicting a percentile, used for prediction intervals.
- Residual plot
- A chart of residuals against predictions to reveal patterns the metric hides.
Explained in detail
Which error measure to use
Suppose a model predicts the prices of four flats as 40, 52, 61 and 75 lakh when the true prices are 42, 50, 60 and 90 lakh. The errors are −2, 2, 1 and −15. MAE is (2 + 2 + 1 + 15) ÷ 4 = 5 lakh: on average we are 5 lakh off. RMSE is √((4 + 4 + 1 + 225) ÷ 4) ≈ 7.6 lakh, higher than MAE because the single 15-lakh miss is squared. The gap between RMSE and MAE is itself informative: a large gap means a few big errors. MAE is easier to explain and less sensitive to outliers; RMSE is better when large errors are disproportionately costly, such as underestimating peak electricity demand. Always report errors in the target's units and alongside a baseline.
R² and its traps
R² compares the model's squared error with that of always predicting the mean. R² = 0.8 means the model removes 80 per cent of that baseline error. It can be negative on test data when the model is worse than the mean. But R² depends on how spread out the target is: the same model can show R² of 0.9 on flats from all over a city and 0.4 on flats from one neighbourhood, with identical rupee errors. Adding features never decreases training R², which is why adjusted R² or, better, validation error should guide feature choice. A high R² also does not prove the relationship is causal or that the model will hold next year.
| Metric | Best for | Caution |
|---|---|---|
| MAE | Typical size of error in plain units | Treats all errors linearly |
| RMSE | When large errors are especially harmful | Dominated by a few outliers |
| MAPE | Comparing series of different scales | Undefined at zero; penalises over-forecasts and under-forecasts unequally |
| R² | Comparing models on the same data | Not comparable across data sets with different spread |
For decisions under uncertainty, a single number is often not enough. Prediction intervals, built with quantile regression or conformal prediction, give a range such as "between 52 and 61 lakh with 90 per cent confidence", which is more useful to a buyer than a point estimate. Evaluate them by coverage: about 90 per cent of true values should fall inside 90 per cent intervals.
How it works
| Metric | Units | Sensitive to outliers | Best value |
|---|---|---|---|
| MAE | Target units | Less | 0 |
| MSE | Squared units | More | 0 |
| RMSE | Target units | More | 0 |
| R² | None | Moderate | 1 (can be negative) |
The maths
In code
All four metrics by hand and from scikit-learn, plus a simple residual check by quartile of the prediction.
import numpy as np
from sklearn.datasets import load_diabetes
from sklearn.linear_model import LinearRegression
from sklearn.metrics import mean_absolute_error, mean_squared_error, r2_score
from sklearn.model_selection import train_test_split
X, y = load_diabetes(return_X_y=True)
X_tr, X_te, y_tr, y_te = train_test_split(X, y, test_size=0.3, random_state=0)
pred = LinearRegression().fit(X_tr, y_tr).predict(X_te)
mae = np.mean(np.abs(y_te - pred))
rmse = np.sqrt(np.mean((y_te - pred) ** 2))
r2 = 1 - np.sum((y_te - pred) ** 2) / np.sum((y_te - y_te.mean()) ** 2)
print(f"By hand: MAE {mae:.1f} RMSE {rmse:.1f} R2 {r2:.3f}")
print(f"scikit-learn: MAE {mean_absolute_error(y_te, pred):.1f} RMSE {np.sqrt(mean_squared_error(y_te, pred)):.1f} R2 {r2_score(y_te, pred):.3f}")
print("Target range:", int(y.min()), "to", int(y.max()))
residuals = y_te - pred
for i, chunk in enumerate(np.array_split(np.argsort(pred), 4), start=1):
print(f"Prediction quartile {i}: mean residual {residuals[chunk].mean():6.1f}")Output
By hand: MAE 44.6 RMSE 55.7 R2 0.393 scikit-learn: MAE 44.6 RMSE 55.7 R2 0.393 Target range: 25 to 346 Prediction quartile 1: mean residual 8.6 Prediction quartile 2: mean residual -13.4 Prediction quartile 3: mean residual 8.5 Prediction quartile 4: mean residual -25.2
Try it yourself
Choose the metric
Match each requirement to the metric that fits it best.
- Report error in rupees, treating all errors equally
- Big misses are much worse than small ones
- Compare with always predicting the average
- A smooth loss to optimise during training
Common mistakes
- Reporting R² without RMSE or MAE, which hides the size of errors.
- Comparing RMSE across datasets with different scales.
- Ignoring residual patterns, which reveal a missing feature or curve.
Check yourself
Summary
- MAE and RMSE are in target units; RMSE punishes large errors more.
- R² compares the model with predicting the mean.
- Always check residuals for patterns.
Finished reading?
Sources: scikit-learn: Regression metrics · James et al., An Introduction to Statistical Learning