Definition
When is 99 per cent accuracy useless?
Classification metrics summarise the confusion matrix of true positives (TP), false positives (FP), true negatives (TN) and false negatives (FN). Precision is the share of predicted positives that are correct; recall (sensitivity) the share of actual positives found; specificity the share of actual negatives correctly rejected; and F1 the harmonic mean of precision and recall. The right metric depends on the relative cost of each error.
In simple words
If 1 in 100 patients has a disease, a model saying 'healthy' to everyone is 99 per cent accurate and misses every sick patient: recall 0. Precision asks 'when it raises an alarm, how often is it right?'; recall asks 'of all real cases, how many did it catch?'. Raising one usually lowers the other.
Why it matters
Choosing the metric is a product decision. Cancer screening prioritises recall; a spam filter that hides important email needs high precision; credit decisions weigh both with money. Reporting accuracy alone on imbalanced data is a classic error.
Key terms
- Confusion matrix
- A table of true positives (TP), false positives (FP), true negatives (TN) and false negatives (FN).
- Precision
- TP ÷ (TP + FP): of the cases flagged positive, how many really are.
- Recall (sensitivity)
- TP ÷ (TP + FN): of the real positives, how many were found.
- Specificity
- TN ÷ (TN + FP): of the real negatives, how many were correctly cleared.
- F1 and Fβ score
- The harmonic mean of precision and recall; Fβ weights recall β times as much.
- Macro and weighted averages
- Averaging a metric across classes equally, or in proportion to class size.
Explained in detail
Reading a confusion matrix
Suppose a tuberculosis screening model is tested on 1,000 people, 50 of whom have the disease. It flags 60 people: 40 truly have TB (TP) and 20 do not (FP). It misses 10 cases (FN) and correctly clears 930 (TN). Accuracy is (40 + 930) ÷ 1,000 = 97 per cent, which sounds excellent, yet a model that flagged nobody would score 95 per cent. Precision is 40 ÷ 60 = 0.67: two-thirds of flagged people actually have TB. Recall is 40 ÷ 50 = 0.80: the model finds 80 per cent of cases. Specificity is 930 ÷ 950 = 0.98. F1 is 2 × 0.67 × 0.80 ÷ (0.67 + 0.80) ≈ 0.73. These numbers tell a far richer story than accuracy alone.
Choosing the metric for the cost of errors
| Application | Costlier error | Focus on |
|---|---|---|
| Cancer or TB screening | Missing a sick person (FN) | Recall, or F2 |
| Spam filter | Hiding a real email (FP) | Precision |
| Fraud alerts reviewed by staff | Both, limited reviewer time | Precision at the number of alerts staff can handle |
| Search results | Irrelevant top results | Precision at k |
Precision and recall trade off: lowering the threshold catches more positives (higher recall) but also more false alarms (lower precision). F1 balances the two, but it treats them as equally important, which is rarely true; F2 weights recall more and F0.5 weights precision more. For multiple classes, compute precision and recall per class and average them. A macro average treats every class equally, exposing poor performance on rare classes; a weighted average reflects the class mix; a micro average pools all decisions. Cohen's kappa and the Matthews correlation coefficient (MCC) summarise performance on imbalanced data more fairly than accuracy, and MCC is high only when all four cells of the confusion matrix are good.
Always report the confusion matrix alongside any summary number, state the threshold used, and give results for important subgroups. A model with 0.80 recall overall may have 0.90 for urban patients and 0.60 for rural ones, a difference that the overall figure hides and that matters greatly for fairness and safety.
How it works
| Predicted positive | Predicted negative | |
|---|---|---|
| Actually positive | TP: caught | FN: missed |
| Actually negative | FP: false alarm | TN: correctly ignored |
The maths
A fraud model flags 50 transactions. 40 really are fraud. There were 60 frauds in total among 1,000 transactions.
TP = 40, FP = 10, FN = 20, TN = 930 Precision = 40 / 50 = 0.80 Recall = 40 / 60 = 0.67 F1 = 2 × 0.80 × 0.67 / (0.80 + 0.67) = 0.73 Accuracy = (40 + 930) / 1000 = 0.97
In code
The worked example computed by hand and checked with scikit-learn, then how precision and recall move as the threshold changes.
import numpy as np
from sklearn.datasets import load_breast_cancer
from sklearn.linear_model import LogisticRegression
from sklearn.metrics import confusion_matrix, f1_score, precision_score, recall_score
from sklearn.model_selection import train_test_split
from sklearn.pipeline import make_pipeline
from sklearn.preprocessing import StandardScaler
y_true = np.array([1] * 40 + [0] * 10 + [1] * 20 + [0] * 930)
y_pred = np.array([1] * 50 + [0] * 950)
tn, fp, fn, tp = confusion_matrix(y_true, y_pred).ravel()
print("TP FP FN TN:", tp, fp, fn, tn)
print(f"precision {precision_score(y_true, y_pred):.2f} recall {recall_score(y_true, y_pred):.2f} F1 {f1_score(y_true, y_pred):.2f} specificity {tn / (tn + fp):.3f}")
X, y = load_breast_cancer(return_X_y=True)
y = 1 - y # make 'malignant' the positive class
X_tr, X_te, y_tr, y_te = train_test_split(X, y, test_size=0.3, stratify=y, random_state=0)
proba = make_pipeline(StandardScaler(), LogisticRegression(max_iter=1000)).fit(X_tr, y_tr).predict_proba(X_te)[:, 1]
print("threshold precision recall")
for t in [0.2, 0.5, 0.8]:
pred = (proba >= t).astype(int)
print(f"{t:>9} {precision_score(y_te, pred):9.3f} {recall_score(y_te, pred):6.3f}")Output
TP FP FN TN: 40 10 20 930
precision 0.80 recall 0.67 F1 0.73 specificity 0.989
threshold precision recall
0.2 0.824 0.953
0.5 0.938 0.938
0.8 1.000 0.891Try it yourself
Compute precision and recall
TP = 30, FP = 10, FN = 20, TN = 140. Match each metric to its value.
- Precision
- Recall
- Specificity
- Accuracy
Which metric matters most?
For each application, should you prioritise precision or recall?
Flagging possible fraud for a human to review
Screening for a serious, treatable disease
A spam filter that might hide a job offer
Automatically blocking user accounts
Common mistakes
- Reporting accuracy alone on imbalanced data.
- Mixing up which class is 'positive' when computing precision and recall.
- Optimising F1 when the costs of FP and FN are very different. Use F-beta or a cost.
Check yourself
Summary
- Precision: how often alarms are right. Recall: how many positives are caught.
- Specificity, F1 and F-beta complete the picture.
- Choose the metric from the real cost of each error.
Read it. Now try it yourself.
Open the interactive activityFinished reading?
Sources: scikit-learn: Classification metrics · Google ML Crash Course: Accuracy, precision, recall