A1 · Supervised Machine Learning in Depth

Classification Metrics in Depth

Classification metrics summarise the confusion matrix of true positives (TP), false positives (FP), true negatives (TN) and false negatives (FN). Precision is the share of predicted positives that are correct; recall (sensitivity) the share of actual positives found; specificity the share of actual negatives correctly rejected; and F1 the harmonic mean of precision and recall. The right metric depends on the relative cost of each error.

12 min Advanced Jump to the activity

What you need: The earlier topics of this module, and its prerequisites.

Definition

When is 99 per cent accuracy useless?

Classification metrics summarise the confusion matrix of true positives (TP), false positives (FP), true negatives (TN) and false negatives (FN). Precision is the share of predicted positives that are correct; recall (sensitivity) the share of actual positives found; specificity the share of actual negatives correctly rejected; and F1 the harmonic mean of precision and recall. The right metric depends on the relative cost of each error.

In simple words

If 1 in 100 patients has a disease, a model saying 'healthy' to everyone is 99 per cent accurate and misses every sick patient: recall 0. Precision asks 'when it raises an alarm, how often is it right?'; recall asks 'of all real cases, how many did it catch?'. Raising one usually lowers the other.

Why it matters

Choosing the metric is a product decision. Cancer screening prioritises recall; a spam filter that hides important email needs high precision; credit decisions weigh both with money. Reporting accuracy alone on imbalanced data is a classic error.

Key terms

Confusion matrix
A table of true positives (TP), false positives (FP), true negatives (TN) and false negatives (FN).
Precision
TP ÷ (TP + FP): of the cases flagged positive, how many really are.
Recall (sensitivity)
TP ÷ (TP + FN): of the real positives, how many were found.
Specificity
TN ÷ (TN + FP): of the real negatives, how many were correctly cleared.
F1 and Fβ score
The harmonic mean of precision and recall; Fβ weights recall β times as much.
Macro and weighted averages
Averaging a metric across classes equally, or in proportion to class size.

Explained in detail

Reading a confusion matrix

Suppose a tuberculosis screening model is tested on 1,000 people, 50 of whom have the disease. It flags 60 people: 40 truly have TB (TP) and 20 do not (FP). It misses 10 cases (FN) and correctly clears 930 (TN). Accuracy is (40 + 930) ÷ 1,000 = 97 per cent, which sounds excellent, yet a model that flagged nobody would score 95 per cent. Precision is 40 ÷ 60 = 0.67: two-thirds of flagged people actually have TB. Recall is 40 ÷ 50 = 0.80: the model finds 80 per cent of cases. Specificity is 930 ÷ 950 = 0.98. F1 is 2 × 0.67 × 0.80 ÷ (0.67 + 0.80) ≈ 0.73. These numbers tell a far richer story than accuracy alone.

Choosing the metric for the cost of errors

ApplicationCostlier errorFocus on
Cancer or TB screeningMissing a sick person (FN)Recall, or F2
Spam filterHiding a real email (FP)Precision
Fraud alerts reviewed by staffBoth, limited reviewer timePrecision at the number of alerts staff can handle
Search resultsIrrelevant top resultsPrecision at k

Precision and recall trade off: lowering the threshold catches more positives (higher recall) but also more false alarms (lower precision). F1 balances the two, but it treats them as equally important, which is rarely true; F2 weights recall more and F0.5 weights precision more. For multiple classes, compute precision and recall per class and average them. A macro average treats every class equally, exposing poor performance on rare classes; a weighted average reflects the class mix; a micro average pools all decisions. Cohen's kappa and the Matthews correlation coefficient (MCC) summarise performance on imbalanced data more fairly than accuracy, and MCC is high only when all four cells of the confusion matrix are good.

Always report the confusion matrix alongside any summary number, state the threshold used, and give results for important subgroups. A model with 0.80 recall overall may have 0.90 for urban patients and 0.60 for rural ones, a difference that the overall figure hides and that matters greatly for fairness and safety.

How it works

Predicted positivePredicted negative
Actually positiveTP: caughtFN: missed
Actually negativeFP: false alarmTN: correctly ignored

The maths

Precision and recall
Specificity and F1
F-beta: weight recall β times as much as precision

A fraud model flags 50 transactions. 40 really are fraud. There were 60 frauds in total among 1,000 transactions.

TP = 40, FP = 10, FN = 20, TN = 930
Precision = 40 / 50 = 0.80
Recall    = 40 / 60 = 0.67
F1        = 2 × 0.80 × 0.67 / (0.80 + 0.67) = 0.73
Accuracy  = (40 + 930) / 1000 = 0.97

In code

The worked example computed by hand and checked with scikit-learn, then how precision and recall move as the threshold changes.

classification_metrics.py
import numpy as np
from sklearn.datasets import load_breast_cancer
from sklearn.linear_model import LogisticRegression
from sklearn.metrics import confusion_matrix, f1_score, precision_score, recall_score
from sklearn.model_selection import train_test_split
from sklearn.pipeline import make_pipeline
from sklearn.preprocessing import StandardScaler

y_true = np.array([1] * 40 + [0] * 10 + [1] * 20 + [0] * 930)
y_pred = np.array([1] * 50 + [0] * 950)
tn, fp, fn, tp = confusion_matrix(y_true, y_pred).ravel()
print("TP FP FN TN:", tp, fp, fn, tn)
print(f"precision {precision_score(y_true, y_pred):.2f}  recall {recall_score(y_true, y_pred):.2f}  F1 {f1_score(y_true, y_pred):.2f}  specificity {tn / (tn + fp):.3f}")

X, y = load_breast_cancer(return_X_y=True)
y = 1 - y                                         # make 'malignant' the positive class
X_tr, X_te, y_tr, y_te = train_test_split(X, y, test_size=0.3, stratify=y, random_state=0)
proba = make_pipeline(StandardScaler(), LogisticRegression(max_iter=1000)).fit(X_tr, y_tr).predict_proba(X_te)[:, 1]
print("threshold  precision  recall")
for t in [0.2, 0.5, 0.8]:
    pred = (proba >= t).astype(int)
    print(f"{t:>9}  {precision_score(y_te, pred):9.3f}  {recall_score(y_te, pred):6.3f}")

Output

TP FP FN TN: 40 10 20 930
precision 0.80  recall 0.67  F1 0.73  specificity 0.989
threshold  precision  recall
      0.2      0.824   0.953
      0.5      0.938   0.938
      0.8      1.000   0.891

Try it yourself

Compute precision and recall

TP = 30, FP = 10, FN = 20, TN = 140. Match each metric to its value.

  • Precision
  • Recall
  • Specificity
  • Accuracy

Which metric matters most?

For each application, should you prioritise precision or recall?

  • Flagging possible fraud for a human to review

  • Screening for a serious, treatable disease

  • A spam filter that might hide a job offer

  • Automatically blocking user accounts

Common mistakes

  • Reporting accuracy alone on imbalanced data.
  • Mixing up which class is 'positive' when computing precision and recall.
  • Optimising F1 when the costs of FP and FN are very different. Use F-beta or a cost.

Check yourself

Question 1

1. Recall answers which question?

Question 2

2. Lowering the decision threshold usually…

Question 3

3. F1 is the … of precision and recall.

Summary

  • Precision: how often alarms are right. Recall: how many positives are caught.
  • Specificity, F1 and F-beta complete the picture.
  • Choose the metric from the real cost of each error.

Read it. Now try it yourself.

Open the interactive activity

Finished reading?

Sources: scikit-learn: Classification metrics · Google ML Crash Course: Accuracy, precision, recall