A1 · Supervised Machine Learning in Depth

ROC-AUC and Precision-Recall Curves

A ROC curve plots the true positive rate (recall) against the false positive rate for every decision threshold; the area under it, ROC-AUC, is the probability that the model ranks a random positive above a random negative. A precision-recall (PR) curve plots precision against recall; its summary, average precision, is more informative than ROC-AUC when positives are rare.

12 min Advanced Jump to the activity

What you need: The earlier topics of this module, and its prerequisites.

Definition

Which of two models is better at every threshold, not just at 0.5?

A ROC curve plots the true positive rate (recall) against the false positive rate for every decision threshold; the area under it, ROC-AUC, is the probability that the model ranks a random positive above a random negative. A precision-recall (PR) curve plots precision against recall; its summary, average precision, is more informative than ROC-AUC when positives are rare.

In simple words

Instead of judging a model at one threshold, slide the threshold from strict to lenient and record what happens. A perfect model's ROC curve hugs the top-left corner (AUC 1); random guessing gives the diagonal (AUC 0.5). When only 1 in 1,000 cases is positive, the PR curve shows how painful the false alarms really are.

Why it matters

Curves separate how well a model ranks cases from where you set the threshold. They let you compare models fairly and then choose an operating point that meets a business constraint, such as 'at most 2 per cent false positives'.

Key terms

ROC curve
A plot of true positive rate against false positive rate as the threshold varies.
AUC (area under the ROC curve)
The probability that a random positive is scored higher than a random negative.
Precision-recall curve
A plot of precision against recall across thresholds.
Average precision (PR-AUC)
A summary of the precision-recall curve; its baseline equals the positive rate.
Threshold-independent metric
A metric that evaluates ranking quality across all thresholds rather than at one.
Calibration curve
Predicted probability versus observed frequency, showing whether scores mean what they say.

Explained in detail

Evaluating across every threshold

A classifier usually outputs a score, and the threshold turns it into a decision. Different users may need different thresholds, so it is useful to evaluate the model's ranking ability across all of them. The ROC curve plots the true positive rate (recall) on the y-axis against the false positive rate (1 − specificity) on the x-axis. A random model follows the diagonal, with AUC 0.5; a perfect model reaches the top-left corner, with AUC 1.0. AUC has a direct meaning: an AUC of 0.85 means that if you pick one positive and one negative case at random, the model gives the positive case the higher score 85 per cent of the time.

Why precision-recall curves matter for rare events

When positives are rare, ROC curves can look deceptively good. Imagine fraud in 1 transaction in 1,000. A model with a false positive rate of 1 per cent looks excellent on the ROC curve, but across 1,000,000 transactions it raises about 10,000 false alarms to catch at most 1,000 frauds, so precision is below 10 per cent. The precision-recall curve exposes this, because precision depends directly on how many negatives are wrongly flagged. The baseline of average precision equals the positive rate, here 0.001, so an average precision of 0.3 is a large improvement even though it sounds low. For imbalanced problems such as fraud, rare diseases and equipment failure, report PR curves and average precision alongside or instead of ROC-AUC.

MetricUse whenCaution
ROC-AUCClasses are reasonably balanced; comparing ranking abilityOptimistic when positives are rare
PR-AUC / average precisionPositives are rare and false alarms are costlyHarder to compare across data sets with different positive rates
Calibration (Brier score)Probabilities themselves are used, such as risk scoresGood ranking does not imply good calibration

Curves also help choose a threshold: read off the point that gives the recall you need and check the precision there, or choose the point that maximises expected benefit given the costs of each error. Finally, remember that AUC summarises the whole curve, including regions you will never use; partial AUC or precision at a fixed recall can better reflect the operating range of a real system.

How it works

ROC curvePrecision-recall curve
AxesTPR vs FPRPrecision vs recall
Random modelDiagonal, AUC 0.5Flat at the positive rate
Best whenClasses are roughly balancedPositives are rare
SummaryROC-AUCAverage precision

The maths

Rates on the ROC axes
AUC as a ranking probability
The model's score for an example

In code

ROC-AUC and average precision for two models on a rare-positive problem, a few points on each curve, and AUC computed as a ranking probability.

roc_pr.py
import numpy as np
from sklearn.datasets import make_classification
from sklearn.ensemble import RandomForestClassifier
from sklearn.linear_model import LogisticRegression
from sklearn.metrics import average_precision_score, precision_recall_curve, roc_auc_score, roc_curve
from sklearn.model_selection import train_test_split

X, y = make_classification(n_samples=5000, n_features=20, weights=[0.95], random_state=0)
X_tr, X_te, y_tr, y_te = train_test_split(X, y, test_size=0.4, stratify=y, random_state=0)
print("Positive rate in test:", round(y_te.mean(), 3))

for name, model in [("logistic", LogisticRegression(max_iter=1000)), ("forest", RandomForestClassifier(200, random_state=0))]:
    s = model.fit(X_tr, y_tr).predict_proba(X_te)[:, 1]
    print(f"{name:>8}: ROC-AUC {roc_auc_score(y_te, s):.3f}  average precision {average_precision_score(y_te, s):.3f}")

fpr, tpr, _ = roc_curve(y_te, s)
for target in [0.01, 0.05, 0.2]:
    print(f"At FPR {target:.2f}, the forest's TPR is {tpr[np.searchsorted(fpr, target)]:.2f}")
precision, recall, _ = precision_recall_curve(y_te, s)
print(f"At recall 0.8 or more, best precision is {precision[recall >= 0.8].max():.2f}")

pos, neg = s[y_te == 1], s[y_te == 0]
print("AUC as ranking probability:", round(float(np.mean(pos[:, None] > neg[None, :])), 3))

Output

Positive rate in test: 0.055
logistic: ROC-AUC 0.952  average precision 0.897
  forest: ROC-AUC 0.943  average precision 0.898
At FPR 0.01, the forest's TPR is 0.88
At FPR 0.05, the forest's TPR is 0.89
At FPR 0.20, the forest's TPR is 0.93
At recall 0.8 or more, best precision is 0.98
AUC as ranking probability: 0.932

Try it yourself

Read the curve

Match each curve description to what it tells you.

  • ROC curve along the diagonal
  • ROC curve hugging the top-left corner
  • High ROC-AUC but low average precision
  • One model's ROC curve is above another's everywhere

Common mistakes

  • Relying on ROC-AUC when positives are very rare.
  • Computing curves from hard 0/1 predictions instead of scores.
  • Choosing the threshold on the test set.

Check yourself

Question 1

1. A ROC-AUC of 0.5 means…

Question 2

2. Which curve is better for rare positives?

Question 3

3. What do the curves need from the model?

Summary

  • ROC: TPR against FPR; AUC is a ranking probability.
  • PR curves and average precision suit rare positives.
  • Compare models by curves; pick thresholds on validation data.

Finished reading?

Sources: scikit-learn: ROC · Saito and Rehmsmeier, PR vs ROC on imbalanced data (2015)