Definition
Which of two models is better at every threshold, not just at 0.5?
A ROC curve plots the true positive rate (recall) against the false positive rate for every decision threshold; the area under it, ROC-AUC, is the probability that the model ranks a random positive above a random negative. A precision-recall (PR) curve plots precision against recall; its summary, average precision, is more informative than ROC-AUC when positives are rare.
In simple words
Instead of judging a model at one threshold, slide the threshold from strict to lenient and record what happens. A perfect model's ROC curve hugs the top-left corner (AUC 1); random guessing gives the diagonal (AUC 0.5). When only 1 in 1,000 cases is positive, the PR curve shows how painful the false alarms really are.
Why it matters
Curves separate how well a model ranks cases from where you set the threshold. They let you compare models fairly and then choose an operating point that meets a business constraint, such as 'at most 2 per cent false positives'.
Key terms
- ROC curve
- A plot of true positive rate against false positive rate as the threshold varies.
- AUC (area under the ROC curve)
- The probability that a random positive is scored higher than a random negative.
- Precision-recall curve
- A plot of precision against recall across thresholds.
- Average precision (PR-AUC)
- A summary of the precision-recall curve; its baseline equals the positive rate.
- Threshold-independent metric
- A metric that evaluates ranking quality across all thresholds rather than at one.
- Calibration curve
- Predicted probability versus observed frequency, showing whether scores mean what they say.
Explained in detail
Evaluating across every threshold
A classifier usually outputs a score, and the threshold turns it into a decision. Different users may need different thresholds, so it is useful to evaluate the model's ranking ability across all of them. The ROC curve plots the true positive rate (recall) on the y-axis against the false positive rate (1 − specificity) on the x-axis. A random model follows the diagonal, with AUC 0.5; a perfect model reaches the top-left corner, with AUC 1.0. AUC has a direct meaning: an AUC of 0.85 means that if you pick one positive and one negative case at random, the model gives the positive case the higher score 85 per cent of the time.
Why precision-recall curves matter for rare events
When positives are rare, ROC curves can look deceptively good. Imagine fraud in 1 transaction in 1,000. A model with a false positive rate of 1 per cent looks excellent on the ROC curve, but across 1,000,000 transactions it raises about 10,000 false alarms to catch at most 1,000 frauds, so precision is below 10 per cent. The precision-recall curve exposes this, because precision depends directly on how many negatives are wrongly flagged. The baseline of average precision equals the positive rate, here 0.001, so an average precision of 0.3 is a large improvement even though it sounds low. For imbalanced problems such as fraud, rare diseases and equipment failure, report PR curves and average precision alongside or instead of ROC-AUC.
| Metric | Use when | Caution |
|---|---|---|
| ROC-AUC | Classes are reasonably balanced; comparing ranking ability | Optimistic when positives are rare |
| PR-AUC / average precision | Positives are rare and false alarms are costly | Harder to compare across data sets with different positive rates |
| Calibration (Brier score) | Probabilities themselves are used, such as risk scores | Good ranking does not imply good calibration |
Curves also help choose a threshold: read off the point that gives the recall you need and check the precision there, or choose the point that maximises expected benefit given the costs of each error. Finally, remember that AUC summarises the whole curve, including regions you will never use; partial AUC or precision at a fixed recall can better reflect the operating range of a real system.
How it works
| ROC curve | Precision-recall curve | |
|---|---|---|
| Axes | TPR vs FPR | Precision vs recall |
| Random model | Diagonal, AUC 0.5 | Flat at the positive rate |
| Best when | Classes are roughly balanced | Positives are rare |
| Summary | ROC-AUC | Average precision |
The maths
- The model's score for an example
In code
ROC-AUC and average precision for two models on a rare-positive problem, a few points on each curve, and AUC computed as a ranking probability.
import numpy as np
from sklearn.datasets import make_classification
from sklearn.ensemble import RandomForestClassifier
from sklearn.linear_model import LogisticRegression
from sklearn.metrics import average_precision_score, precision_recall_curve, roc_auc_score, roc_curve
from sklearn.model_selection import train_test_split
X, y = make_classification(n_samples=5000, n_features=20, weights=[0.95], random_state=0)
X_tr, X_te, y_tr, y_te = train_test_split(X, y, test_size=0.4, stratify=y, random_state=0)
print("Positive rate in test:", round(y_te.mean(), 3))
for name, model in [("logistic", LogisticRegression(max_iter=1000)), ("forest", RandomForestClassifier(200, random_state=0))]:
s = model.fit(X_tr, y_tr).predict_proba(X_te)[:, 1]
print(f"{name:>8}: ROC-AUC {roc_auc_score(y_te, s):.3f} average precision {average_precision_score(y_te, s):.3f}")
fpr, tpr, _ = roc_curve(y_te, s)
for target in [0.01, 0.05, 0.2]:
print(f"At FPR {target:.2f}, the forest's TPR is {tpr[np.searchsorted(fpr, target)]:.2f}")
precision, recall, _ = precision_recall_curve(y_te, s)
print(f"At recall 0.8 or more, best precision is {precision[recall >= 0.8].max():.2f}")
pos, neg = s[y_te == 1], s[y_te == 0]
print("AUC as ranking probability:", round(float(np.mean(pos[:, None] > neg[None, :])), 3))Output
Positive rate in test: 0.055 logistic: ROC-AUC 0.952 average precision 0.897 forest: ROC-AUC 0.943 average precision 0.898 At FPR 0.01, the forest's TPR is 0.88 At FPR 0.05, the forest's TPR is 0.89 At FPR 0.20, the forest's TPR is 0.93 At recall 0.8 or more, best precision is 0.98 AUC as ranking probability: 0.932
Try it yourself
Read the curve
Match each curve description to what it tells you.
- ROC curve along the diagonal
- ROC curve hugging the top-left corner
- High ROC-AUC but low average precision
- One model's ROC curve is above another's everywhere
Common mistakes
- Relying on ROC-AUC when positives are very rare.
- Computing curves from hard 0/1 predictions instead of scores.
- Choosing the threshold on the test set.
Check yourself
Summary
- ROC: TPR against FPR; AUC is a ranking probability.
- PR curves and average precision suit rare positives.
- Compare models by curves; pick thresholds on validation data.
Finished reading?
Sources: scikit-learn: ROC · Saito and Rehmsmeier, PR vs ROC on imbalanced data (2015)