Definition
A diabetes screening model gives each person a risk score. Set the cut-off low and you catch almost everyone with diabetes but alarm many healthy people; set it high and you miss cases. A ROC curve shows every possible cut-off at once.
A receiver operating characteristic (ROC) curve plots a classifier's true positive rate (recall) against its false positive rate as the decision threshold moves from high to low. The area under the curve (AUC) summarises the curve in one number between 0 and 1: 0.5 means no better than random guessing, 1.0 means perfect separation. AUC equals the probability that the model scores a randomly chosen positive case higher than a randomly chosen negative case. For rare positives, the precision-recall curve is often more informative.
In simple words
Imagine lining up everyone by their risk score, highest first. A good model puts most sick people near the front. The ROC curve shows how many sick and how many healthy people you pass as you walk down the line; AUC measures how well the line is sorted.
Overview
- A ROC curve plots the true positive rate against the false positive rate across all thresholds.
- AUC, the area under the curve, ranges from 0.5 (random) to 1.0 (perfect).
- AUC equals the probability that a random positive is scored above a random negative.
- ROC curves compare ranking ability independently of any single threshold.
- For rare positives, precision-recall curves are more informative.
- The operating threshold is chosen afterwards from real costs or capacity.
Why it matters
Most classifiers output scores, and the right threshold depends on costs that change from one user to another. ROC and precision-recall curves let you judge a model's ranking ability independently of any one threshold and then choose a threshold that fits the situation.
Key terms
- True positive rate (TPR)
- TP ÷ (TP + FN), the same as recall or sensitivity.
- False positive rate (FPR)
- FP ÷ (FP + TN), the share of negatives wrongly flagged; equals 1 − specificity.
- Threshold
- The score above which a case is labelled positive.
- AUC
- Area under the ROC curve; the chance a random positive outranks a random negative.
- Precision-recall curve
- Precision plotted against recall across thresholds, better for rare positives.
- Operating point
- The threshold actually chosen for use, based on costs and capacity.
Types
Several types of curve and summary are used to evaluate classifiers across thresholds. These are the main ones.
1. ROC curve
The plot of TPR (recall) against FPR as the threshold moves.
- How it works: Sort cases by score; each positive passed moves the curve up and each negative moves it right, from (0, 0) to (1, 1).
- Example: A curve bowing towards the top-left corner shows a strong model.
- When to use it: Use it to compare models' ranking ability on reasonably balanced data.
2. ROC-AUC
The area under the ROC curve as one number.
- How it works: roc_auc_score computes it from true labels and scores; it can be read as the chance that a positive outranks a negative.
- Example: Six patients with 7 of 9 sick-healthy pairs correctly ordered give AUC = 0.78.
- When to use it: Use it to compare classifiers without choosing a threshold; very high values on hard problems may signal leakage.
3. Precision-recall curve and average precision
Precision plotted against recall across thresholds, summarised by average precision (AP).
- How it works: Precision falls as false alarms grow, so the curve reveals problems that ROC hides when negatives vastly outnumber positives; the baseline AP equals the positive rate.
- Example: A fraud model with ROC-AUC 0.73 but AP only 0.36 at a 2.6% positive rate.
- When to use it: Use it for rare-event problems such as fraud, disease and defects.
4. Choosing an operating point
Selecting the threshold actually used in practice.
- How it works: Pick the point on the curve that minimises expected cost (for example 10 × FN + FP), meets a recall target, or flags as many cases as reviewers can handle.
- Example: A hospital that can follow up 50 patients a week sets the threshold that flags about 50.
- When to use it: Use validation data for this choice, never the test set.
5. Calibration curves
Plots comparing predicted probabilities with observed frequencies.
- How it works: Group predictions into bins (0.1, 0.2, …) and plot the fraction of actual positives in each; a well-calibrated model lies on the diagonal.
- Example: Cases given 70% risk should turn out positive about 70% of the time.
- When to use it: Use them when probabilities themselves are used for decisions; fix poor calibration with CalibratedClassifierCV.
Explained in detail
Building a ROC curve
Sort the test cases by the model's score. Start with the threshold above the highest score: nothing is flagged, so TPR = 0 and FPR = 0, the bottom-left corner. Lower the threshold past each case in turn. Each positive case passed moves the curve up by 1 ÷ (number of positives); each negative moves it right by 1 ÷ (number of negatives). When the threshold is below every score, everything is flagged and the curve reaches the top-right corner (1, 1). A perfect model climbs straight up to (0, 1) before moving right; a random model wanders along the diagonal. The area under the curve can be computed from these steps.
Interpreting AUC
| AUC | Rough interpretation |
|---|---|
| 0.5 | No skill: as good as a coin toss |
| 0.7 to 0.8 | Useful ranking for many tasks |
| 0.8 to 0.9 | Strong |
| Above 0.95 | Excellent, or a sign of leakage worth checking |
| Below 0.5 | Worse than random: labels or scores may be reversed |
When to prefer precision-recall curves
ROC curves ignore how rare the positive class is, because FPR divides by the large number of negatives. Suppose 1 in 1,000 transactions is fraud. A model with FPR of 2 per cent looks excellent on a ROC plot, but among 100,000 transactions it flags about 2,000 innocent ones to find at most 100 frauds, so most alerts are false. The precision-recall curve shows this directly, because precision falls as false alarms grow. For rare events such as fraud, disease or equipment failure, report average precision (precision averaged over recall increases) alongside ROC-AUC, and remember that its baseline is the positive rate, here 0.001, not 0.5. The code below shows the same effect: with 2.6 per cent positives, a model with ROC-AUC 0.73 has average precision of only 0.36, and meeting a validation recall target of at least 80 per cent means flagging many more loans or transactions than there are real positives. The displayed threshold belongs to the validation exercise. Evaluate a fixed threshold on fresh data before using it.
Curves help choose the operating point. If a hospital can follow up 50 patients a week, choose the threshold that flags about 50 and read the recall there. If missing a case is ten times worse than a false alarm, choose the point that minimises 10 × FN + FP on validation data.
How it works
Can you trust a score of 0.8?
A return predictor assigns scores near 0.8 to 50 validation loans. Only 25 actually arrive late. The observed late fraction in this bin is 25/50 = 0.5. A reliability diagram places the mean predicted probability on its horizontal axis and the observed fraction on its vertical axis. This bin suggests overconfidence. Use more examples and examine several bins before drawing a conclusion about the whole model.
A strictly increasing transformation of every score preserves ranking and ROC AUC but can change calibration. If probabilities matter, fit a calibrator on predictions from data held out from the original model fit. If an action matters, choose its threshold using the cost of a missed case and a false alarm on validation data. Freeze the choices before evaluating once on the final test set.
- Get predicted probabilities or scores on validation or test data.
- Compute the ROC curve and AUC with roc_curve and roc_auc_score.
- For rare positives, also compute the precision-recall curve and average precision.
- Compare models by AUC, then choose an operating threshold from the costs.
- Report the confusion matrix at that threshold.
The maths
- The model's score for a case
Six patients get scores 0.9 (sick), 0.8 (healthy), 0.7 (sick), 0.4 (sick), 0.3 (healthy) and 0.1 (healthy). There are 3 sick and 3 healthy, so 9 sick-healthy pairs. Count pairs where the sick person scores higher: 0.9 beats all three healthy (3), 0.7 beats 0.3 and 0.1 (2), 0.4 beats 0.3 and 0.1 (2). That is 7 of 9 pairs, so AUC = 7 ÷ 9 ≈ 0.78. At threshold 0.5, the flagged patients are 0.9, 0.8 and 0.7: TPR = 2 ÷ 3 ≈ 0.67 and FPR = 1 ÷ 3 ≈ 0.33.
In code
import matplotlib.pyplot as plt
import numpy as np
from sklearn.datasets import make_classification
from sklearn.linear_model import LogisticRegression
from sklearn.metrics import average_precision_score, roc_auc_score, roc_curve
from sklearn.model_selection import train_test_split
# The worked example
y_small = np.array([1, 0, 1, 1, 0, 0]); s_small = np.array([0.9, 0.8, 0.7, 0.4, 0.3, 0.1])
print("Worked example AUC:", round(roc_auc_score(y_small, s_small), 3))
# A rare-event problem: about 2-3% positives
X, y = make_classification(n_samples=5000, weights=[0.98], n_informative=4, class_sep=0.8, random_state=3)
X_tr, X_valid, y_tr, y_valid = train_test_split(X, y, stratify=y, random_state=0)
scores = LogisticRegression(max_iter=1000).fit(X_tr, y_tr).predict_proba(X_valid)[:, 1]
fpr, tpr, thr = roc_curve(y_valid, scores)
print(f"ROC-AUC = {roc_auc_score(y_valid, scores):.3f} average precision = {average_precision_score(y_valid, scores):.3f} positive rate = {y_valid.mean():.3f}")
i = np.flatnonzero(tpr >= 0.8)[0] # first threshold reaching the recall target
flagged = (scores >= thr[i]).sum()
print(f"Validation recall {tpr[i]:.1%}: threshold {thr[i]:.3f}, FPR {fpr[i]:.3f}, flags {flagged} of {len(y_valid)} cases for {y_valid.sum()} real positives")
fig, ax = plt.subplots(figsize=(4, 4))
ax.plot(fpr, tpr, label="model"); ax.plot([0, 1], [0, 1], "--", color="grey", label="random")
ax.set_xlabel("False positive rate"); ax.set_ylabel("True positive rate"); ax.set_title("ROC curve"); ax.legend()Output
Worked example AUC: 0.778 ROC-AUC = 0.727 average precision = 0.356 positive rate = 0.026 Validation recall 81.8%: threshold 0.011, FPR 0.606, flags 765 of 1250 cases for 33 real positives
Try it yourself
ROC-AUC or precision-recall?
Which summary suits each problem better?
Predicting whether a customer buys, 40% do
Classifying cats versus dogs, half each
Detecting fraud in 1 of 2,000 transactions
Screening for a rare genetic disorder
Match the AUC to its meaning
What does each value suggest?
- 0.5
- 0.85
- 0.30
- 0.999 on a hard task
Common mistakes
- Reporting ROC-AUC alone for very rare events.
- Treating AUC as accuracy at a particular threshold.
- Choosing the threshold on the test set.
Check yourself
Summary
- A ROC curve plots TPR against FPR across all thresholds; AUC summarises ranking quality.
- AUC 0.5 is random and 1.0 is perfect; it equals the chance a positive outranks a negative.
- For rare positives use precision-recall curves, and choose the operating threshold from real costs.
Finished reading?
Sources: scikit-learn, choosing a decision threshold · scikit-learn, probability calibration · Tom Fawcett (2006), An introduction to ROC analysis · scikit-learn user guide