Definition
Two loan models both reach 85 per cent accuracy. One approves 60 per cent of men and 58 per cent of women; the other approves 70 per cent of men and 35 per cent of women. Accuracy alone cannot tell them apart. Fairness metrics can.
Measuring bias in a model means comparing its behaviour across groups defined by sensitive attributes. Common fairness metrics include demographic parity (each group receives positive predictions at a similar rate), equal opportunity (the true positive rate, or recall, is similar across groups), equalised odds (both true positive and false positive rates are similar) and predictive parity (precision is similar). Results are reported as differences or ratios between groups. A key insight is that these metrics often cannot all be satisfied at once when base rates differ, so choosing which to prioritise is a value judgement.
In simple words
A teacher marks papers. Fair could mean the same pass rate for every section, or that every student who really knows the subject passes regardless of section, or that a pass means the same level of knowledge everywhere. These are different, and sometimes you cannot have all three.
Overview
- Model bias means a model's predictions or errors differ unfairly between groups.
- It is measured with fairness metrics that compare outcomes for groups defined by a sensitive attribute.
- Common metrics compare selection rates, true positive rates and false positive rates.
- The "four-fifths rule" flags concern if one group's selection rate is below 80% of another's.
- Different fairness metrics can conflict; it is mathematically impossible to satisfy all at once in most cases.
- Tools such as Fairlearn and AIF360 compute these metrics in Python.
Why it matters
Without group-level metrics, unfair models pass every test and reach users. Fairness metrics make disparities visible, allow comparison between models and support audits required by organisations and, increasingly, by law.
Key terms
- Demographic parity
- Similar rates of positive predictions across groups.
- Equal opportunity
- Similar true positive rates: qualified people in every group are equally likely to be approved.
- Equalised odds
- Both true positive and false positive rates are similar across groups.
- Disparate impact ratio
- The positive rate of one group divided by another's; below 0.8 is a common warning sign.
- Calibration by group
- A score of 0.7 means about 70 per cent chance of the outcome in every group.
- Impossibility result
- Proofs that some fairness criteria cannot all hold when base rates differ.
Types
There are several types of model fairness metric. Each defines "fair" differently.
1. Demographic parity (statistical parity)
Each group receives positive predictions at the same rate.
- How it works: Compute the selection rate per group and compare them as a difference or ratio.
- Example: Selection rates of 30% for men and 18% for women give a ratio of 0.6, below the 0.8 rule.
- When to use it: Use it when outcomes should be shared equally, such as showing job ads.
2. Equal opportunity
Qualified people in each group are correctly selected at the same rate.
- How it works: Compare true positive rates (recall) across groups using true labels.
- Example: A scholarship model catching 90% of deserving urban students but only 70% of deserving rural ones.
- When to use it: Use it when missing qualified people is the main harm, such as admissions or medical screening.
3. Equalised odds
Both true positive and false positive rates are equal across groups.
- How it works: Compare TPR and FPR per group; the largest gap shows how far the model is from equalised odds.
- Example: A fraud model flagging 5% of honest transactions in one region and 15% in another.
- When to use it: Use it when both wrongful acceptance and wrongful rejection matter, such as criminal-justice risk scores.
4. Predictive parity and calibration
A given score means the same chance of the outcome for every group.
- How it works: Compare precision per group and plot calibration curves by group.
- Example: COMPAS risk scores (ProPublica, 2016) were similarly calibrated across groups but had unequal false positive rates.
- When to use it: Use it when scores are read as probabilities, and remember it can conflict with equalised odds.
5. Individual and counterfactual fairness
Similar people get similar predictions, and changing only a sensitive attribute does not change the result.
- How it works: Create test pairs that differ only in gender, name or region and check whether predictions change.
- Example: Swapping the name "Priya" for "Rahul" on the same CV changes the hiring score.
- When to use it: Use it to audit models and LLMs for direct discrimination.
Explained in detail
Computing fairness metrics
Split the test set by group and build a confusion matrix for each. Suppose that among 100 male applicants the model approves 60, and among 100 female applicants it approves 40: the selection rates are 0.60 and 0.40, a demographic-parity difference of 0.20 and a disparate-impact ratio of 0.67, below the four-fifths (0.8) guideline used in US employment law as a warning sign. Now look at people who actually repaid: if the model approved 90 per cent of men who repaid but only 70 per cent of women who repaid, the equal-opportunity gap is 0.20, meaning qualified women are turned away more often. Also compare false positive rates and precision. Libraries such as Fairlearn and IBM's AI Fairness 360 compute these metrics and plot them for every group.
Which metric fits which situation
| Metric | Use when | Caution |
|---|---|---|
| Demographic parity | Opportunities should be shared, such as interview shortlists or scholarships | Ignores real differences in qualification in the data |
| Equal opportunity | Missing qualified people is the main harm, such as loans or medical screening | Relies on labels that may themselves be biased |
| Equalised odds | Both wrongful approval and wrongful denial matter, such as fraud flags | Harder to satisfy; may lower accuracy |
| Calibration | Scores are used directly by people, such as risk scores | Can coexist with large gaps in error rates |
The impossibility results
In 2016 ProPublica reported that the COMPAS risk tool used in some US courts wrongly labelled Black defendants as high risk at nearly twice the rate of white defendants. The tool's maker replied that its scores were equally calibrated for both groups. Both statements were true. Researchers including Jon Kleinberg and colleagues and Alexandra Chouldechova proved in 2016 and 2017 that when two groups have different base rates, a model cannot in general be both calibrated and have equal false positive and false negative rates for both. So fairness cannot be reduced to one number: teams must decide, with the people affected, which errors matter most in their context, measure several metrics and report the trade-offs openly.
Fairness audits should also be repeated over time, because a model that is fair at launch can drift as the population, economy or data collection changes.
How it works
- Choose the groups to compare, including intersections where data allows.
- Compute selection rate, true positive rate, false positive rate and precision per group.
- Report differences and ratios, with sample sizes and uncertainty.
- Decide with stakeholders which metrics matter most for this use.
- Repeat the audit after deployment, as data and populations change.
In code
import numpy as np
rng = np.random.default_rng(0)
n = 1000
group = rng.choice(["A", "B"], size=n, p=[0.6, 0.4])
repaid = rng.random(n) < np.where(group == "A", 0.7, 0.6) # true outcome (base rates differ)
score = np.clip(repaid * 0.3 + rng.normal(0.45, 0.15, n) - np.where(group == "B", 0.08, 0), 0, 1)
approved = score >= 0.55
def rates(mask):
y, p = repaid[mask], approved[mask]
return {"selection": p.mean(), "TPR": p[y].mean(), "FPR": p[~y].mean(), "precision": y[p].mean()}
r = {g: rates(group == g) for g in ["A", "B"]}
print(f"{'metric':<10}{'group A':>9}{'group B':>9}{'gap':>8}")
for m in ["selection", "TPR", "FPR", "precision"]:
print(f"{m:<10}{r['A'][m]:>9.2f}{r['B'][m]:>9.2f}{r['A'][m] - r['B'][m]:>8.2f}")
print(f"Accuracy overall: {(approved == repaid).mean():.2f}")
print(f"Disparate impact ratio (B/A selection): {r['B']['selection'] / r['A']['selection']:.2f} (warning below 0.80)")Output
metric group A group B gap selection 0.71 0.53 0.18 TPR 0.90 0.79 0.11 FPR 0.20 0.11 0.09 precision 0.92 0.92 0.00 Accuracy overall: 0.85 Disparate impact ratio (B/A selection): 0.74 (warning below 0.80)
Try it yourself
Two models, same accuracy
A microfinance company compares two loan models before choosing one.
Match the metric to what it compares
What does each metric compare across groups?
- Demographic parity
- Equal opportunity
- Equalised odds
- Predictive parity
Common mistakes
- Reporting only overall accuracy.
- Assuming one fairness metric captures everything.
- Ignoring small sample sizes when comparing groups.
Check yourself
Summary
- Compare model behaviour by group: selection rate, TPR, FPR, precision and calibration.
- Different metrics capture different ideas of fairness and can conflict.
- Choose priorities with those affected, report several metrics and re-audit after deployment.
Finished reading?
Sources: Barocas, Hardt and Narayanan, Fairness and Machine Learning · Fairlearn: fairness assessment and mitigation toolkit