I10 · Responsible AI in Practice

Reducing Bias

Bias mitigation methods fall into three stages. Pre-processing changes the training data: collecting more data from under-represented groups, re-weighting or re-sampling examples, fixing biased labels or removing proxy features. In-processing changes how the model is trained, for example by adding a fairness constraint or penalty to the objective. Post-processing changes decisions after the model scores, for example by choosing group-specific thresholds to equalise error rates. Beyond technical fixes, reducing bias needs diverse teams, involvement of affected communities, human review and the option not to deploy.

12 min Intermediate · Sublevel Advanced Jump to the activity

What you need: The earlier topics of this module.

Definition

You have measured a 20-point gap in how often qualified women are approved. Now what? You can fix the data, change how the model learns, or adjust its decisions afterwards, and each choice has costs. Reducing bias is engineering plus judgement.

Bias mitigation methods fall into three stages. Pre-processing changes the training data: collecting more data from under-represented groups, re-weighting or re-sampling examples, fixing biased labels or removing proxy features. In-processing changes how the model is trained, for example by adding a fairness constraint or penalty to the objective. Post-processing changes decisions after the model scores, for example by choosing group-specific thresholds to equalise error rates. Beyond technical fixes, reducing bias needs diverse teams, involvement of affected communities, human review and the option not to deploy.

In simple words

If a recipe keeps coming out too salty, you can buy different ingredients (fix the data), change how you cook (fix the training) or add water at the end (fix the output). Sometimes you need all three, and sometimes you should not serve the dish at all.

Overview

  • Bias can be reduced at three stages: before training (data), during training (model) and after training (outputs).
  • No single method removes all bias; teams combine methods and measure the effect.
  • Removing the sensitive attribute alone rarely works because other features act as proxies.
  • Fixes often trade a little overall accuracy for fairer results.
  • Process changes, diverse teams, community input and audits, matter as much as algorithms.
  • Bias must be re-measured after deployment because data and populations change.

Why it matters

Measuring bias without acting on it changes nothing for the people harmed. Knowing the options and their trade-offs lets teams choose fixes that work, explain them and check that they did not create new problems.

Key terms

Pre-processing
Mitigating bias by changing the training data.
Re-weighting
Giving examples from under-represented groups more weight during training.
In-processing
Adding fairness constraints or penalties to the training objective.
Post-processing
Adjusting decisions after scoring, such as group-specific thresholds.
Fairness-accuracy trade-off
Improving a fairness metric may reduce overall accuracy slightly, or sometimes not at all.
Human-in-the-loop
People review uncertain or high-stakes decisions before they take effect.

Types

Bias reduction methods come in several types, grouped by the stage at which they act.

1. Better data collection

Gathering more representative and accurately labelled data.

  • How it works: Collect from under-represented groups, review labelling guidelines and use diverse annotators.
  • Example: Recording speech samples in 22 scheduled Indian languages for a voice assistant.
  • When to use it: Use it first whenever possible; better data fixes problems at the root. Plan it with the communities affected, pay contributors fairly and record who is still missing.

2. Pre-processing: reweighting and resampling

Adjusting the training data so groups and labels are balanced.

  • How it works: Give higher weights to under-represented group-label combinations, oversample them or remove biased features and proxies.
  • Example: Weighting women applicants' successful examples more heavily in a hiring dataset.
  • When to use it: Use it when you can change the training data but not the learning algorithm.

3. In-processing: fairness constraints

Building fairness into the training objective.

  • How it works: Add a penalty for unfairness or a constraint such as equal opportunity; methods include Fairlearn's ExponentiatedGradient and adversarial debiasing.
  • Example: Training a credit model with a constraint that true positive rates differ by under 5 points between groups.
  • When to use it: Use it when you control training and need a specific fairness guarantee.

4. Post-processing: threshold adjustment

Changing how predictions are turned into decisions after the model is trained.

  • How it works: Choose decision thresholds per group, or calibrate scores, so that a chosen fairness metric is met.
  • Example: Using a slightly lower threshold for a group whose qualified members were being missed.
  • When to use it: Use it when the model cannot be retrained; check that group-specific rules are legal in your setting.

5. Process and governance measures

Organisational practices that catch and prevent bias.

  • How it works: Diverse teams, impact assessments, external audits, user feedback channels and regular re-testing after launch.
  • Example: A city council publishing an annual audit of its benefits-eligibility algorithm.
  • When to use it: Use them for every system that affects people, alongside technical fixes. Give someone clear ownership of acting on what audits find.

Explained in detail

Three points of intervention

StageMethodsPros and cons
Data (pre-processing)Collect more data, re-weight, re-sample, relabel, remove proxiesFixes root causes; collecting good data takes time
Training (in-processing)Fairness constraints, adversarial debiasing, fairness-aware lossTargets a chosen metric directly; more complex to build
Decisions (post-processing)Group-specific thresholds, reject-option reviewSimple and transparent; may need group data at decision time, which can raise legal questions

The best fixes usually start with the data. If a speech model fails rural women, collecting recordings from rural women, with consent and payment, helps far more than any algorithmic trick. If labels reflect past discrimination, such as historical hiring decisions, consider a better label, such as actual job performance, or human review of a sample. Removing proxy features can help, but check that the model has not simply found other proxies. Re-weighting is a simple and often effective step: if group B makes up 20 per cent of the data but you want it to count equally, give each of its examples four times the weight of a group A example, though check that the model is not then overfitting to a small group.

Checking the fix

After any mitigation, re-run the full evaluation: overall accuracy, every fairness metric you care about, performance for each group and intersection, and qualitative review of examples. Fixes can move the problem rather than solve it; equalising selection rates might raise false approvals in one group, harming borrowers who are then over-indebted. Often the accuracy cost is small, and sometimes accuracy improves because the model now works better for groups it had neglected. Record what you changed and why in the model card. Finally, remember that some problems cannot be fixed by tuning: if a task such as predicting "criminality" from faces has no valid scientific basis, the responsible choice is not to build it.

Who does the work also matters. Teams that include people from the communities a system serves notice problems earlier, ask better questions about labels and proxies, and design more useful fixes. Paying community members for their time, sharing results with them and giving them a voice in deployment decisions turns bias mitigation from a technical exercise into a partnership. Organisations can also set up external review boards or invite independent auditors, so that the people who built a model are not the only ones judging whether it is fair enough to use.

How it works

  1. Diagnose the cause: missing data, biased labels, proxies or model behaviour.
  2. Fix the data first where possible: collect, re-weight or relabel.
  3. Add in-training or post-processing methods if gaps remain.
  4. Re-evaluate all metrics and groups, and look for new harms.
  5. Add human review for uncertain cases, document the changes and monitor after launch.

In code

reduce_bias.py
import numpy as np
from sklearn.linear_model import LogisticRegression

rng = np.random.default_rng(1)

def make(n_a, n_b):
    X_a = rng.normal(0, 1, (n_a, 2)); y_a = (X_a[:, 0] + 0.3 * X_a[:, 1] > 0).astype(int)
    X_b = rng.normal(0, 1, (n_b, 2)); y_b = (X_b[:, 1] + 0.3 * X_b[:, 0] > 0).astype(int)   # B's pattern differs
    X = np.vstack([X_a, X_b]); y = np.concatenate([y_a, y_b])
    g = np.array(["A"] * n_a + ["B"] * n_b)
    flag = (g == "B").astype(float)[:, None]
    return X, np.hstack([X, flag, X * flag]), y, g          # plain features, and features with group terms

X_tr, F_tr, y_tr, g_tr = make(900, 100)                      # group B is under-represented in training
X_te, F_te, y_te, g_te = make(500, 500)                      # a fresh, balanced test set

def report(name, model, X):
    pred = model.predict(X)
    acc = {grp: (pred[g_te == grp] == y_te[g_te == grp]).mean() for grp in ["A", "B"]}
    print(f"{name:<26} test accuracy A={acc['A']:.2f}  B={acc['B']:.2f}  gap={acc['A'] - acc['B']:.2f}")

plain = LogisticRegression().fit(X_tr, y_tr)
report("plain model", plain, X_te)

weights = np.where(g_tr == "B", 900 / 100, 1.0)             # re-weight so both groups count equally
fairer = LogisticRegression().fit(F_tr, y_tr, sample_weight=weights)
report("re-weighted + group terms", fairer, F_te)

Output

plain model                test accuracy A=0.97  B=0.71  gap=0.26
re-weighted + group terms  test accuracy A=0.99  B=0.98  gap=0.01

Try it yourself

Fixing a speech model for rural women

A health helpline's speech recogniser makes twice as many errors for rural women as for urban men.

Which stage is the fix?

Classify each mitigation.

  • Send borderline cases to a human reviewer

  • Add a fairness penalty to the loss function

  • Give examples from a small group more weight

  • Collect more recordings from under-represented speakers

  • Use different approval thresholds to equalise error rates

Common mistakes

  • Applying a mitigation without re-checking all groups and metrics.
  • Assuming fairness always costs a lot of accuracy.
  • Trying to "debias" a task that should not be automated at all.

Check yourself

Question 1

1. Collecting more data from an under-represented group is a…

Question 2

2. Group-specific thresholds are an example of…

Question 3

3. After applying a fairness fix, you should…

Summary

  • Mitigate bias in the data, in training or in decisions after scoring.
  • Better data for neglected groups is often the most effective fix; re-weighting is simple.
  • Re-evaluate everything, add human review, document changes, and do not build tasks that cannot be fair.

Finished reading?

Sources: Fairlearn: fairness assessment and mitigation toolkit · Barocas, Hardt and Narayanan, Fairness and Machine Learning