B6 · Responsible and Safe Use of AI

How Bias Enters AI Systems

Bias can enter an AI system at every stage of its creation: in the choice of goal, in the data collected, in the labels attached, in the features used, in the way it is tested and in the way it is put to use. Most often it comes from data that under-represents some groups or records unfair decisions made in the past.

8 min Beginner Jump to the activity

What you need: Nothing. This page starts from the beginning, with no programming and no formulas.

Definition

A team removes "gender" from its data so that its hiring model cannot be unfair to women. The model is unfair to women anyway. How?

Bias can enter an AI system at every stage of its creation: in the choice of goal, in the data collected, in the labels attached, in the features used, in the way it is tested and in the way it is put to use. Most often it comes from data that under-represents some groups or records unfair decisions made in the past.

In simple words

A river can be polluted at its source, by a factory half-way down, or by drains near the sea. Wherever the dirt enters, the water at the end is dirty. An AI system is built in stages, like a river flowing from source to sea, and unfairness can be poured in at any of them. To clean it, you have to know where to look.

Why it matters

People often assume bias can be fixed by deleting a sensitive column from the data. It cannot. Knowing each of the doors through which bias enters is what makes it possible to find and close them.

Key terms

Sampling bias
Data that does not contain the same mix of people or cases as the real world.
Historical bias
Unfairness in past decisions that has been recorded in the data.
Label bias
Unfairness introduced by the people or rules that decide the "right answers".
Proxy
A feature that stands in for another, such as a pin code standing in for income or community.
Measurement bias
Measuring the wrong thing, or measuring it less accurately for some groups.
Feedback loop
A system's decisions shaping the data it later learns from.

Explained in detail

Door 1: the goal

Before any data is touched, people decide what the system should predict. That choice can carry bias on its own. A health system in the United States used a model to find patients who needed extra care. The designers could not measure "need" directly, so they asked the model to predict something easier to count: how much money would be spent on each patient. A study published in 2019 found the flaw. Less money had historically been spent on Black patients than on white patients with the same illnesses. The model therefore judged them healthier than they were, and far fewer were offered the extra care. When the goal was changed to predict health itself, the gap narrowed sharply.

Door 2: the data collected

This is the widest door. Data is usually gathered from wherever it is easiest to obtain. Photographs come from the countries that post most online. Speech recordings come from people who own smartphones. Medical records come from city hospitals. Whoever is harder to reach is thinly represented, and a model learns less about them. No clever method later can restore what was never collected.

Door 3: the labels

Labels are treated as the truth, but people create them, and people bring their own judgements. If reviewers are asked to mark comments as "offensive", their views of what counts will enter the data. Studies have found that tools trained on such labels flagged posts written in some dialects as offensive far more often than similar posts in standard language. Labels taken from past decisions, such as "hired" or "not hired", carry whatever unfairness those decisions contained.

Door 4: features and proxies

This answers the puzzle at the top of the page. Removing the column "gender" does not remove gender from the data, because other columns give it away. A first name, a college attended, a hobby, even the wording of a sentence can stand in for it. In the same way a pin code can stand in for income or community, and the brand of a phone for wealth. Such stand-ins are called proxies. A model has no idea which features are sensitive. It uses whatever predicts the label, and if the label is biased, it will find the proxies that reproduce the bias.

Door 5: testing

A model tested only as a whole can hide a serious failure. Suppose it is right 95 times in 100 overall. If one group makes up a tenth of the test set, the model could be wrong for half of that group and still report 95 per cent. Unless results are counted separately for each group, nobody will know.

Door 6: use, and the loop it creates

A system built for one setting may be used in another. A tool trained on hospital patients in one country is sold to clinics in another, where people and equipment differ. Finally, a system's own decisions can become its future training data. If a lending model refuses loans to a locality, there will be no record of people there repaying loans, and the next version of the model will be even more certain that the locality is risky.

How it works

StageQuestion to askWhat can go wrong
GoalAre we predicting what we truly care about?A convenient stand-in hides unfairness
DataWho is missing or thinly represented?The model works poorly for them
LabelsWho decided the right answers, and how?Old judgements are learned as truth
FeaturesCould any feature stand in for a sensitive one?Bias returns through a proxy
TestingHave results been counted for each group?A failure for one group stays hidden
UseIs it being used where and how it was designed for?New settings and feedback loops

Try it yourself

Find the missing group

The model stays the same. Only the mix of examples changes. Move the slider and watch who it works for.

Through which door did the bias enter?

Match each situation to the stage at which the problem began.

  • A model predicts spending as a stand-in for health need
  • Nearly all the training photos show people from two countries
  • Reviewers marked one dialect as rude more often than another
  • A pin code is used, and it reveals a person's community
  • Only the overall accuracy was reported
  • People refused a loan never appear in the repayment records

Common mistakes

  • Believing that removing gender, caste or religion from the data removes the bias.
  • Blaming only the data, when the goal or the labels were the problem.
  • Reporting one overall score and assuming it holds for everyone.

Check yourself

Question 1

1. What is a proxy?

Question 2

2. A model predicted health spending, and wrongly judged one group healthier. Where did the bias enter?

Question 3

3. Why should accuracy be reported separately for each group?

Summary

  • Bias can enter through the goal, the data, the labels, the features, the testing and the use.
  • Removing a sensitive column does not help when other features act as proxies.
  • Finding bias requires checking every stage and counting results for each group.

Finished reading?

Sources: AI4K12: Five Big Ideas in AI · UNESCO: Recommendation on the Ethics of Artificial Intelligence