B6 · Responsible and Safe Use of AI

Real-World Cases of AI Bias

Real-world cases of AI bias are documented situations in which deployed or tested systems produced unfair results for particular groups of people. They show how bias arises in practice, what harm it causes, and how it was discovered and corrected.

8 min Beginner Jump to the activity

What you need: Nothing. This page starts from the beginning, with no programming and no formulas.

Definition

In 2020 an algorithm lowered the examination grades of thousands of students who had never sat the exam. Within a week it was withdrawn. What went wrong?

Real-world cases of AI bias are documented situations in which deployed or tested systems produced unfair results for particular groups of people. They show how bias arises in practice, what harm it causes, and how it was discovered and corrected.

In simple words

Engineers study bridges that fell down. They do it not to shame the builders but to learn exactly which bolt, which calculation or which assumption failed, so that the next bridge stands. The cases on this page are the fallen bridges of AI. Each one teaches a specific lesson.

Why it matters

It is easy to agree that bias is bad in general. Real cases show what it looks like in a job application, a courtroom or an examination result, and they show that it can be found and fixed when people look for it.

Key terms

Audit
An independent examination of a system to see whether it works fairly and as claimed.
Error rate
The share of cases a system gets wrong.
Disparity
A gap in results between groups.
Risk score
A number a system gives to a person to show how risky it judges them to be.
Algorithm
A set of steps a computer follows to reach a result.
Accountability
Being answerable for a system's results and for putting them right.

Explained in detail

Case 1: faces

In 2018 Joy Buolamwini and Timnit Gebru tested three commercial systems that judged a person's gender from a photograph of their face. They built a balanced set of test photos and counted the results separately for four groups. For lighter-skinned men the systems were wrong less than one time in a hundred. For darker-skinned women they were wrong up to about thirty-five times in a hundred. The training photos had contained far more lighter-skinned and male faces. After the study was published the companies improved their systems, and in 2020 several large companies stopped or paused the sale of face recognition to police forces.

Case 2: hiring

From 2014 a large technology company built an experimental tool to rate job applications automatically. It learned from the applications the company had received over the previous ten years, most of which came from men. The tool taught itself that male candidates were preferable. It marked down applications that contained the word "women's", as in "captain of the women's chess club", and those from graduates of two women's colleges. Engineers tried to correct it but could not be sure it would not find other ways to discriminate. The project was abandoned, and the story became public in 2018.

Case 3: speech

In 2020 researchers at Stanford University tested the speech recognition systems of five major technology companies using recorded interviews. On average the systems misheard about thirty-five words in every hundred spoken by Black speakers, against about nineteen in every hundred for white speakers. The companies had not set out to treat anyone differently. Their training data simply contained far less speech of the kind the first group used.

Case 4: risk scores in courts

Courts in parts of the United States used software that gave each arrested person a score for how likely they were to commit another crime. In 2016 journalists examined the scores of thousands of people and what happened afterwards. Among people who did not go on to commit another crime, Black defendants had been labelled high-risk almost twice as often as white defendants. The company replied that by another measure its scores were equally accurate for both groups. Both statements were true. The case showed that "fair" can be defined in more than one way, and that a system can satisfy one definition while failing another.

Case 5: examination grades

In 2020 school-leaving examinations in England were cancelled because of the pandemic. Teachers predicted each student's grades, and an algorithm then adjusted them, using mainly the past results of each school. About four in ten of the teachers' grades were lowered. A brilliant student at a school that had done poorly in earlier years could not receive a top grade, because nobody from that school had received one before. Students at small private schools were largely spared. After protests in the streets, the government withdrew the algorithm within days and used the teachers' grades.

What the cases share

  • Nobody set out to discriminate. The unfairness came from data and design.
  • Each system looked acceptable when judged by its overall accuracy.
  • The problems were found by outsiders, researchers, journalists and the people affected, who counted results group by group.
  • The remedy involved people: better data, a changed goal, human review, or withdrawing the system.

A note on India

Most well-known cases come from countries where researchers and journalists have had access to test systems. Fewer published audits in India does not mean fewer risks. The variety of languages, accents, skin tones, names, regions, castes and religions is greater here, and systems built elsewhere are often used without local testing. Researchers who have tested language models with Indian examples have found that they can reproduce stereotypes about caste and religion. Careful local testing is a necessary part of using AI responsibly in India.

How it works

CaseWhat went wrongRoot cause
Face analysis, 2018Far more errors for darker-skinned womenUnbalanced training photos
Hiring tool, reported 2018Women's applications marked downPast hiring decisions used as labels
Speech recognition, 2020Nearly twice the errors for one groupToo little speech from that group
Court risk scores, 2016One group wrongly labelled high-risk more oftenBiased records, and competing ideas of fairness
Examination grades, 2020Able students at weaker schools downgradedA school's past used to judge an individual

Try it yourself

How a hiring tool learned the wrong lesson

A company wants software to pick the best applications automatically.

Match each case to its lesson

What does each case teach?

  • Face analysis and skin tone
  • The abandoned hiring tool
  • Risk scores in courts
  • The examination grades algorithm

Common mistakes

  • Thinking these were rare accidents. The same causes are present in many systems.
  • Assuming the companies involved were careless or malicious. Most believed their systems worked well.
  • Concluding that AI should never be used. The lesson is to test, to count by group and to keep people accountable.

Check yourself

Question 1

1. Why did the experimental hiring tool mark down women's applications?

Question 2

2. How was the bias in the face analysis systems discovered?

Question 3

3. What did the dispute over court risk scores show?

Summary

  • Bias has been found in face analysis, hiring, speech recognition, court risk scores and examination grading.
  • In each case the cause lay in data, labels or goals, not in intention.
  • Each was uncovered by counting results group by group and corrected by people taking responsibility.

Finished reading?

Sources: Gender Shades (MIT Media Lab) · UNESCO: Recommendation on the Ethics of Artificial Intelligence