B2 · How Artificial Intelligence Works: Data and Learning

How Data Shapes What AI Learns

A model learns only from the data it is given, so data that leaves out some groups of people produces a model that works worse, or unfairly, for them. This is one of the main sources of bias in AI.

6 min Beginner Jump to the activity

What you need: Nothing. This page starts from the beginning, with no programming and no formulas.

Definition

What if a face model only ever saw one kind of face?

A model learns only from the data it is given, so data that leaves out some groups of people produces a model that works worse, or unfairly, for them. This is one of the main sources of bias in AI.

In simple words

Imagine teaching someone what 'food' looks like using only photos of pizza. They might not recognise dosa or rajma chawal as food. Real systems have done this: some early face-recognition systems made far more mistakes on darker-skinned women, because few such faces were in their training data.

Why it matters

AI is used in hiring, loans, healthcare and policing. If its data leaves people out, it can treat them unfairly at a huge scale. Checking who is in the data, and who is missing, is a responsibility for everyone who builds or buys AI.

Key terms

Bias (in AI)
A consistent unfairness in a system's results that disadvantages some groups of people.
Representation
How well each group of people or kind of case appears in the data.
Under-represented group
A group that appears in the data far less than it does in real life.
Historical bias
Unfairness in past decisions that is recorded in data and then learned by a model.
Fairness
Treating people justly, so that a system works well for every group it affects.

Explained in detail

A model is a mirror of its data

A model has no opinions and no intentions. It reflects whatever patterns its training data contains. If the data shows the world fully and fairly, the model can be fair. If the data shows only part of the world, or records unfair treatment from the past, the model reproduces that, and it does so at a scale no single person could: the same model may make millions of decisions a day.

Who is missing?

The most common problem is that some groups are barely present in the data. In 2018 researchers Joy Buolamwini and Timnit Gebru tested commercial systems that guessed a person's gender from a photo of their face. For lighter-skinned men the error rate was under one per cent. For darker-skinned women it rose as high as about 35 per cent. The systems were not designed to discriminate. Their training photos contained far more lighter-skinned and male faces, so that is what they learned best.

The same thing happens with voices and languages. A speech recogniser trained mostly on city accents struggles with rural ones. A translation system trained on huge amounts of English and Hindi text and very little Santali or Bodo text serves those speakers poorly. For a country as varied as India, with its many languages, accents, skin tones, kinds of clothing and ways of living, this matters a great deal.

When the past was unfair

A second problem arises even when everyone is present in the data. Suppose a company trains a model on ten years of its own hiring decisions, and in those years it mostly hired men. The model learns that successful applicants look like men, and marks women down. A real company found exactly this in an experimental hiring tool and abandoned it. The data was accurate; it faithfully recorded what happened. What happened was unfair, and the model learned the unfairness as if it were a rule.

Easy data is not fair data

Bias is rarely deliberate. It usually comes from collecting the data that is easiest to get. Photos on the internet over-represent some countries. Medical records come from people who could afford to visit a hospital. Online reviews come from people who have smartphones and time to write. Each convenient source leaves somebody out, and those left out are often the people who are already less well served.

What careful teams do

  • Before collecting, list every group the system will be used on.
  • Count how many examples each group has, and collect more where numbers are low.
  • Test the finished model on each group separately, not only overall.
  • Include people from the affected groups in designing and testing.
  • Keep a person able to review and overturn decisions that matter.

How it works

Questions to ask about any dataset

  • Who collected it, and for what purpose?
  • Which groups of people, places or situations are in it?
  • Who is missing or rare?
  • Does the model work equally well for every group? Has anyone tested this?

Try it yourself

The bag of examples

The model is the same each time. Only the examples change. Slide to see who it works for.

The biased bag of examples

Try this with paper slips or small objects.

  1. Put 18 red slips and 2 blue slips in a bag. Each slip is an 'example'.
  2. A friend draws 10 slips without looking. This is the 'training data'.
  3. Ask: from this training data, what would a model think slips usually look like?
  4. Now imagine the blue slips are a group of people. How well would the model know them?

Think about it

  • How could you change the bag to make the training data fair?

Fair or unfair data?

Would each dataset likely give a model that works well for everyone?

  • Photos of faces of every skin tone, balanced in number

  • Job applications from the last ten years, when one group was rarely hired

  • Medical scans from hospitals across many regions, ages and genders

  • Voice recordings only from people in one city

Common mistakes

  • Thinking AI is neutral because it's a machine.
  • Testing only on average and never per group.
  • Thinking bias is someone else's problem. Users can ask questions and report unfair results.

Check yourself

Question 1

1. A model works badly for a group missing from its data. This is…

Question 2

2. Which question helps find bias?

Question 3

3. Is bias in AI always deliberate?

Summary

  • A model learns only what its data shows.
  • Data that leaves people out makes AI unfair to them.
  • Always ask who is in the data and who is missing.

Finished reading?

Sources: Gender Shades (Buolamwini and Gebru, 2018)