B6 · Responsible and Safe Use of AI

What Is Bias in AI?

Bias in AI is a consistent pattern of unfair results, in which a system works better for some groups of people than for others, or treats them differently without good reason. It usually comes from the data and from the decisions people made in building the system, not from any intention of the machine.

8 min Beginner Jump to the activity

What you need: Nothing. This page starts from the beginning, with no programming and no formulas.

Definition

A voice assistant understands one grandmother perfectly and another hardly at all. Neither is speaking unclearly. What is going on?

Bias in AI is a consistent pattern of unfair results, in which a system works better for some groups of people than for others, or treats them differently without good reason. It usually comes from the data and from the decisions people made in building the system, not from any intention of the machine.

In simple words

A tailor who has only ever measured tall customers will cut clothes that hang badly on short ones. The tailor is not unkind. They have simply never practised on shorter people. An AI system that learned mostly from one group fits that group well and everyone else poorly. Nobody intended unfairness, yet the result is unfair.

Why it matters

AI now helps to decide who is shortlisted for a job, who is offered a loan, whose face unlocks a door and whose voice is understood. If such systems are unfair, they are unfair to very large numbers of people at once, and often to those who were already at a disadvantage.

Key terms

Bias (in AI)
A consistent unfairness in a system's results towards some groups.
Fairness
Treating people justly, so that a system serves every group well.
Discrimination
Treating people worse because of who they are, such as their gender, colour, language or religion.
Stereotype
A fixed and oversimplified belief about a group of people.
Allocation harm
Unfairly withholding an opportunity or resource, such as a job or a loan.
Representation harm
Showing a group in a demeaning or stereotyped way, or leaving it out altogether.

Explained in detail

Unfair without meaning to be

When we call a person biased, we usually mean they hold a prejudice. A machine holds nothing. It has no opinions about accents, skin colours or surnames. But it learns from examples, and examples carry the shape of the world that produced them. A speech system trained mostly on recordings from big cities will hear village speech less well. It has no dislike of villages. It simply had less practice. For the person who is misunderstood, the reason makes no difference. The service fails them either way.

Different is not always unfair

Not every difference in treatment is bias. A medical tool that recommends different doses for children and adults is correct to do so, because the difference is relevant to the task. Bias is a difference that has no good justification and causes harm: a loan refused because of a surname, a face not recognised because of its colour, a job application marked down because of a gap taken to raise a child.

Two kinds of harm

Researchers describe two main ways in which biased systems hurt people. The first is in allocation: an opportunity or a resource is unfairly given or withheld. Loans, jobs, admissions, insurance and medical care fall here. The second is in representation: a group is shown in a stereotyped or demeaning way, or is invisible. An image tool that draws only men when asked for an engineer, or a translation tool that turns a sentence with no gender into "he is a doctor" and "she is a nurse", teaches its users a narrow picture of the world.

Why machine bias is especially serious

  • Scale: one biased officer affects the people who meet them. One biased system affects millions.
  • Invisibility: each person sees only their own result. The pattern appears only when someone measures across groups.
  • Authority: people tend to believe a computer is neutral, so its decisions are questioned less.
  • Repetition: decisions made today become the data that tomorrow's system learns from.

There is a hopeful side. A biased system can be measured in ways a biased person cannot. Its results can be counted group by group, its data examined and its errors corrected. Whether that is done depends on people insisting that it should be.

Who is most often affected

The people least present in the data tend to be those with less power and money to begin with: speakers of smaller languages, people with regional accents, women in fields dominated by men, people with darker skin, older people, people with disabilities, and communities who were treated unfairly in the past records a system learns from. In India this includes differences of language, region, caste, religion and income. A system built without them in mind can quietly widen gaps that society is trying to close.

How it works

ExampleWho is disadvantagedKind of harm
A voice assistant that fails on regional accentsSpeakers outside big citiesA service that does not work for them
A hiring tool that ranks women's applications lowerWomen applicantsAllocation
An image tool that shows only men as scientistsWomen and girlsRepresentation
Face unlock that fails more often on darker skinPeople with darker skinA service that does not work for them
A loan model that scores a locality as riskyEveryone living thereAllocation

Try it yourself

The voice assistant and three speakers

Three people ask the same assistant the same question: "What time does the bus leave?"

Allocation or representation?

Is an opportunity being withheld, or is a group being shown unfairly?

  • A job filter rejects anyone with a gap in their work history

  • A story generator always makes the hero a boy

  • An insurance model charges one community more for no good reason

  • A scholarship tool gives lower scores to applicants from rural schools

  • A search for "nurse" shows only women

  • An image tool shows people from one region only as farm workers

Common mistakes

  • Thinking a machine cannot be biased because it has no feelings.
  • Thinking bias always means someone intended to discriminate.
  • Judging a system by its overall accuracy without looking at each group.

Check yourself

Question 1

1. Where does bias in AI usually come from?

Question 2

2. An image tool shows only men when asked for "a pilot". Which harm is this?

Question 3

3. Why can bias in an AI system be more serious than bias in one person?

Summary

  • Bias in AI is a consistent pattern of unfair results for some groups.
  • It comes from data and design choices, not from intention.
  • It can withhold opportunities or spread stereotypes, at a very large scale.

Finished reading?

Sources: AI4K12: Five Big Ideas in AI · UNESCO: Recommendation on the Ethics of Artificial Intelligence