Definition
A voice assistant understands one grandmother perfectly and another hardly at all. Neither is speaking unclearly. What is going on?
Bias in AI is a consistent pattern of unfair results, in which a system works better for some groups of people than for others, or treats them differently without good reason. It usually comes from the data and from the decisions people made in building the system, not from any intention of the machine.
In simple words
A tailor who has only ever measured tall customers will cut clothes that hang badly on short ones. The tailor is not unkind. They have simply never practised on shorter people. An AI system that learned mostly from one group fits that group well and everyone else poorly. Nobody intended unfairness, yet the result is unfair.
Why it matters
AI now helps to decide who is shortlisted for a job, who is offered a loan, whose face unlocks a door and whose voice is understood. If such systems are unfair, they are unfair to very large numbers of people at once, and often to those who were already at a disadvantage.
Key terms
- Bias (in AI)
- A consistent unfairness in a system's results towards some groups.
- Fairness
- Treating people justly, so that a system serves every group well.
- Discrimination
- Treating people worse because of who they are, such as their gender, colour, language or religion.
- Stereotype
- A fixed and oversimplified belief about a group of people.
- Allocation harm
- Unfairly withholding an opportunity or resource, such as a job or a loan.
- Representation harm
- Showing a group in a demeaning or stereotyped way, or leaving it out altogether.
Explained in detail
Unfair without meaning to be
When we call a person biased, we usually mean they hold a prejudice. A machine holds nothing. It has no opinions about accents, skin colours or surnames. But it learns from examples, and examples carry the shape of the world that produced them. A speech system trained mostly on recordings from big cities will hear village speech less well. It has no dislike of villages. It simply had less practice. For the person who is misunderstood, the reason makes no difference. The service fails them either way.
Different is not always unfair
Not every difference in treatment is bias. A medical tool that recommends different doses for children and adults is correct to do so, because the difference is relevant to the task. Bias is a difference that has no good justification and causes harm: a loan refused because of a surname, a face not recognised because of its colour, a job application marked down because of a gap taken to raise a child.
Two kinds of harm
Researchers describe two main ways in which biased systems hurt people. The first is in allocation: an opportunity or a resource is unfairly given or withheld. Loans, jobs, admissions, insurance and medical care fall here. The second is in representation: a group is shown in a stereotyped or demeaning way, or is invisible. An image tool that draws only men when asked for an engineer, or a translation tool that turns a sentence with no gender into "he is a doctor" and "she is a nurse", teaches its users a narrow picture of the world.
Why machine bias is especially serious
- Scale: one biased officer affects the people who meet them. One biased system affects millions.
- Invisibility: each person sees only their own result. The pattern appears only when someone measures across groups.
- Authority: people tend to believe a computer is neutral, so its decisions are questioned less.
- Repetition: decisions made today become the data that tomorrow's system learns from.
There is a hopeful side. A biased system can be measured in ways a biased person cannot. Its results can be counted group by group, its data examined and its errors corrected. Whether that is done depends on people insisting that it should be.
Who is most often affected
The people least present in the data tend to be those with less power and money to begin with: speakers of smaller languages, people with regional accents, women in fields dominated by men, people with darker skin, older people, people with disabilities, and communities who were treated unfairly in the past records a system learns from. In India this includes differences of language, region, caste, religion and income. A system built without them in mind can quietly widen gaps that society is trying to close.
How it works
| Example | Who is disadvantaged | Kind of harm |
|---|---|---|
| A voice assistant that fails on regional accents | Speakers outside big cities | A service that does not work for them |
| A hiring tool that ranks women's applications lower | Women applicants | Allocation |
| An image tool that shows only men as scientists | Women and girls | Representation |
| Face unlock that fails more often on darker skin | People with darker skin | A service that does not work for them |
| A loan model that scores a locality as risky | Everyone living there | Allocation |
Try it yourself
The voice assistant and three speakers
Three people ask the same assistant the same question: "What time does the bus leave?"
Allocation or representation?
Is an opportunity being withheld, or is a group being shown unfairly?
A job filter rejects anyone with a gap in their work history
A story generator always makes the hero a boy
An insurance model charges one community more for no good reason
A scholarship tool gives lower scores to applicants from rural schools
A search for "nurse" shows only women
An image tool shows people from one region only as farm workers
Common mistakes
- Thinking a machine cannot be biased because it has no feelings.
- Thinking bias always means someone intended to discriminate.
- Judging a system by its overall accuracy without looking at each group.
Check yourself
Summary
- Bias in AI is a consistent pattern of unfair results for some groups.
- It comes from data and design choices, not from intention.
- It can withhold opportunities or spread stereotypes, at a very large scale.
Finished reading?
Sources: AI4K12: Five Big Ideas in AI · UNESCO: Recommendation on the Ethics of Artificial Intelligence