I10 · Responsible AI in Practice

Measuring Bias in Data

Data bias is any systematic difference between the training data and the reality the model will face, or any pattern in the data that reflects unfair treatment. Common forms are representation bias (some groups are under-represented), historical bias (the data records past discrimination, such as fewer loans approved for women), measurement bias (a feature or label means different things for different groups, such as using healthcare spending as a proxy for health need), label bias (labellers apply labels inconsistently) and sampling bias (the way data was collected excludes some people). Measuring it involves comparing group counts, label rates and data quality across groups.

12 min Intermediate · Sublevel Advanced Jump to the activity

What you need: The earlier topics of this module.

Definition

A health app's training data contains 80,000 records from city hospitals and 3,000 from rural clinics. The model may work beautifully in Mumbai and poorly in a village in Jharkhand, and no one notices until people are harmed. Measuring bias starts with looking hard at the data.

Data bias is any systematic difference between the training data and the reality the model will face, or any pattern in the data that reflects unfair treatment. Common forms are representation bias (some groups are under-represented), historical bias (the data records past discrimination, such as fewer loans approved for women), measurement bias (a feature or label means different things for different groups, such as using healthcare spending as a proxy for health need), label bias (labellers apply labels inconsistently) and sampling bias (the way data was collected excludes some people). Measuring it involves comparing group counts, label rates and data quality across groups.

In simple words

If you learn about India only from photos of Delhi, you will get many things wrong about Kerala or Nagaland. A model learns only from its data, so if the data leaves people out or carries old unfairness, the model will too.

Overview

  • Data bias means the data does not fairly represent the people or situations the model will serve.
  • Biased data produces biased models, even with a perfect algorithm.
  • Measuring bias starts with counting how many examples each group has.
  • Label rates, missing values and data quality should also be compared across groups.
  • Comparisons need a reference, such as census figures for the population served.
  • Datasheets record known gaps so later users are warned.

Why it matters

Most unfair AI outcomes begin in the data. Checking data before training is cheaper than fixing a deployed model, and it often reveals problems that no algorithm can correct on its own.

Key terms

Representation bias
Some groups appear far less often in the data than in the real population served.
Historical bias
Data that faithfully records past unfair decisions or inequalities.
Measurement bias
A feature or label that is a poorer proxy for some groups than others.
Proxy variable
A feature that indirectly encodes a sensitive attribute, such as PIN code standing in for caste or religion.
Base rate
How often the positive label occurs in a group, such as the approval rate.
Sensitive attribute
A characteristic such as gender, caste, religion, disability or age that should not drive unfair treatment.

Types

Data bias comes in several types. Each has its own way to measure it.

1. Representation (sampling) bias

Some groups are under- or over-represented compared with the real population.

  • How it works: Count rows per group and compare shares with a reference; a ratio far from 1 shows imbalance.
  • Example: A face dataset that is 80% light-skinned while the users are mostly darker-skinned.
  • When to use it: Check it first in every dataset used for decisions about people.

2. Historical bias

Data that accurately records past unfair decisions or social inequality.

  • How it works: Compare positive-label rates across groups, for example approval or hiring rates, and ask whether differences reflect merit or past discrimination.
  • Example: Past hiring data in which only 10% of engineers hired were women.
  • When to use it: Watch for it whenever labels come from human decisions.

3. Measurement bias

Features or labels measured differently or less accurately for some groups.

  • How it works: Compare missing-value rates, sensor errors and proxy quality by group.
  • Example: Pulse oximeters that read less accurately on darker skin, giving biased health labels.
  • When to use it: Check it when using proxies, such as healthcare spending, in place of true outcomes such as health need.

4. Label (annotation) bias

Labels influenced by the opinions or culture of the people who created them.

  • How it works: Measure agreement between annotators and compare label rates for similar items from different groups.
  • Example: Posts in some dialects being labelled "toxic" more often than the same meaning in standard English.
  • When to use it: Use diverse annotators, clear guidelines and audits of label differences.

5. Aggregation and temporal bias

Treating different groups as one, or using data from a time that no longer matches today.

  • How it works: Compare patterns within subgroups and across time periods; large differences warn against one model for all.
  • Example: A diabetes model trained on adults that performs poorly for teenagers.
  • When to use it: Check it when populations are diverse or when behaviour has changed, such as after the pandemic.

Explained in detail

Real cases of data bias

In 2018 Reuters reported that Amazon had scrapped an experimental recruiting tool because, trained on ten years of mostly male résumés, it had learned to downgrade résumés containing the word "women's", as in "women's chess club captain". In 2019 a study in Science by Ziad Obermeyer and colleagues found that a widely used US healthcare algorithm, which used past healthcare costs as a proxy for health needs, under-referred Black patients, because less money had historically been spent on them at the same level of illness; correcting the label would have more than doubled the share of Black patients flagged for extra care. In speech recognition and face analysis, under-representation of some accents and skin tones led to much higher error rates. Each case shows that the data, not only the algorithm, carried the bias.

What to measure

CheckHowWarning sign
RepresentationShare of each group in the data versus in the population servedA group is far smaller than its real share
Label ratesPositive-label rate per group (for example, approval rate)Large gaps that may reflect past discrimination
Data qualityMissing values, errors and label agreement per groupOne group's records are noisier or more incomplete
ProxiesCorrelation between features and sensitive attributesPIN code, surname or school strongly predicts caste or religion
IntersectionsCombinations such as rural women or elderly Dalit menSmall subgroups hidden inside large averages

In India, relevant groups include gender, caste, religion, region, rural or urban location, language, disability and income. Collecting such attributes is sensitive and must follow the law and consent; sometimes it is better to estimate group composition from aggregate statistics or to audit with a separate, carefully governed sample. Removing a sensitive column does not remove bias, because proxies such as PIN code, surname or language can carry the same information.

Document every finding in a datasheet: who collected the data, when, how, which groups it covers well or poorly, known gaps and recommended uses. A clear record of data limits stops a model from later being used for populations it was never designed for, which is one of the most common ways AI systems cause harm. It also helps future teams decide what extra data to collect, and gives auditors and regulators the evidence they need.

How it works

  1. List the groups the system will serve and the sensitive attributes that matter.
  2. Compare each group's share in the data with its real share.
  3. Compare label rates, missing values and label quality by group.
  4. Look for proxies that encode sensitive attributes.
  5. Record findings in a datasheet and plan fixes before training.

In code

data_bias_audit.py
import pandas as pd

# A small synthetic loan data set
df = pd.DataFrame({
    "region":   ["urban"] * 14 + ["rural"] * 6,
    "gender":   ["M", "M", "F", "M", "F", "M", "M", "F", "M", "M", "F", "M", "M", "F", "M", "F", "M", "F", "F", "M"],
    "pin_zone": ["A"] * 14 + ["B"] * 6,
    "approved": [1, 1, 1, 0, 1, 1, 1, 0, 1, 1, 1, 0, 1, 1, 0, 0, 1, 0, 0, 0],
    "income":   [55, 60, 48, None, 52, 70, 65, 45, 58, 61, 50, 40, 66, 47, None, None, 38, 30, None, 35],
})
population_share = {"urban": 0.35, "rural": 0.65}            # who the bank actually serves

print("Representation (data vs population):")
for region, share in df["region"].value_counts(normalize=True).items():
    print(f"   {region:<6} {share:.0%} in data vs {population_share[region]:.0%} in population")

print("\nApproval rate by group:")
print(df.groupby("region")["approved"].mean().round(2).to_string())
print(df.groupby(["region", "gender"])["approved"].mean().round(2).to_string())

print("\nMissing income by region:")
print(df.groupby("region")["income"].apply(lambda s: s.isna().mean()).round(2).to_string())

proxy = pd.crosstab(df["pin_zone"], df["region"], normalize="index")
print("\nPIN zone predicts region perfectly (a proxy):\n", proxy)

Output

Representation (data vs population):
   urban  70% in data vs 35% in population
   rural  30% in data vs 65% in population

Approval rate by group:
region
rural    0.17
urban    0.79
region  gender
rural   F         0.00
        M         0.33
urban   F         0.80
        M         0.78

Missing income by region:
region
rural    0.50
urban    0.07

PIN zone predicts region perfectly (a proxy):
 region    rural  urban
pin_zone              
A           0.0    1.0
B           1.0    0.0

Try it yourself

Auditing a crop-insurance data set

A state plans an AI model to speed up crop-insurance claims.

Match the example to the bias type

Which kind of data bias is it?

  • A voice data set with almost no elderly speakers
  • Past hiring data with few women promoted
  • Using hospital spending as a measure of how sick someone is
  • PIN code that closely tracks community

Common mistakes

  • Assuming a large data set is automatically representative.
  • Dropping sensitive columns and declaring the data fair.
  • Checking only single groups and missing intersections.

Check yourself

Question 1

1. A data set has far fewer rural records than the rural share of customers. This is…

Question 2

2. Why did the US healthcare algorithm studied in 2019 under-refer Black patients?

Question 3

3. Why is removing the caste column not enough?

Summary

  • Data bias includes representation, historical, measurement, label and sampling bias.
  • Measure group shares, label rates, data quality, proxies and intersections.
  • Real cases in hiring and healthcare show that data, not only algorithms, carries bias.

Read it. Now try it yourself.

Open the interactive activity

Finished reading?

Sources: Obermeyer et al. (2019), Dissecting racial bias in an algorithm used to manage the health of populations, Science · Barocas, Hardt and Narayanan, Fairness and Machine Learning