Definition
In 1936 a magazine asked over two million people how they would vote, and predicted the wrong winner by a landslide. A smaller poll of fifty thousand got it right. How?
A population is the whole group you want to learn about. A sample is the smaller group you actually collect data from. A sample is representative when it contains the same mix of people as the population, so that its results can be trusted to describe the whole group.
In simple words
To know whether a pot of dal needs more salt, you do not eat the whole pot. You stir it well and taste one spoonful. The spoonful is the sample; the pot is the population. Stirring is what makes the spoonful representative. Taste only from the top without stirring and you may get a very different answer.
Why it matters
Almost all surveys, polls and studies use samples, because asking everyone is too slow and costly. AI training data is also a sample of the world. A sample that leaves people out produces conclusions, and AI systems, that fail those people.
Key terms
- Population
- The entire group you want to understand.
- Sample
- The part of the population you actually collect data from.
- Representative sample
- A sample with the same mix of people as the population.
- Random sample
- A sample in which every member of the population has an equal chance of being chosen.
- Sampling bias
- A systematic difference between the sample and the population.
- Margin of error
- How far a sample's result may differ from the true population value, because of chance.
Explained in detail
The poll that failed
In 1936 the Literary Digest, a popular American magazine, mailed questionnaires to about ten million people and received around 2.4 million replies. It predicted that Alf Landon would defeat President Franklin Roosevelt easily. Roosevelt won one of the largest victories in American history. The magazine had drawn its names mainly from telephone directories and car registration lists. In the middle of the Great Depression, people who owned telephones and cars were wealthier than average, and wealthier voters favoured Landon. Many people also did not reply, and those who did were not typical. Meanwhile George Gallup, using a much smaller but carefully chosen sample, predicted Roosevelt's win. Size did not save a biased sample.
How to choose a fair sample
| Method | How it works | Strength |
|---|---|---|
| Simple random | Draw names by lottery from the full list | Everyone has an equal chance |
| Stratified | Split the population into groups, then sample from each in proportion | Every group is properly represented |
| Systematic | Choose every tenth name on a list | Simple and spread out |
| Convenience | Ask whoever is easiest to reach | Quick, but often biased |
How big should a sample be?
Bigger samples reduce the effect of chance, but only up to a point, and only if they are representative. Surprisingly, the size needed depends far more on how precise you want the answer to be than on how big the population is. National opinion polls in large countries often use a few thousand people, and a carefully chosen sample of about a thousand can usually estimate a percentage to within about three percentage points either way. This allowance for chance is called the margin of error. It does not cover bias: a biased sample of a million is still wrong.
Samples and AI
An AI model's training data is a sample of the world it will be used in. If a speech model's sample contains mostly city voices, or a face model's sample mostly lighter skin, it will perform worse for those left out, exactly as the 1936 poll misjudged the voters it missed. Asking "is this sample representative of the people the system will serve?" is one of the most important checks in building fair AI.
How it works
- Define the population exactly: whom do you want to learn about?
- Get a list of the population if possible.
- Choose people randomly, or randomly within important groups.
- Make the sample big enough for the precision you need.
- Compare the sample's mix with the population and note any gaps.
Try it yourself
The poll with two million answers
1936, United States: who will win the presidential election?
Representative or biased?
Would each sample give a fair picture of the whole school?
Ten students drawn by lottery from every class
Your own friends
Every tenth name on the full school register
Students who reply to an optional online poll
Only the students in the cricket team
Common mistakes
- Believing a bigger sample is always better, even if it is biased.
- Asking only people who are easy to reach.
- Treating an open online poll as the view of the public.
Check yourself
Summary
- A sample is a part used to learn about the whole population.
- Representative samples, often chosen randomly, give trustworthy results.
- Size cannot fix bias, and AI training data is a sample too.
Finished reading?
Sources: Pew Research Center: Writing survey questions · AI4K12: Five Big Ideas in AI