Definition
A test for a rare disease is 99% accurate, and you test positive. What is the chance you're actually sick? Most people, including many doctors, guess wrong.
Bayes' theorem tells you how to update the probability of a hypothesis after seeing evidence. The posterior equals the likelihood times the prior, divided by the total probability of the evidence.
In simple words
You start with a belief, then see a clue, and adjust. If a friend is late, you may first think traffic. If you then hear the metro has stopped, your belief changes. Bayes' theorem is the exact maths of that adjustment: how much to believe now, given what you believed before and how well the clue fits.
Overview
- Bayes' theorem reverses a conditional probability: P(A | B) = P(B | A) × P(A) ÷ P(B).
- P(A) is the prior, P(B | A) the likelihood, P(B) the evidence and P(A | B) the posterior.
- It was published after Thomas Bayes's death in 1763 and developed by Pierre-Simon Laplace.
- It explains the base-rate fallacy: a positive test for a rare disease often still means the person is probably healthy.
- Each new piece of evidence can update the posterior, which becomes the next prior.
- Spam filters, medical diagnosis and many AI systems reason this way.
Why it matters
It powers Naive Bayes spam filters and Bayesian networks, and it's the base of MAP estimation and Bayesian machine learning. It also protects you from a very common error: ignoring how rare something is to begin with. That matters in fraud detection, medical AI and anywhere one class is rare.
Key terms
- Prior
- P(H), the probability of a hypothesis before seeing the evidence.
- Likelihood
- P(E|H), how probable the evidence is if the hypothesis is true.
- Posterior
- P(H|E), the updated probability of the hypothesis after the evidence.
- Evidence
- P(E), the total probability of seeing the evidence under all hypotheses.
- False positive
- A positive test result when the condition is absent.
- Base-rate neglect
- Ignoring how rare or common something is when judging evidence.
Types
Bayes' theorem is used in several types of application and model. These are the main ones.
1. Diagnostic reasoning (testing)
Working out the chance of a condition after a test result.
- How it works: Combine the base rate (prior) with the test's sensitivity and false-positive rate to get the posterior.
- Example: A disease affects 1% of people; a test catches 99% of cases with a 5% false-positive rate; P(sick | positive) = 0.0099 ÷ (0.0099 + 0.0495) ≈ 17%.
- When to use it: Use it for medical tests, fraud alerts and any detector of rare events, where the base rate dominates the answer.
2. Naive Bayes classification
A classifier that applies Bayes' theorem with the naive assumption that features are independent given the class.
- How it works: For each class it multiplies the prior by the likelihood of each feature and picks the class with the highest result; variants are Gaussian, multinomial and Bernoulli.
- Example: Classifying an email as spam from the words "free", "winner" and "lottery".
- When to use it: Use it as a fast, strong baseline for text classification and when training data is small.
3. Sequential (online) updating
Updating beliefs step by step as evidence arrives.
- How it works: After each observation the posterior is computed and used as the prior for the next one.
- Example: A robot's location estimate sharpening as each new sensor reading arrives, as in a Kalman or particle filter.
- When to use it: Use it for tracking, navigation and any system that learns continuously from a stream of data.
4. Bayesian inference for parameters
Treating an unknown quantity, such as a conversion rate, as uncertain and updating its distribution with data.
- How it works: A prior distribution is combined with the likelihood of the data to give a posterior distribution, often with conjugate priors such as Beta for rates.
- Example: After 12 clicks out of 100 visitors, a Beta(1, 1) prior becomes a Beta(13, 89) posterior for the click rate.
- When to use it: Use it when you need uncertainty ranges, have small data or want to include expert knowledge.
5. Bayesian networks
Graphs of variables linked by arrows, each with a conditional probability table.
- How it works: The network encodes which variables directly influence which, so the joint probability is the product of each variable's probability given its parents.
- Example: A network linking season, rain, sprinkler and wet grass, used to reason about why the grass is wet.
- When to use it: Use them for diagnosis and decision support where cause-and-effect structure is known; module I4 covers them further.
Explained in detail
Updating beliefs with evidence
Bayes' theorem, published after the death of the English minister Thomas Bayes in 1763 and developed further by Pierre-Simon Laplace, says P(H|E) = P(E|H) × P(H) / P(E). In words: start with how likely the hypothesis was before (the prior), multiply by how well it explains the evidence (the likelihood), and divide by how likely the evidence was overall. The result is the posterior. Yesterday's posterior becomes today's prior when new evidence arrives, so belief is updated step by step, which is a good description of how careful thinking, and many AI systems, work.
The medical test example
A disease affects 1 in 100 people. A test catches 99 per cent of cases (sensitivity) and wrongly flags 5 per cent of healthy people (a 5 per cent false-positive rate). Someone tests positive. What is the chance they have the disease? Most people guess about 95 per cent. Imagine 10,000 people instead of using the formula.
| Group | People | Test positive |
|---|---|---|
| Have the disease (1%) | 100 | 99 |
| Healthy (99%) | 9,900 | 495 (5%) |
| Total | 10,000 | 594 |
Of the 594 positives, only 99 are ill: 99 / 594 ≈ 0.17, about 17 per cent. The test is good, but the disease is rare, so false alarms from the large healthy group outnumber true cases. This is base-rate neglect, and it matters in real decisions: it is why screening programmes follow a positive result with a second, more specific test, and why a fraud system that flags "only 1 per cent" of transactions may still produce mostly false alarms.
Bayes in AI
- Naive Bayes classifiers apply the theorem directly, combining the likelihood of each word given spam or not spam with the prior share of spam.
- Spam filters built this way, popularised by Paul Graham's 2002 essay "A Plan for Spam", were among the first widely used machine-learning systems.
- Bayesian methods quantify uncertainty, giving a range of plausible answers rather than one number, which matters in medicine and science.
- Robots and self-driving cars use Bayesian filters to combine noisy sensor readings into an estimate of where they are.
How it works
| Word | Meaning | Disease example |
|---|---|---|
| Prior P(H) | What you believed before the evidence | 1 in 1,000 people have it |
| Likelihood P(E|H) | How likely the evidence is if the hypothesis is true | A sick person tests positive 99% of the time |
| Evidence P(E) | How common the evidence is overall | All positives, true and false |
| Posterior P(H|E) | What you should believe now | The chance you're sick, given a positive test |
The maths
- The hypothesis, such as "has the disease"
- The evidence, such as "tested positive"
- "Not H": the hypothesis is false
1 in 1,000 people have the disease. The test catches 99% of cases, and wrongly flags 1% of healthy people.
P(sick) = 0.001 P(healthy) = 0.999 P(+ | sick) = 0.99 P(+ | healthy) = 0.01 P(+) = 0.99×0.001 + 0.01×0.999 = 0.00099 + 0.00999 = 0.01098 P(sick | +) = 0.00099 / 0.01098 ≈ 0.09
Only about 9%, despite the "99% accurate" test. Out of 100,000 people, about 99 sick people test positive, but so do about 999 healthy people. Because the disease is rare, false alarms outnumber real cases.
In code
def posterior(prior, true_positive, false_positive):
evidence = true_positive * prior + false_positive * (1 - prior)
return true_positive * prior / evidence
print("Rare disease (1 in 1000):", round(posterior(0.001, 0.99, 0.01), 3))
print("Common disease (1 in 10):", round(posterior(0.1, 0.99, 0.01), 3))
# A second positive test: the first result becomes the new prior
first = posterior(0.001, 0.99, 0.01)
print("After a second positive test:", round(posterior(first, 0.99, 0.01), 3))Output
Rare disease (1 in 1000): 0.09 Common disease (1 in 10): 0.917 After a second positive test: 0.908
Common mistakes
- Base-rate neglect: ignoring the prior. This is the classic error, and the worked example shows how big it can be.
- Mixing up P(positive | sick) and P(sick | positive). They can be 99% and 9%.
Check yourself
Summary
- Posterior = likelihood × prior ÷ evidence.
- It updates a belief using new evidence.
- Never ignore the base rate: rare things stay fairly unlikely even after a good test.
Finished reading?
Sources: Deisenroth, Faisal & Ong, Mathematics for Machine Learning · Russell & Norvig, AI: A Modern Approach, ch. 12