I3 · Mathematics for AI

Bayes' Theorem

Bayes' theorem tells you how to update the probability of a hypothesis after seeing evidence. The posterior equals the likelihood times the prior, divided by the total probability of the evidence.

12 min Intermediate · Sublevel Beginner

What you need: The earlier topics of this module.

Definition

A test for a rare disease is 99% accurate, and you test positive. What is the chance you're actually sick? Most people, including many doctors, guess wrong.

Bayes' theorem tells you how to update the probability of a hypothesis after seeing evidence. The posterior equals the likelihood times the prior, divided by the total probability of the evidence.

In simple words

You start with a belief, then see a clue, and adjust. If a friend is late, you may first think traffic. If you then hear the metro has stopped, your belief changes. Bayes' theorem is the exact maths of that adjustment: how much to believe now, given what you believed before and how well the clue fits.

Overview

  • Bayes' theorem reverses a conditional probability: P(A | B) = P(B | A) × P(A) ÷ P(B).
  • P(A) is the prior, P(B | A) the likelihood, P(B) the evidence and P(A | B) the posterior.
  • It was published after Thomas Bayes's death in 1763 and developed by Pierre-Simon Laplace.
  • It explains the base-rate fallacy: a positive test for a rare disease often still means the person is probably healthy.
  • Each new piece of evidence can update the posterior, which becomes the next prior.
  • Spam filters, medical diagnosis and many AI systems reason this way.

Why it matters

It powers Naive Bayes spam filters and Bayesian networks, and it's the base of MAP estimation and Bayesian machine learning. It also protects you from a very common error: ignoring how rare something is to begin with. That matters in fraud detection, medical AI and anywhere one class is rare.

Key terms

Prior
P(H), the probability of a hypothesis before seeing the evidence.
Likelihood
P(E|H), how probable the evidence is if the hypothesis is true.
Posterior
P(H|E), the updated probability of the hypothesis after the evidence.
Evidence
P(E), the total probability of seeing the evidence under all hypotheses.
False positive
A positive test result when the condition is absent.
Base-rate neglect
Ignoring how rare or common something is when judging evidence.

Types

Bayes' theorem is used in several types of application and model. These are the main ones.

1. Diagnostic reasoning (testing)

Working out the chance of a condition after a test result.

  • How it works: Combine the base rate (prior) with the test's sensitivity and false-positive rate to get the posterior.
  • Example: A disease affects 1% of people; a test catches 99% of cases with a 5% false-positive rate; P(sick | positive) = 0.0099 ÷ (0.0099 + 0.0495) ≈ 17%.
  • When to use it: Use it for medical tests, fraud alerts and any detector of rare events, where the base rate dominates the answer.

2. Naive Bayes classification

A classifier that applies Bayes' theorem with the naive assumption that features are independent given the class.

  • How it works: For each class it multiplies the prior by the likelihood of each feature and picks the class with the highest result; variants are Gaussian, multinomial and Bernoulli.
  • Example: Classifying an email as spam from the words "free", "winner" and "lottery".
  • When to use it: Use it as a fast, strong baseline for text classification and when training data is small.

3. Sequential (online) updating

Updating beliefs step by step as evidence arrives.

  • How it works: After each observation the posterior is computed and used as the prior for the next one.
  • Example: A robot's location estimate sharpening as each new sensor reading arrives, as in a Kalman or particle filter.
  • When to use it: Use it for tracking, navigation and any system that learns continuously from a stream of data.

4. Bayesian inference for parameters

Treating an unknown quantity, such as a conversion rate, as uncertain and updating its distribution with data.

  • How it works: A prior distribution is combined with the likelihood of the data to give a posterior distribution, often with conjugate priors such as Beta for rates.
  • Example: After 12 clicks out of 100 visitors, a Beta(1, 1) prior becomes a Beta(13, 89) posterior for the click rate.
  • When to use it: Use it when you need uncertainty ranges, have small data or want to include expert knowledge.

5. Bayesian networks

Graphs of variables linked by arrows, each with a conditional probability table.

  • How it works: The network encodes which variables directly influence which, so the joint probability is the product of each variable's probability given its parents.
  • Example: A network linking season, rain, sprinkler and wet grass, used to reason about why the grass is wet.
  • When to use it: Use them for diagnosis and decision support where cause-and-effect structure is known; module I4 covers them further.

Explained in detail

Updating beliefs with evidence

Bayes' theorem, published after the death of the English minister Thomas Bayes in 1763 and developed further by Pierre-Simon Laplace, says P(H|E) = P(E|H) × P(H) / P(E). In words: start with how likely the hypothesis was before (the prior), multiply by how well it explains the evidence (the likelihood), and divide by how likely the evidence was overall. The result is the posterior. Yesterday's posterior becomes today's prior when new evidence arrives, so belief is updated step by step, which is a good description of how careful thinking, and many AI systems, work.

The medical test example

A disease affects 1 in 100 people. A test catches 99 per cent of cases (sensitivity) and wrongly flags 5 per cent of healthy people (a 5 per cent false-positive rate). Someone tests positive. What is the chance they have the disease? Most people guess about 95 per cent. Imagine 10,000 people instead of using the formula.

GroupPeopleTest positive
Have the disease (1%)10099
Healthy (99%)9,900495 (5%)
Total10,000594

Of the 594 positives, only 99 are ill: 99 / 594 ≈ 0.17, about 17 per cent. The test is good, but the disease is rare, so false alarms from the large healthy group outnumber true cases. This is base-rate neglect, and it matters in real decisions: it is why screening programmes follow a positive result with a second, more specific test, and why a fraud system that flags "only 1 per cent" of transactions may still produce mostly false alarms.

Bayes in AI

  • Naive Bayes classifiers apply the theorem directly, combining the likelihood of each word given spam or not spam with the prior share of spam.
  • Spam filters built this way, popularised by Paul Graham's 2002 essay "A Plan for Spam", were among the first widely used machine-learning systems.
  • Bayesian methods quantify uncertainty, giving a range of plausible answers rather than one number, which matters in medicine and science.
  • Robots and self-driving cars use Bayesian filters to combine noisy sensor readings into an estimate of where they are.

How it works

WordMeaningDisease example
Prior P(H)What you believed before the evidence1 in 1,000 people have it
Likelihood P(E|H)How likely the evidence is if the hypothesis is trueA sick person tests positive 99% of the time
Evidence P(E)How common the evidence is overallAll positives, true and false
Posterior P(H|E)What you should believe nowThe chance you're sick, given a positive test

The maths

Bayes' theorem
The hypothesis, such as "has the disease"
The evidence, such as "tested positive"
Total probability of the evidence
"Not H": the hypothesis is false

1 in 1,000 people have the disease. The test catches 99% of cases, and wrongly flags 1% of healthy people.

P(sick) = 0.001       P(healthy) = 0.999
P(+ | sick) = 0.99    P(+ | healthy) = 0.01

P(+) = 0.99×0.001 + 0.01×0.999 = 0.00099 + 0.00999 = 0.01098

P(sick | +) = 0.00099 / 0.01098 ≈ 0.09

Only about 9%, despite the "99% accurate" test. Out of 100,000 people, about 99 sick people test positive, but so do about 999 healthy people. Because the disease is rare, false alarms outnumber real cases.

In code

bayes.py
def posterior(prior, true_positive, false_positive):
    evidence = true_positive * prior + false_positive * (1 - prior)
    return true_positive * prior / evidence

print("Rare disease (1 in 1000):", round(posterior(0.001, 0.99, 0.01), 3))
print("Common disease (1 in 10):", round(posterior(0.1, 0.99, 0.01), 3))

# A second positive test: the first result becomes the new prior
first = posterior(0.001, 0.99, 0.01)
print("After a second positive test:", round(posterior(first, 0.99, 0.01), 3))

Output

Rare disease (1 in 1000): 0.09
Common disease (1 in 10): 0.917
After a second positive test: 0.908

Common mistakes

  • Base-rate neglect: ignoring the prior. This is the classic error, and the worked example shows how big it can be.
  • Mixing up P(positive | sick) and P(sick | positive). They can be 99% and 9%.

Check yourself

Question 1

1. In Bayes' theorem, what is the prior?

Question 2

2. A fraud model is 99% accurate, and 1 in 10,000 payments is fraud. It flags a payment. Is it probably fraud?

Question 3

3. What happens to the posterior if the prior goes up?

Summary

  • Posterior = likelihood × prior ÷ evidence.
  • It updates a belief using new evidence.
  • Never ignore the base rate: rare things stay fairly unlikely even after a good test.

Finished reading?

Sources: Deisenroth, Faisal & Ong, Mathematics for Machine Learning · Russell & Norvig, AI: A Modern Approach, ch. 12