I3 · Mathematics for AI

Probability Basics

Probability is a number between 0 and 1 that measures how likely an event is, where 0 means impossible and 1 means certain. When all outcomes are equally likely, it's the number of favourable outcomes divided by the total number of outcomes.

10 min Intermediate · Sublevel Beginner

What you need: The earlier topics of this module.

Definition

Probability is a number between 0 and 1 that measures how likely an event is, where 0 means impossible and 1 means certain. When all outcomes are equally likely, it's the number of favourable outcomes divided by the total number of outcomes.

In simple words

Roll a die. There are 6 faces and 3 of them are even, so the chance of an even number is 3/6 = 0.5. A weather app saying 70% rain is a probability too: of many days like today, about 7 in 10 had rain.

Overview

  • Probability measures how likely an event is, on a scale from 0 (impossible) to 1 (certain).
  • The sample space lists every possible outcome; an event is a set of outcomes.
  • For equally likely outcomes, P(event) = favourable outcomes ÷ total outcomes.
  • The complement rule: P(not A) = 1 − P(A).
  • Addition rule: P(A or B) = P(A) + P(B) − P(A and B); multiplication for independent events: P(A and B) = P(A) × P(B).
  • Every classifier that outputs a probability is applying these ideas.

Why it matters

A parcel classifier may return a label such as delayed, or estimate how likely a delay is. A language model predicts a distribution over its next token, which could be a whole word or a piece of one. Probability gives us a way to describe and test such predictions.

Key terms

Sample space
The set of all possible outcomes of an experiment.
Event
A set of outcomes we are interested in, such as "an even number".
Complement
The event "not A"; its probability is 1 − P(A).
Mutually exclusive
Events that cannot happen together, so P(A or B) = P(A) + P(B).
Addition rule
P(A or B) = P(A) + P(B) − P(A and B), avoiding double counting.
Law of large numbers
Over many trials, the observed frequency approaches the true probability.

Types

Probability can be understood in three classic ways, and events relate to each other in several types. Both are explained here.

1. Classical (theoretical) probability

Probability worked out by counting equally likely outcomes.

  • How it works: Divide the number of favourable outcomes by the total number of outcomes, without doing any experiment.
  • Example: Rolling a fair die: P(6) = 1 ÷ 6; drawing a heart from a deck: 13 ÷ 52 = 0.25.
  • When to use it: Use it for games, coins, cards and any situation where outcomes are known to be equally likely.

2. Empirical (experimental) probability

Probability estimated from how often something happened in past trials or data.

  • How it works: Divide the number of times the event occurred by the number of trials; the law of large numbers says it approaches the true probability as trials grow.
  • Example: If it rained on 45 of the 90 monsoon days last year, the estimated chance of rain is 0.5.
  • When to use it: Use it whenever outcomes are not equally likely and data is available, which is how machine learning estimates probabilities.

3. Subjective probability

A personal degree of belief based on knowledge and judgement.

  • How it works: It is stated by experts when there is little data, and can be updated with evidence using Bayes' theorem.
  • Example: A doctor estimating a 30% chance that a patient's symptoms are caused by dengue before tests.
  • When to use it: Use it carefully as a starting prior in Bayesian reasoning, and state it openly.

4. Mutually exclusive and independent events

Mutually exclusive events cannot happen together; independent events do not affect each other's probability.

  • How it works: For exclusive events P(A or B) = P(A) + P(B); for independent events P(A and B) = P(A) × P(B); exclusive events with non-zero chances are never independent.
  • Example: Getting 2 and 5 on one die roll is exclusive; heads on two separate coin tosses are independent (0.5 × 0.5 = 0.25).
  • When to use it: Use these rules to combine probabilities correctly, and check independence instead of assuming it.

5. Dependent (conditional) events

Events where one outcome changes the probability of the other.

  • How it works: P(A and B) = P(A) × P(B | A), where P(B | A) is the probability of B once A has happened.
  • Example: Drawing two aces without replacement: 4/52 × 3/51 ≈ 0.0045.
  • When to use it: Use it whenever events are linked, such as weather on consecutive days, which leads to conditional probability.

Explained in detail

Two ways to think about probability

The frequentist view treats probability as long-run frequency: a fair coin has P(heads) = 0.5 because in thousands of tosses, heads comes up about half the time. The Bayesian view treats probability as a degree of belief, which can apply to one-off events: "there is a 70 per cent chance this email is spam" or "a 30 per cent chance this patient has the disease". Machine learning uses both. Training data is analysed with frequencies, while a model's output for one particular email is best read as a calibrated degree of belief.

The core rules

RuleStatementExample with one die
Range0 ≤ P(A) ≤ 1P(7) = 0, P(1 to 6) = 1
Equally likely outcomesP(A) = favourable ÷ totalP(even) = 3/6 = 0.5
ComplementP(not A) = 1 − P(A)P(not a six) = 5/6
AdditionP(A or B) = P(A) + P(B) − P(A and B)P(even or > 4) = 3/6 + 2/6 − 1/6 = 4/6
Multiplication (independent)P(A and B) = P(A) × P(B)Two sixes in two rolls: 1/36

Counting carefully

Many probability mistakes are counting mistakes. With two dice there are 36 equally likely ordered outcomes, not 11 possible totals, so a total of 7 (six ways) is six times as likely as a total of 2 (one way). The complement rule often turns a hard count into an easy one. The chance that at least two people in a group of 23 share a birthday seems small, but it is easier to compute the chance that all 23 birthdays differ, which is about 49 per cent, so the chance of a shared birthday is about 51 per cent. This famous birthday paradox shows how badly intuition handles combinations.

Probability inside AI

  • Classifiers output probabilities: a spam model might return 0.92 for spam and 0.08 for not spam, and the two must add to 1.
  • Language models predict a probability for every possible next token, then sample from that distribution.
  • Simulation: when exact calculation is hard, programs estimate probabilities by running thousands of random trials, a method called Monte Carlo.
  • Python's random module and NumPy's np.random generate random numbers for such simulations; setting a seed makes results reproducible.

How it works

TermMeaningDie example
ExperimentSomething with an uncertain resultRolling a die
Sample space (S)All possible outcomes{1, 2, 3, 4, 5, 6}
EventA set of outcomes you care aboutRolling an even number

The rules

  • Every probability is between 0 and 1, and all outcomes together add to 1.
  • Complement: the chance something does not happen is 1 minus the chance it does.
  • Or: add the two chances, then subtract the overlap so it isn't counted twice.
  • And: for independent events, multiply the two chances.

Counting outcomes

Use permutations when order matters, such as a PIN. Use combinations when it doesn't, such as choosing a team.

The maths

Equally likely outcomes
Complement
Addition rule
Or: either event happens
And: both events happen
Independent events
Permutations and combinations

Pick one card from a 52-card deck. What is the chance of a heart or a king?

P(heart) = 13/52
P(king)  = 4/52
P(both)  = 1/52   (the king of hearts)
P(heart or king) = 13/52 + 4/52 − 1/52 = 16/52 ≈ 0.308

In code

Simulating 100,000 rolls gets very close to the exact answer of 0.5.

dice.py
import math
import random

random.seed(1)
rolls = [random.randint(1, 6) for _ in range(100_000)]
even = sum(1 for roll in rolls if roll % 2 == 0)
print("Simulated P(even):", round(even / len(rolls), 3))

print("Ways to pick 2 people from 5, order ignored:", math.comb(5, 2))
print("Ways to arrange 2 people from 5, order matters:", math.perm(5, 2))

Output

Simulated P(even): 0.501
Ways to pick 2 people from 5, order ignored: 10
Ways to arrange 2 people from 5, order matters: 20

Common mistakes

  • Adding probabilities without subtracting the overlap.
  • Multiplying probabilities of events that aren't independent.
  • The gambler's fallacy: after 5 heads, the next toss of a fair coin is still 50/50.

Check yourself

Question 1

1. What is the chance of rolling a number above 4 on a fair die?

Question 2

2. Toss two fair coins. What is the chance of two heads?

Question 3

3. P(rain) = 0.3. What is P(no rain)?

Summary

  • Probability runs from 0 to 1, and all outcomes add to 1.
  • Or: add and subtract the overlap. And (independent): multiply.
  • Order matters: permutation. Order doesn't: combination.

Finished reading?

Sources: Deisenroth, Faisal & Ong, Mathematics for Machine Learning