Definition
Probability is a number between 0 and 1 that measures how likely an event is, where 0 means impossible and 1 means certain. When all outcomes are equally likely, it's the number of favourable outcomes divided by the total number of outcomes.
In simple words
Roll a die. There are 6 faces and 3 of them are even, so the chance of an even number is 3/6 = 0.5. A weather app saying 70% rain is a probability too: of many days like today, about 7 in 10 had rain.
Overview
- Probability measures how likely an event is, on a scale from 0 (impossible) to 1 (certain).
- The sample space lists every possible outcome; an event is a set of outcomes.
- For equally likely outcomes, P(event) = favourable outcomes ÷ total outcomes.
- The complement rule: P(not A) = 1 − P(A).
- Addition rule: P(A or B) = P(A) + P(B) − P(A and B); multiplication for independent events: P(A and B) = P(A) × P(B).
- Every classifier that outputs a probability is applying these ideas.
Why it matters
A parcel classifier may return a label such as delayed, or estimate how likely a delay is. A language model predicts a distribution over its next token, which could be a whole word or a piece of one. Probability gives us a way to describe and test such predictions.
Key terms
- Sample space
- The set of all possible outcomes of an experiment.
- Event
- A set of outcomes we are interested in, such as "an even number".
- Complement
- The event "not A"; its probability is 1 − P(A).
- Mutually exclusive
- Events that cannot happen together, so P(A or B) = P(A) + P(B).
- Addition rule
- P(A or B) = P(A) + P(B) − P(A and B), avoiding double counting.
- Law of large numbers
- Over many trials, the observed frequency approaches the true probability.
Types
Probability can be understood in three classic ways, and events relate to each other in several types. Both are explained here.
1. Classical (theoretical) probability
Probability worked out by counting equally likely outcomes.
- How it works: Divide the number of favourable outcomes by the total number of outcomes, without doing any experiment.
- Example: Rolling a fair die: P(6) = 1 ÷ 6; drawing a heart from a deck: 13 ÷ 52 = 0.25.
- When to use it: Use it for games, coins, cards and any situation where outcomes are known to be equally likely.
2. Empirical (experimental) probability
Probability estimated from how often something happened in past trials or data.
- How it works: Divide the number of times the event occurred by the number of trials; the law of large numbers says it approaches the true probability as trials grow.
- Example: If it rained on 45 of the 90 monsoon days last year, the estimated chance of rain is 0.5.
- When to use it: Use it whenever outcomes are not equally likely and data is available, which is how machine learning estimates probabilities.
3. Subjective probability
A personal degree of belief based on knowledge and judgement.
- How it works: It is stated by experts when there is little data, and can be updated with evidence using Bayes' theorem.
- Example: A doctor estimating a 30% chance that a patient's symptoms are caused by dengue before tests.
- When to use it: Use it carefully as a starting prior in Bayesian reasoning, and state it openly.
4. Mutually exclusive and independent events
Mutually exclusive events cannot happen together; independent events do not affect each other's probability.
- How it works: For exclusive events P(A or B) = P(A) + P(B); for independent events P(A and B) = P(A) × P(B); exclusive events with non-zero chances are never independent.
- Example: Getting 2 and 5 on one die roll is exclusive; heads on two separate coin tosses are independent (0.5 × 0.5 = 0.25).
- When to use it: Use these rules to combine probabilities correctly, and check independence instead of assuming it.
5. Dependent (conditional) events
Events where one outcome changes the probability of the other.
- How it works: P(A and B) = P(A) × P(B | A), where P(B | A) is the probability of B once A has happened.
- Example: Drawing two aces without replacement: 4/52 × 3/51 ≈ 0.0045.
- When to use it: Use it whenever events are linked, such as weather on consecutive days, which leads to conditional probability.
Explained in detail
Two ways to think about probability
The frequentist view treats probability as long-run frequency: a fair coin has P(heads) = 0.5 because in thousands of tosses, heads comes up about half the time. The Bayesian view treats probability as a degree of belief, which can apply to one-off events: "there is a 70 per cent chance this email is spam" or "a 30 per cent chance this patient has the disease". Machine learning uses both. Training data is analysed with frequencies, while a model's output for one particular email is best read as a calibrated degree of belief.
The core rules
| Rule | Statement | Example with one die |
|---|---|---|
| Range | 0 ≤ P(A) ≤ 1 | P(7) = 0, P(1 to 6) = 1 |
| Equally likely outcomes | P(A) = favourable ÷ total | P(even) = 3/6 = 0.5 |
| Complement | P(not A) = 1 − P(A) | P(not a six) = 5/6 |
| Addition | P(A or B) = P(A) + P(B) − P(A and B) | P(even or > 4) = 3/6 + 2/6 − 1/6 = 4/6 |
| Multiplication (independent) | P(A and B) = P(A) × P(B) | Two sixes in two rolls: 1/36 |
Counting carefully
Many probability mistakes are counting mistakes. With two dice there are 36 equally likely ordered outcomes, not 11 possible totals, so a total of 7 (six ways) is six times as likely as a total of 2 (one way). The complement rule often turns a hard count into an easy one. The chance that at least two people in a group of 23 share a birthday seems small, but it is easier to compute the chance that all 23 birthdays differ, which is about 49 per cent, so the chance of a shared birthday is about 51 per cent. This famous birthday paradox shows how badly intuition handles combinations.
Probability inside AI
- Classifiers output probabilities: a spam model might return 0.92 for spam and 0.08 for not spam, and the two must add to 1.
- Language models predict a probability for every possible next token, then sample from that distribution.
- Simulation: when exact calculation is hard, programs estimate probabilities by running thousands of random trials, a method called Monte Carlo.
- Python's random module and NumPy's np.random generate random numbers for such simulations; setting a seed makes results reproducible.
How it works
| Term | Meaning | Die example |
|---|---|---|
| Experiment | Something with an uncertain result | Rolling a die |
| Sample space (S) | All possible outcomes | {1, 2, 3, 4, 5, 6} |
| Event | A set of outcomes you care about | Rolling an even number |
The rules
- Every probability is between 0 and 1, and all outcomes together add to 1.
- Complement: the chance something does not happen is 1 minus the chance it does.
- Or: add the two chances, then subtract the overlap so it isn't counted twice.
- And: for independent events, multiply the two chances.
Counting outcomes
Use permutations when order matters, such as a PIN. Use combinations when it doesn't, such as choosing a team.
The maths
- Or: either event happens
- And: both events happen
Pick one card from a 52-card deck. What is the chance of a heart or a king?
P(heart) = 13/52 P(king) = 4/52 P(both) = 1/52 (the king of hearts) P(heart or king) = 13/52 + 4/52 − 1/52 = 16/52 ≈ 0.308
In code
Simulating 100,000 rolls gets very close to the exact answer of 0.5.
import math
import random
random.seed(1)
rolls = [random.randint(1, 6) for _ in range(100_000)]
even = sum(1 for roll in rolls if roll % 2 == 0)
print("Simulated P(even):", round(even / len(rolls), 3))
print("Ways to pick 2 people from 5, order ignored:", math.comb(5, 2))
print("Ways to arrange 2 people from 5, order matters:", math.perm(5, 2))Output
Simulated P(even): 0.501 Ways to pick 2 people from 5, order ignored: 10 Ways to arrange 2 people from 5, order matters: 20
Common mistakes
- Adding probabilities without subtracting the overlap.
- Multiplying probabilities of events that aren't independent.
- The gambler's fallacy: after 5 heads, the next toss of a fair coin is still 50/50.
Check yourself
Summary
- Probability runs from 0 to 1, and all outcomes add to 1.
- Or: add and subtract the overlap. And (independent): multiply.
- Order matters: permutation. Order doesn't: combination.
Finished reading?
Sources: Deisenroth, Faisal & Ong, Mathematics for Machine Learning