Definition
A continuous distribution describes a variable that can take any value in a range, using a probability density function (PDF). The chance of landing in a range is the area under the curve there. The most important one is the normal (Gaussian) distribution, the symmetric bell curve set by its mean and standard deviation.
In simple words
Measure the height of thousands of adults. Most are near the average, and fewer are very short or very tall. Draw it and you get a bell. Exact values have no chance of their own: nobody is exactly 170.000000 cm. So we ask for ranges, like "between 165 and 175".
Overview
- A probability distribution describes how likely each value of a random variable is.
- Discrete variables take separate values (counts); continuous variables take any value in a range (heights, times).
- Continuous distributions use a probability density function; probabilities are areas under the curve.
- The probability of any single exact value of a continuous variable is zero.
- Each distribution is defined by parameters, such as the mean and standard deviation of a normal curve.
- The central limit theorem explains why sums and averages are often normally distributed.
Why it matters
Weights are initialised from these distributions. Linear regression assumes normal errors, and squared error comes from that. VAEs and diffusion models work with Gaussian noise. Standardising features turns them towards the standard normal scale.
Key terms
- Continuous variable
- A variable that can take any value in a range, such as height or time.
- Probability density function
- A curve whose area over an interval gives the probability of landing in that interval.
- Normal distribution
- The symmetric bell curve described by its mean μ and standard deviation σ.
- Uniform distribution
- Every value in an interval is equally likely.
- Cumulative distribution function
- CDF: the probability that the variable is at most x.
- Central limit theorem
- Averages of many independent values tend towards a normal distribution, whatever the original shape.
Types
A few distributions appear again and again in data and AI. These are the most important types, both continuous and discrete.
1. Normal (Gaussian) distribution
The symmetric bell curve defined by a mean μ and standard deviation σ.
- How it works: About 68% of values lie within μ ± σ, 95% within μ ± 2σ and 99.7% within μ ± 3σ.
- Example: Adult heights in a population, measurement errors and the averages of many samples.
- When to use it: Use it to model naturally varying measurements, compute z-scores and initialise neural-network weights.
2. Uniform distribution
Every value between a minimum a and maximum b is equally likely.
- How it works: The density is flat at 1 ÷ (b − a); the mean is (a + b) ÷ 2.
- Example: A random arrival time between 9:00 and 9:10 when buses run every 10 minutes.
- When to use it: Use it when nothing is known except the range, for random sampling and for random search over hyperparameters.
3. Exponential distribution
The time until the next event when events happen at a constant average rate.
- How it works: With rate λ the mean waiting time is 1 ÷ λ; it is memoryless: having waited does not change the future wait.
- Example: Time between customer arrivals at a counter that averages 12 per hour, with mean 5 minutes.
- When to use it: Use it for waiting times, time between failures and queueing models.
4. Binomial distribution (discrete)
The number of successes in n independent yes/no trials, each with probability p.
- How it works: P(k) = C(n, k) pᵏ (1 − p)ⁿ⁻ᵏ; the mean is np and the variance np(1 − p).
- Example: The number of heads in 10 coin tosses, or of 100 visitors who click an ad with p = 0.12.
- When to use it: Use it for counts of successes such as clicks, passes and defects, and for A/B test calculations.
5. Poisson distribution (discrete)
The number of events in a fixed interval when they occur independently at a constant average rate λ.
- How it works: P(k) = λᵏ e^(−λ) ÷ k!; the mean and the variance both equal λ.
- Example: Calls to a helpline per hour, accidents per day on a highway stretch.
- When to use it: Use it to model rare-event counts and to judge whether a count is unusually high.
6. Log-normal and skewed distributions
Distributions with a long right tail, where the logarithm of the values is roughly normal.
- How it works: Values are always positive and multiplicative effects create the tail; taking logs makes them symmetric.
- Example: Incomes, house prices, city sizes and order values in online shops.
- When to use it: Use log transforms for such features before linear models, and report medians rather than means.
Explained in detail
Why a single value has zero probability
For a continuous variable, there are infinitely many possible values, so the probability of any exact value, such as a height of exactly 165.000… cm, is zero. Instead we ask about intervals: what is the probability of a height between 160 and 170 cm? A probability density function answers this through area: the probability is the area under the curve between 160 and 170. The total area under any density curve is 1. The height of the curve at a point is a density, not a probability, and can even exceed 1 for very narrow distributions.
The normal distribution
The normal or Gaussian distribution, named after the mathematician Carl Friedrich Gauss, who used it around 1809 to analyse errors in astronomical measurements, is defined by two numbers: the mean μ sets the centre and the standard deviation σ sets the width. About 68 per cent of values fall within μ ± σ, 95 per cent within μ ± 2σ and 99.7 per cent within μ ± 3σ. Many natural measurements are roughly normal: heights, measurement errors, blood pressure. Many others are not: incomes, earthquake sizes and city populations have long right tails, and assuming normality for them underestimates extreme events.
Other useful distributions
| Distribution | Shape | Where it appears |
|---|---|---|
| Uniform | Flat between a and b | Random numbers from random.random(); initial guesses |
| Exponential | Starts high, decays | Time between arrivals: calls, customers, failures |
| Log-normal | Right-skewed | Incomes, file sizes, time to finish tasks |
| Normal | Symmetric bell | Measurement noise; weights in neural networks are often initialised from it |
The central limit theorem
One reason the normal curve is everywhere is the central limit theorem: if you take the average of many independent random values, the distribution of that average becomes approximately normal, even if the individual values come from a skewed or flat distribution. Roll one die and every face is equally likely; average 30 dice many times and the averages form a bell curve around 3.5. This is why opinion polls can report a margin of error, and why averaged noise in measurements tends to look Gaussian.
- In AI, many models assume Gaussian noise; linear regression with squared error is the best fit under that assumption.
- Generative models such as diffusion models start from pure Gaussian noise and gradually turn it into an image.
- NumPy draws samples with np.random.normal(mu, sigma, size) and np.random.uniform(a, b, size).
How it works
| Distribution | Shape and use |
|---|---|
| Uniform | Every value in a range is equally likely. Random initialisation. |
| Normal | Bell curve. Errors, measurements, and noise in diffusion models. |
| Exponential | Waiting time until an event, such as the next customer. |
| Beta | Values between 0 and 1. Beliefs about a probability, and A/B tests. |
| Student's t | Like the normal with heavier tails. Small samples. |
| Chi-square | Sums of squared normals. Statistical tests. |
The 68-95-99.7 rule
For a normal distribution, about 68% of values fall within 1 standard deviation of the mean, 95% within 2, and 99.7% within 3.
The maths
- The mean: the centre of the bell
- The standard deviation: how wide the bell is
- Fixed constants, about 2.718 and 3.142
Exam marks have mean 60 and standard deviation 10. You scored 80.
z = (80 − 60) / 10 = 2 You're 2 standard deviations above the mean. About 95% of marks lie within ±2σ, so about 2.5% lie above you.
In code
import numpy as np
rng = np.random.default_rng(0)
marks = rng.normal(loc=60, scale=10, size=100_000)
print("Mean:", round(marks.mean(), 1), "Std:", round(marks.std(), 1))
for k in (1, 2, 3):
inside = np.mean(np.abs(marks - 60) < k * 10)
print(f"Within {k} std: {inside:.1%}")
print("Scored above 80:", f"{np.mean(marks > 80):.1%}")Output
Mean: 60.0 Std: 10.0 Within 1 std: 68.4% Within 2 std: 95.5% Within 3 std: 99.7% Scored above 80: 2.3%
Common mistakes
- Reading the height of the PDF as a probability. Only the area under it is. A narrow PDF can even rise above 1.
- Assuming all data is normal. Incomes, for example, are heavily skewed. Plot your data first.
Check yourself
Summary
- Continuous variables use densities, and probabilities are areas.
- The normal distribution is set by μ and σ.
- A z-score says how many σ a value is from the mean.
Finished reading?
Sources: Deisenroth, Faisal & Ong, Mathematics for Machine Learning