I3 · Mathematics for AI

Probability Distributions and the Normal Curve

A continuous distribution describes a variable that can take any value in a range, using a probability density function (PDF). The chance of landing in a range is the area under the curve there. The most important one is the normal (Gaussian) distribution, the symmetric bell curve set by its mean and standard deviation.

12 min Intermediate · Sublevel Beginner

What you need: The earlier topics of this module.

Definition

A continuous distribution describes a variable that can take any value in a range, using a probability density function (PDF). The chance of landing in a range is the area under the curve there. The most important one is the normal (Gaussian) distribution, the symmetric bell curve set by its mean and standard deviation.

In simple words

Measure the height of thousands of adults. Most are near the average, and fewer are very short or very tall. Draw it and you get a bell. Exact values have no chance of their own: nobody is exactly 170.000000 cm. So we ask for ranges, like "between 165 and 175".

Overview

  • A probability distribution describes how likely each value of a random variable is.
  • Discrete variables take separate values (counts); continuous variables take any value in a range (heights, times).
  • Continuous distributions use a probability density function; probabilities are areas under the curve.
  • The probability of any single exact value of a continuous variable is zero.
  • Each distribution is defined by parameters, such as the mean and standard deviation of a normal curve.
  • The central limit theorem explains why sums and averages are often normally distributed.

Why it matters

Weights are initialised from these distributions. Linear regression assumes normal errors, and squared error comes from that. VAEs and diffusion models work with Gaussian noise. Standardising features turns them towards the standard normal scale.

Key terms

Continuous variable
A variable that can take any value in a range, such as height or time.
Probability density function
A curve whose area over an interval gives the probability of landing in that interval.
Normal distribution
The symmetric bell curve described by its mean μ and standard deviation σ.
Uniform distribution
Every value in an interval is equally likely.
Cumulative distribution function
CDF: the probability that the variable is at most x.
Central limit theorem
Averages of many independent values tend towards a normal distribution, whatever the original shape.

Types

A few distributions appear again and again in data and AI. These are the most important types, both continuous and discrete.

1. Normal (Gaussian) distribution

The symmetric bell curve defined by a mean μ and standard deviation σ.

  • How it works: About 68% of values lie within μ ± σ, 95% within μ ± 2σ and 99.7% within μ ± 3σ.
  • Example: Adult heights in a population, measurement errors and the averages of many samples.
  • When to use it: Use it to model naturally varying measurements, compute z-scores and initialise neural-network weights.

2. Uniform distribution

Every value between a minimum a and maximum b is equally likely.

  • How it works: The density is flat at 1 ÷ (b − a); the mean is (a + b) ÷ 2.
  • Example: A random arrival time between 9:00 and 9:10 when buses run every 10 minutes.
  • When to use it: Use it when nothing is known except the range, for random sampling and for random search over hyperparameters.

3. Exponential distribution

The time until the next event when events happen at a constant average rate.

  • How it works: With rate λ the mean waiting time is 1 ÷ λ; it is memoryless: having waited does not change the future wait.
  • Example: Time between customer arrivals at a counter that averages 12 per hour, with mean 5 minutes.
  • When to use it: Use it for waiting times, time between failures and queueing models.

4. Binomial distribution (discrete)

The number of successes in n independent yes/no trials, each with probability p.

  • How it works: P(k) = C(n, k) pᵏ (1 − p)ⁿ⁻ᵏ; the mean is np and the variance np(1 − p).
  • Example: The number of heads in 10 coin tosses, or of 100 visitors who click an ad with p = 0.12.
  • When to use it: Use it for counts of successes such as clicks, passes and defects, and for A/B test calculations.

5. Poisson distribution (discrete)

The number of events in a fixed interval when they occur independently at a constant average rate λ.

  • How it works: P(k) = λᵏ e^(−λ) ÷ k!; the mean and the variance both equal λ.
  • Example: Calls to a helpline per hour, accidents per day on a highway stretch.
  • When to use it: Use it to model rare-event counts and to judge whether a count is unusually high.

6. Log-normal and skewed distributions

Distributions with a long right tail, where the logarithm of the values is roughly normal.

  • How it works: Values are always positive and multiplicative effects create the tail; taking logs makes them symmetric.
  • Example: Incomes, house prices, city sizes and order values in online shops.
  • When to use it: Use log transforms for such features before linear models, and report medians rather than means.

Explained in detail

Why a single value has zero probability

For a continuous variable, there are infinitely many possible values, so the probability of any exact value, such as a height of exactly 165.000… cm, is zero. Instead we ask about intervals: what is the probability of a height between 160 and 170 cm? A probability density function answers this through area: the probability is the area under the curve between 160 and 170. The total area under any density curve is 1. The height of the curve at a point is a density, not a probability, and can even exceed 1 for very narrow distributions.

The normal distribution

The normal or Gaussian distribution, named after the mathematician Carl Friedrich Gauss, who used it around 1809 to analyse errors in astronomical measurements, is defined by two numbers: the mean μ sets the centre and the standard deviation σ sets the width. About 68 per cent of values fall within μ ± σ, 95 per cent within μ ± 2σ and 99.7 per cent within μ ± 3σ. Many natural measurements are roughly normal: heights, measurement errors, blood pressure. Many others are not: incomes, earthquake sizes and city populations have long right tails, and assuming normality for them underestimates extreme events.

Other useful distributions

DistributionShapeWhere it appears
UniformFlat between a and bRandom numbers from random.random(); initial guesses
ExponentialStarts high, decaysTime between arrivals: calls, customers, failures
Log-normalRight-skewedIncomes, file sizes, time to finish tasks
NormalSymmetric bellMeasurement noise; weights in neural networks are often initialised from it

The central limit theorem

One reason the normal curve is everywhere is the central limit theorem: if you take the average of many independent random values, the distribution of that average becomes approximately normal, even if the individual values come from a skewed or flat distribution. Roll one die and every face is equally likely; average 30 dice many times and the averages form a bell curve around 3.5. This is why opinion polls can report a margin of error, and why averaged noise in measurements tends to look Gaussian.

  • In AI, many models assume Gaussian noise; linear regression with squared error is the best fit under that assumption.
  • Generative models such as diffusion models start from pure Gaussian noise and gradually turn it into an image.
  • NumPy draws samples with np.random.normal(mu, sigma, size) and np.random.uniform(a, b, size).

How it works

DistributionShape and use
UniformEvery value in a range is equally likely. Random initialisation.
NormalBell curve. Errors, measurements, and noise in diffusion models.
ExponentialWaiting time until an event, such as the next customer.
BetaValues between 0 and 1. Beliefs about a probability, and A/B tests.
Student's tLike the normal with heavier tails. Small samples.
Chi-squareSums of squared normals. Statistical tests.

The 68-95-99.7 rule

For a normal distribution, about 68% of values fall within 1 standard deviation of the mean, 95% within 2, and 99.7% within 3.

The maths

Normal distribution
The mean: the centre of the bell
The standard deviation: how wide the bell is
Fixed constants, about 2.718 and 3.142
Z-score (standardising)

Exam marks have mean 60 and standard deviation 10. You scored 80.

z = (80 − 60) / 10 = 2
You're 2 standard deviations above the mean.
About 95% of marks lie within ±2σ, so about 2.5% lie above you.

In code

normal.py
import numpy as np

rng = np.random.default_rng(0)
marks = rng.normal(loc=60, scale=10, size=100_000)

print("Mean:", round(marks.mean(), 1), "Std:", round(marks.std(), 1))
for k in (1, 2, 3):
    inside = np.mean(np.abs(marks - 60) < k * 10)
    print(f"Within {k} std: {inside:.1%}")
print("Scored above 80:", f"{np.mean(marks > 80):.1%}")

Output

Mean: 60.0 Std: 10.0
Within 1 std: 68.4%
Within 2 std: 95.5%
Within 3 std: 99.7%
Scored above 80: 2.3%

Common mistakes

  • Reading the height of the PDF as a probability. Only the area under it is. A narrow PDF can even rise above 1.
  • Assuming all data is normal. Incomes, for example, are heavily skewed. Plot your data first.

Check yourself

Question 1

1. About what share of values lie within 2 standard deviations of the mean in a normal distribution?

Question 2

2. Mean 50 and σ = 5. What is the z-score of 40?

Summary

  • Continuous variables use densities, and probabilities are areas.
  • The normal distribution is set by μ and σ.
  • A z-score says how many σ a value is from the mean.

Finished reading?

Sources: Deisenroth, Faisal & Ong, Mathematics for Machine Learning