Definition
A coaching centre claims its new study method raises test scores. Twenty-five students try it and their average rises by 4 marks. Is that real improvement, or could it happen by luck? Hypothesis testing gives a careful way to answer.
Statistical inference draws conclusions about a population from a sample. A confidence interval gives a range of plausible values for a population quantity, such as a mean. A hypothesis test starts with a null hypothesis (H₀), usually "no effect" or "no difference", and asks how surprising the observed data would be if H₀ were true. That surprise is measured by the p-value: the probability of results at least as extreme as those seen, assuming H₀. If the p-value is below a chosen significance level α, often 0.05, we reject H₀. An A/B test applies this to compare two versions, such as two web pages.
In simple words
A friend says their coin is fair. You toss it 10 times and get 10 heads. If the coin really were fair, that would happen only about once in a thousand tries, so you stop believing the friend. Hypothesis testing does the same reasoning with numbers.
Overview
- A sample is a subset of a population used to learn about the whole.
- How a sample is chosen matters more than its size: a biased sample gives wrong answers however large it is.
- A hypothesis test compares data with a null hypothesis of no effect and gives a p-value.
- The significance level α, often 0.05, is the false-alarm rate we accept.
- Confidence intervals and effect sizes are more informative than a p-value alone.
- A/B tests apply these ideas to compare two versions of a product.
Why it matters
AI work is full of comparisons: is model B really better than model A, did the new recommendation layout increase sales, is a drug effective? Without inference, random noise in small samples is easily mistaken for real effects, leading to bad decisions.
Key terms
- Population and sample
- The whole group of interest, and the part of it actually measured.
- Standard error
- The typical size of the error in a sample mean, s ÷ √n.
- Confidence interval
- A range built so that, over many samples, 95 per cent of such ranges contain the true value.
- Null hypothesis (H₀)
- The default claim of no effect or no difference that the test tries to reject.
- p-value
- The chance of data at least this extreme if H₀ were true; it is not the chance that H₀ is true.
- Type I and Type II errors
- A false alarm (rejecting a true H₀) and a miss (failing to reject a false H₀).
Types
Two families of types matter here: ways of sampling, and kinds of hypothesis test. These are the main ones.
1. Simple random sampling
Every member of the population has the same chance of being chosen.
- How it works: Number the population and pick members with a random-number generator, such as rng.choice(ids, 500, replace=False).
- Example: Choosing 500 students at random from a university's list of 20,000 for a survey.
- When to use it: Use it when you have a full list of the population and no important sub-groups to guarantee.
2. Stratified and cluster sampling
Stratified sampling draws randomly within each sub-group; cluster sampling picks whole groups at random.
- How it works: Stratified sampling keeps each group's share fixed (as stratify=y does in train_test_split); cluster sampling surveys every member of chosen villages or schools.
- Example: A national health survey that samples within every state, then selects whole villages within each.
- When to use it: Use stratified sampling to represent small groups fairly, and cluster sampling when a full list is unavailable or travel is costly.
3. Convenience and other biased samples
Samples chosen because they are easy to reach, or where people choose themselves.
- How it works: They skip random selection, so certain groups are over- or under-represented; voluntary online polls are a common example.
- Example: A phone survey during office hours that reaches mostly retired people and homemakers.
- When to use it: Recognise them and avoid drawing population-wide conclusions from them; much AI training data is a convenience sample.
4. t-tests
Tests that compare means: a one-sample t-test against a value, a two-sample test between groups, and a paired test for before-and-after data.
- How it works: The t statistic is the difference in means divided by its standard error; scipy.stats.ttest_ind and ttest_rel compute p-values.
- Example: Comparing average marks of students taught with and without a new method.
- When to use it: Use them for numeric outcomes in two groups; use ANOVA for three or more groups.
5. Tests for proportions and categories
The z-test for two proportions and the chi-square test of independence.
- How it works: The z-test compares rates such as click-through rates; the chi-square test compares observed counts in a table with the counts expected if variables were unrelated.
- Example: An A/B test of 12.0% versus 15.5% clicks gives p = 0.023; a chi-square test checks whether pass rates differ by school.
- When to use it: Use them for yes/no outcomes and counts, which are very common in product and survey data.
Explained in detail
Samples vary, and the standard error measures how much
If you measure the heights of 25 random students, the average will differ a little from the average of all students, and a different 25 would give a slightly different answer. The spread of these sample means is the standard error, s ÷ √n, where s is the sample standard deviation and n the sample size. Quadrupling the sample size halves the standard error. The central limit theorem says that for reasonably large samples the sample mean is roughly normally distributed, whatever the shape of the population, which is why the same tools work for test scores, delivery times and click rates. A 95 per cent confidence interval for a mean is approximately the sample mean plus or minus two standard errors.
The logic of a hypothesis test
Testing works like a court case. H₀ is "innocent": the new method makes no difference. The evidence is the data. We compute a test statistic, such as t = (x̄ − μ₀) ÷ (s ÷ √n), which counts how many standard errors the result is from what H₀ predicts, and convert it to a p-value. A small p-value means the data would be unusual if H₀ were true, so we reject H₀; a large p-value means the data are consistent with H₀, which is not the same as proving H₀. The significance level α is the false-alarm rate we accept: with α = 0.05, one in twenty tests of a true H₀ will wrongly reject it.
| H₀ is true | H₀ is false | |
|---|---|---|
| Reject H₀ | Type I error (false alarm), probability α | Correct: an effect is found (power) |
| Do not reject H₀ | Correct | Type II error (missed effect) |
Common traps
Statistical significance is not practical importance: with a million users, a 0.1 per cent change in clicks can have a tiny p-value but no business value, so always report the effect size and a confidence interval. Running many tests inflates false alarms: test 20 useless features at α = 0.05 and on average one will look significant by chance; corrections such as Bonferroni (divide α by the number of tests) guard against this. Stopping an A/B test as soon as the p-value dips below 0.05 also inflates false alarms, so fix the sample size in advance. Finally, a test only says whether a difference is likely to be real, not why it happened; randomised assignment is what lets you claim cause. In the code below, the A/B test with 120 versus 155 clicks per 1,000 visitors gives z = 2.27 and p = 0.023; the spread of 2,000 sample means (2.04) matches the formula σ ÷ √n (2.00); and 100 tests on pure noise produce 4 false alarms at 0.05 but none after the Bonferroni correction.
How it works
- State H₀ and the alternative, and choose α before looking at the data.
- Decide the sample size; for A/B tests, assign users at random.
- Compute the test statistic and p-value, for example with scipy.stats.
- Report the effect size with a confidence interval, not just the p-value.
- Correct for multiple tests and interpret in plain language.
The maths
- The mean claimed by the null hypothesis
The usual average score is μ₀ = 60. Twenty-five students using the new method average x̄ = 64 with standard deviation s = 10. The standard error is 10 ÷ √25 = 2, so t = (64 − 60) ÷ 2 = 2.0. With 24 degrees of freedom, the two-sided p-value is about 0.057, just above 0.05, so at α = 0.05 we cannot reject H₀. The 95 per cent confidence interval is 64 ± 2.064 × 2, about 59.9 to 68.1, which includes 60, telling the same story. The method may help, but this sample is too small to be sure; a study with 100 students would halve the standard error.
In code
import numpy as np
from scipy import stats
# The worked example from summary numbers
n, mean, sd, mu0 = 25, 64, 10, 60
se = sd / np.sqrt(n)
t = (mean - mu0) / se
p = 2 * stats.t.sf(abs(t), df=n - 1)
crit = stats.t.ppf(0.975, df=n - 1)
print(f"t = {t:.2f}, p = {p:.3f}, 95% CI = ({mean - crit * se:.1f}, {mean + crit * se:.1f})")
# An A/B test: clicks on two versions of a page, 1,000 visitors each
clicks_a, clicks_b, visitors = 120, 155, 1000
pa, pb = clicks_a / visitors, clicks_b / visitors
pool = (clicks_a + clicks_b) / (2 * visitors)
z = (pb - pa) / np.sqrt(pool * (1 - pool) * 2 / visitors)
print(f"A/B test: {pa:.1%} vs {pb:.1%}, z = {z:.2f}, p = {2 * stats.norm.sf(abs(z)):.3f}")
# The standard error in action: means of many samples of 25 from a skewed population
rng = np.random.default_rng(0)
population = rng.exponential(scale=10, size=100_000)
means = [rng.choice(population, 25).mean() for _ in range(2000)]
print(f"Spread of sample means {np.std(means):.2f} vs formula sigma/sqrt(n) = {population.std() / 5:.2f}")
# Multiple testing: 100 tests where nothing is going on
pvals = [stats.ttest_ind(rng.normal(0, 1, 30), rng.normal(0, 1, 30)).pvalue for _ in range(100)]
print("False alarms among 100 null tests at 0.05:", sum(p < 0.05 for p in pvals), "| after Bonferroni:", sum(p < 0.05 / 100 for p in pvals))Output
t = 2.00, p = 0.057, 95% CI = (59.9, 68.1) A/B test: 12.0% vs 15.5%, z = 2.27, p = 0.023 Spread of sample means 2.04 vs formula sigma/sqrt(n) = 2.00 False alarms among 100 null tests at 0.05: 4 | after Bonferroni: 0
Try it yourself
Did the new homepage work?
An online bookshop tests a new homepage design against the old one.
Match the term
What does each mean?
- Type I error
- Type II error
- Standard error
- Bonferroni correction
Common mistakes
- Reading the p-value as the probability that H₀ is true.
- Treating "not significant" as proof of no effect.
- Peeking at an A/B test and stopping the moment p < 0.05.
Check yourself
Summary
- Sample means vary; the standard error s ÷ √n measures by how much.
- A hypothesis test asks how surprising the data are under H₀; small p-values lead us to reject H₀.
- Report effect sizes and intervals, fix sample sizes, and correct for multiple tests.
Finished reading?
Sources: OpenIntro Statistics (free textbook) · SciPy: statistical functions (scipy.stats)