I2 · Working with Data: NumPy, pandas, SQL and Visualisation

Statistics with NumPy

NumPy provides fast functions for descriptive statistics on arrays: np.mean, np.median, np.std, np.var, np.percentile, np.min, np.max, np.corrcoef and np.histogram, among others. Most accept an axis argument to compute per row or per column, and NaN-aware versions such as np.nanmean ignore missing values.

10 min Intermediate · Sublevel Beginner Jump to the activity

What you need: The earlier topics of this module.

Definition

You have a year of daily temperatures for ten cities as one array. Which city is hottest on average, which varies most, and on how many days did Delhi exceed 40 °C? Each answer is one NumPy line.

NumPy provides fast functions for descriptive statistics on arrays: np.mean, np.median, np.std, np.var, np.percentile, np.min, np.max, np.corrcoef and np.histogram, among others. Most accept an axis argument to compute per row or per column, and NaN-aware versions such as np.nanmean ignore missing values.

In simple words

The statistics you learned in the maths module, averages, spread, percentiles and correlation, become one-line commands. NumPy does the arithmetic for millions of numbers instantly, leaving you to choose the right question.

Overview

  • NumPy computes summary statistics of whole arrays or of each row or column.
  • Centre is measured with mean and median; spread with standard deviation, variance, range and percentiles.
  • np.std uses the population formula by default; ddof=1 gives the sample standard deviation.
  • np.nanmean and similar functions ignore missing values marked as NaN.
  • np.corrcoef measures how strongly two variables move together.
  • np.random.default_rng gives reproducible random samples for simulations.

Why it matters

Summarising data is the first step of every analysis and model. Doing it correctly, with the right axis, the right treatment of missing values and the right definition of standard deviation, avoids silent errors.

Key terms

Descriptive statistics
Numbers that summarise data: centre, spread and shape.
NaN
"Not a number", a special float used to mark missing or undefined values.
ddof
"Delta degrees of freedom": ddof=1 gives the sample standard deviation dividing by n − 1.
Histogram
Counts of values falling into equal-width bins.
Correlation matrix
A table of pairwise correlations between variables.
Reproducible randomness
Using a seeded generator, np.random.default_rng(seed), so results repeat.

Types

NumPy's statistics functions fall into five types, each answering a different question about the data.

1. Measures of centre

np.mean, np.median and np.average (weighted).

  • How it works: The mean adds and divides; the median sorts and takes the middle; np.average(x, weights=w) gives a weighted mean.
  • Example: Incomes [20, 25, 30, 200] thousand have mean 68.75 but median 27.5.
  • When to use it: Use the median for skewed data with outliers, and weighted averages for things like credit-weighted grades.

2. Measures of spread

np.std, np.var, np.ptp (range), np.min and np.max.

  • How it works: Variance is the mean squared distance from the mean; standard deviation is its square root, in the original units.
  • Example: Two classes both average 70, but one has a standard deviation of 5 and the other 20, so results differ much more in the second.
  • When to use it: Use them to judge consistency and before scaling features.

3. Position measures: percentiles and quantiles

np.percentile and np.quantile, which find the value below which a given share of data falls.

  • How it works: The 25th, 50th and 75th percentiles are the quartiles Q1, median and Q3; Q3 − Q1 is the interquartile range.
  • Example: np.percentile(marks, 90) gives the mark needed to be in the top 10%.
  • When to use it: Use them to describe distributions robustly and to detect outliers with the IQR rule.

4. Relationship measures

np.corrcoef and np.cov, which measure how two variables vary together.

  • How it works: Correlation runs from −1 (perfect opposite) through 0 (no linear link) to +1 (perfect together).
  • Example: np.corrcoef(hours_studied, marks)[0, 1] might be 0.72, a strong positive link.
  • When to use it: Use them to find related features and spot redundant ones; remember correlation is not causation.

5. Counting and random sampling

np.unique(return_counts=True), np.histogram, np.bincount and the random Generator methods.

  • How it works: Counting functions tally how often values occur; rng.choice, rng.normal and rng.integers draw random samples from a seeded generator.
  • Example: np.unique(grades, return_counts=True) counts how many A, B and C grades there are.
  • When to use it: Use them for frequency tables, histograms, simulations and bootstrapping.

Explained in detail

The essential functions

FunctionReturnsNote
np.mean, np.medianCentreMedian is robust to outliers
np.std, np.varSpreadDefault divides by n; use ddof=1 for a sample
np.percentile(x, [25, 50, 75])QuartilesUse np.quantile with 0.25 and so on for fractions
np.min, np.max, np.ptpExtremes and rangeptp means peak to peak: max − min
np.argmin, np.argmaxPositions of extremesWhich city or day was hottest
np.corrcoefCorrelation matrixRows are treated as variables by default
np.histogramBin counts and edgesThe numbers behind a histogram chart

Working along axes

Suppose temps has shape (365, 10): days by cities. temps.mean(axis=0) gives each city's average over the year, shape (10,), and temps.mean(axis=0).argmax() gives the hottest city's index. temps.std(axis=0) shows which city's temperature varies most. temps.mean(axis=1) gives the average across cities for each day, shape (365,), revealing the seasons. (temps[:, 0] > 40).sum() counts the days Delhi, if it is column 0, exceeded 40 °C. Getting the axis right is the most common challenge; say aloud which dimension you want to collapse before writing the code.

Missing values

Real sensor data has gaps. NumPy marks missing floats as np.nan, and any arithmetic with NaN produces NaN, so a single missing reading makes np.mean return nan for the whole column. This is deliberate: it stops missing data being ignored by accident. When you do want to skip gaps, use the NaN-aware versions np.nanmean, np.nanstd, np.nanmax and so on, and count the gaps with np.isnan(x).sum() so you know how much data is missing. Integer arrays cannot hold NaN, which is one reason data with gaps is usually stored as floats.

Random numbers and simulation

  • rng = np.random.default_rng(42) creates a seeded generator; the same seed always gives the same numbers.
  • rng.normal(mean, std, size), rng.integers(low, high, size) and rng.choice(options, size) cover most needs.
  • Simulations estimate probabilities: draw 100,000 samples and count how often an event happens.
  • Bootstrapping, resampling your data with replacement many times, estimates how uncertain a statistic such as the mean is.

Finally, remember that statistics describe the data you have, not necessarily the world. Temperatures from one weather station describe that station; averaging them for a whole state would mislead. Always ask where the numbers came from before generalising.

How it works

  1. Identify the axes: what do rows and columns represent?
  2. Check for missing values with np.isnan(x).sum().
  3. Compute centre, spread and percentiles along the right axis.
  4. Use argmax or argmin to find which row or column is extreme.
  5. Seed random generators so results can be reproduced.

In code

numpy_stats.py
import numpy as np

rng = np.random.default_rng(42)
days = np.arange(365)
season = 8 * np.sin(2 * np.pi * (days - 80) / 365)
cities = ["Delhi", "Mumbai", "Chennai"]
base = np.array([25, 27, 29])
spread = np.array([1.0, 0.4, 0.5])
temps = base + season[:, None] * spread + rng.normal(0, 1.5, (365, 3))
temps[100, 1] = np.nan  # one missing reading in Mumbai

print("Missing values:", np.isnan(temps).sum())
print("Mean per city (nan-aware):", np.nanmean(temps, axis=0).round(1))
print("Std per city:", np.nanstd(temps, axis=0, ddof=1).round(2))
print("Most variable city:", cities[int(np.nanstd(temps, axis=0).argmax())])
print("Delhi quartiles:", np.percentile(temps[:, 0], [25, 50, 75]).round(1))
print("Days Delhi above 32:", int((temps[:, 0] > 32).sum()))
counts, edges = np.histogram(temps[:, 0], bins=5)
print("Delhi histogram counts:", counts)

Output

Missing values: 1
Mean per city (nan-aware): [25.  26.9 28.9]
Std per city: [5.78 2.68 3.2 ]
Most variable city: Delhi
Delhi quartiles: [19.5 25.1 30.4]
Days Delhi above 32: 52
Delhi histogram counts: [55 92 63 84 71]

Try it yourself

The missing reading

An analyst computes the average temperature for each city.

Match the question to the function

Which NumPy call answers it?

  • Which day was hottest?
  • What is the middle value, robust to outliers?
  • How spread out are the values?
  • Average, ignoring missing values

Common mistakes

  • Computing along the wrong axis.
  • Ignoring NaN, or hiding how much data was missing.
  • Mixing population and sample standard deviations.

Check yourself

Question 1

1. What does np.mean return if the array contains NaN?

Question 2

2. temps has shape (365, 10). What shape is temps.mean(axis=1)?

Question 3

3. How do you make random results reproducible?

Summary

  • NumPy computes centre, spread, percentiles, extremes and correlations in single calls.
  • Choose the axis carefully and handle NaN deliberately.
  • Seed random generators for reproducible simulations.

Finished reading?

Sources: NumPy: the absolute basics for beginners · Python for Data Analysis, 3rd edition (Wes McKinney)