I2 · Working with Data: NumPy, pandas, SQL and Visualisation

Missing Values

Missing values are entries with no recorded data, shown in pandas as NaN or None. Handling them means first measuring how much is missing and why, then choosing to drop rows or columns, fill (impute) values with a statistic such as the median or with a model, or keep the missingness as information with an indicator column.

10 min Intermediate · Sublevel Beginner Jump to the activity

What you need: The earlier topics of this module.

Definition

In a health survey, the income question is blank for 30 per cent of people. If you simply delete those rows, whose voices disappear from your analysis?

Missing values are entries with no recorded data, shown in pandas as NaN or None. Handling them means first measuring how much is missing and why, then choosing to drop rows or columns, fill (impute) values with a statistic such as the median or with a model, or keep the missingness as information with an indicator column.

In simple words

Blank answers on a form are not all the same. Some people skipped a question by accident; some skipped it on purpose; some were never asked. How you fill or ignore the blanks should depend on which it is.

Overview

  • Missing values are empty cells where data was not recorded; pandas marks them as NaN, None or NaT for dates.
  • isna() finds them; isna().sum() counts them per column; isna().mean() gives the share missing.
  • Most machine-learning models in scikit-learn cannot accept NaN, so missing values must be handled.
  • The right fix depends on why the data is missing, not just how much is missing.
  • Fill values must be learned from the training data only, inside a pipeline, to avoid leakage.
  • Sometimes the fact that a value is missing is itself useful information.

Why it matters

Most models cannot accept missing values, and careless handling can bias results by removing particular groups or inventing false patterns. Missing data is one of the most common and consequential problems in real projects.

Key terms

NaN
"Not a number": pandas' marker for a missing numeric value.
Imputation
Filling missing values with estimates such as the median or a prediction.
MCAR
Missing completely at random: missingness unrelated to anything.
MAR
Missing at random: missingness explained by other observed columns.
MNAR
Missing not at random: missingness depends on the missing value itself, such as high earners hiding income.
Missing indicator
A 0/1 column recording whether a value was missing.

Types

Statisticians classify missing data into three types by why it is missing, and there are several types of remedy. Both matter when choosing what to do.

1. Missing completely at random (MCAR)

Values are missing for reasons unrelated to any data, such as a random sensor glitch.

  • How it works: Because the gaps are random, the remaining data is still a fair sample, so dropping rows biases nothing (it only loses data).
  • Example: A few survey forms lost in the post.
  • When to use it: Dropping or simple filling is acceptable here, though it is rarely certain that data is truly MCAR.

2. Missing at random (MAR)

Missingness depends on other recorded columns, but not on the missing value itself.

  • How it works: Because other columns explain the gaps, filling with group statistics or a model using those columns works well.
  • Example: Older respondents skip the online income question more often, and age is recorded.
  • When to use it: Use group-wise filling, KNN imputation or iterative (model-based) imputation.

3. Missing not at random (MNAR)

Missingness depends on the missing value itself.

  • How it works: No filling method can fully correct it from the data alone; it needs domain knowledge, extra data or an indicator column.
  • Example: People with very high incomes leave the income field blank.
  • When to use it: Add a was_missing flag, report the limitation and avoid pretending the gaps are random.

4. Deletion remedies

Removing rows (listwise deletion) or whole columns with too many gaps.

  • How it works: dropna() removes rows with any NaN; dropna(subset=["marks"]) only checks chosen columns; drop columns that are, say, over 60% empty.
  • Example: Dropping 30 rows out of 10,000 that lack the target value.
  • When to use it: Use it when few rows are affected or the target is missing; avoid it when it would throw away much data.

5. Imputation (filling) remedies

Replacing missing values with estimates: mean, median, mode, a constant, a neighbour or a model prediction.

  • How it works: fillna(df["age"].median()) for numbers, the mode for categories, ffill() for time series, and SimpleImputer, KNNImputer or IterativeImputer in scikit-learn pipelines.
  • Example: Filling missing temperature readings with the previous hour's value.
  • When to use it: Use it to keep rows; prefer the median for skewed data and add a missing-indicator feature when gaps may be informative.

Explained in detail

First measure, then ask why

Start with df.isna().sum() for counts and df.isna().mean() for shares per column. Then ask why the values are missing, because the right treatment depends on it. The statistician Donald Rubin classified missing data in 1976 into three types. If data is missing completely at random, such as a sensor failing at random moments, dropping or simple filling does little harm. If it is missing at random given other columns, such as older respondents skipping an online question more often, you can fill values using those other columns. If it is missing not at random, such as people with high incomes declining to state them, any simple fix will bias the result, and you must think carefully and report the limitation.

Options and their costs

Approachpandas / scikit-learnCost
Drop rowsdf.dropna(subset=["income"])Loses data and may remove whole groups
Drop a columndf.drop(columns="x")Sensible if mostly empty and unimportant
Fill with median or modedf["x"].fillna(df["x"].median()), SimpleImputerShrinks variation and can hide differences
Fill within groupsgroupby("city")["x"].transform(lambda s: s.fillna(s.median()))Better when groups differ; still an estimate
Forward fill for time seriesdf["x"].ffill()Assumes the last value still holds
Model-based imputationKNNImputer, IterativeImputerMore accurate but more complex
Add an indicatorSimpleImputer(add_indicator=True)Lets the model learn from the fact that data was missing

Doing it without leakage

Compute fill values, such as the median, on the training data only, then apply them to validation and test data. If you compute the median on all the data before splitting, information from the test set leaks into training. Putting SimpleImputer inside a scikit-learn Pipeline handles this automatically, including during cross-validation. For categorical columns, filling with a new category such as "Unknown" is often better than the most frequent value, because it keeps the fact of missingness visible.

  • Missing values can hide behind codes such as 0, -1, 999 or "NA"; convert them to NaN first, or they distort every statistic.
  • Report how much data was missing and what you did; readers need this to judge your results.
  • Tree-based models such as gradient boosting in some libraries can handle NaN directly, but the reasons for missingness still matter.

How it works

  1. Convert disguised codes such as 999 or "-" to NaN.
  2. Measure missingness per column and per group.
  3. Ask why values are missing.
  4. Choose drop, fill, model-based imputation or an indicator, fitted on training data only.
  5. Record and report what you did.

In code

missing.py
import numpy as np
import pandas as pd
from sklearn.impute import SimpleImputer

df = pd.DataFrame({
    "area": ["urban", "urban", "rural", "rural", "rural", "urban"],
    "income": [42000, np.nan, 18000, np.nan, np.nan, 61000],
    "age": [34, 51, 999, 29, 45, 38],  # 999 is a code for "not given"
})
df["age"] = df["age"].replace(999, np.nan)
print("Missing share:\n", df.isna().mean().round(2))
print("Missing income by area:\n", df.groupby("area")["income"].apply(lambda s: s.isna().mean()))

print("Rows left after dropna:", len(df.dropna()), "of", len(df))
df["income_filled"] = df.groupby("area")["income"].transform(lambda s: s.fillna(s.median()))
print(df)

imp = SimpleImputer(strategy="median", add_indicator=True)
filled = imp.fit_transform(df[["income", "age"]])
print(pd.DataFrame(filled, columns=imp.get_feature_names_out()).astype(int))

Output

Missing share:
 area      0.00
income    0.50
age       0.17
dtype: float64
Missing income by area:
 area
rural    0.666667
urban    0.333333
Name: income, dtype: float64
Rows left after dropna: 2 of 6
    area   income   age  income_filled
0  urban  42000.0  34.0        42000.0
1  urban      NaN  51.0        51500.0
2  rural  18000.0   NaN        18000.0
3  rural      NaN  29.0        18000.0
4  rural      NaN  45.0        18000.0
5  urban  61000.0  38.0        61000.0
   income  age  missingindicator_income  missingindicator_age
0   42000   34                        0                     0
1   42000   51                        1                     0
2   18000   38                        0                     1
3   42000   29                        1                     0
4   42000   45                        1                     0
5   61000   38                        0                     0

Try it yourself

The rural voices that vanished

A team analyses a health survey of 2,000 households, half urban and half rural.

Which type of missingness?

Classify each situation.

  • People with high debts avoid stating them

  • A lab machine randomly fails on some samples

  • Weekend readings missing because the office was closed

  • Older people skip an online question more often

Common mistakes

  • Dropping rows without checking who is removed.
  • Computing fill values on all data, leaking test information.
  • Leaving disguised codes such as 999 in numeric columns.

Check yourself

Question 1

1. What should you do first with missing values?

Question 2

2. Why compute the median for filling on training data only?

Question 3

3. People with high incomes refuse to state them. This is…

Summary

  • Measure missingness and ask why: completely at random, at random, or not at random.
  • Drop, fill or model-impute deliberately, fitted on training data only.
  • Check effects by group, consider an indicator and report what you did.

Read it. Now try it yourself.

Open the interactive activity

Finished reading?

Sources: pandas: Working with missing data · Python for Data Analysis, 3rd edition (Wes McKinney)