Definition
In a health survey, the income question is blank for 30 per cent of people. If you simply delete those rows, whose voices disappear from your analysis?
Missing values are entries with no recorded data, shown in pandas as NaN or None. Handling them means first measuring how much is missing and why, then choosing to drop rows or columns, fill (impute) values with a statistic such as the median or with a model, or keep the missingness as information with an indicator column.
In simple words
Blank answers on a form are not all the same. Some people skipped a question by accident; some skipped it on purpose; some were never asked. How you fill or ignore the blanks should depend on which it is.
Overview
- Missing values are empty cells where data was not recorded; pandas marks them as NaN, None or NaT for dates.
- isna() finds them; isna().sum() counts them per column; isna().mean() gives the share missing.
- Most machine-learning models in scikit-learn cannot accept NaN, so missing values must be handled.
- The right fix depends on why the data is missing, not just how much is missing.
- Fill values must be learned from the training data only, inside a pipeline, to avoid leakage.
- Sometimes the fact that a value is missing is itself useful information.
Why it matters
Most models cannot accept missing values, and careless handling can bias results by removing particular groups or inventing false patterns. Missing data is one of the most common and consequential problems in real projects.
Key terms
- NaN
- "Not a number": pandas' marker for a missing numeric value.
- Imputation
- Filling missing values with estimates such as the median or a prediction.
- MCAR
- Missing completely at random: missingness unrelated to anything.
- MAR
- Missing at random: missingness explained by other observed columns.
- MNAR
- Missing not at random: missingness depends on the missing value itself, such as high earners hiding income.
- Missing indicator
- A 0/1 column recording whether a value was missing.
Types
Statisticians classify missing data into three types by why it is missing, and there are several types of remedy. Both matter when choosing what to do.
1. Missing completely at random (MCAR)
Values are missing for reasons unrelated to any data, such as a random sensor glitch.
- How it works: Because the gaps are random, the remaining data is still a fair sample, so dropping rows biases nothing (it only loses data).
- Example: A few survey forms lost in the post.
- When to use it: Dropping or simple filling is acceptable here, though it is rarely certain that data is truly MCAR.
2. Missing at random (MAR)
Missingness depends on other recorded columns, but not on the missing value itself.
- How it works: Because other columns explain the gaps, filling with group statistics or a model using those columns works well.
- Example: Older respondents skip the online income question more often, and age is recorded.
- When to use it: Use group-wise filling, KNN imputation or iterative (model-based) imputation.
3. Missing not at random (MNAR)
Missingness depends on the missing value itself.
- How it works: No filling method can fully correct it from the data alone; it needs domain knowledge, extra data or an indicator column.
- Example: People with very high incomes leave the income field blank.
- When to use it: Add a was_missing flag, report the limitation and avoid pretending the gaps are random.
4. Deletion remedies
Removing rows (listwise deletion) or whole columns with too many gaps.
- How it works: dropna() removes rows with any NaN; dropna(subset=["marks"]) only checks chosen columns; drop columns that are, say, over 60% empty.
- Example: Dropping 30 rows out of 10,000 that lack the target value.
- When to use it: Use it when few rows are affected or the target is missing; avoid it when it would throw away much data.
5. Imputation (filling) remedies
Replacing missing values with estimates: mean, median, mode, a constant, a neighbour or a model prediction.
- How it works: fillna(df["age"].median()) for numbers, the mode for categories, ffill() for time series, and SimpleImputer, KNNImputer or IterativeImputer in scikit-learn pipelines.
- Example: Filling missing temperature readings with the previous hour's value.
- When to use it: Use it to keep rows; prefer the median for skewed data and add a missing-indicator feature when gaps may be informative.
Explained in detail
First measure, then ask why
Start with df.isna().sum() for counts and df.isna().mean() for shares per column. Then ask why the values are missing, because the right treatment depends on it. The statistician Donald Rubin classified missing data in 1976 into three types. If data is missing completely at random, such as a sensor failing at random moments, dropping or simple filling does little harm. If it is missing at random given other columns, such as older respondents skipping an online question more often, you can fill values using those other columns. If it is missing not at random, such as people with high incomes declining to state them, any simple fix will bias the result, and you must think carefully and report the limitation.
Options and their costs
| Approach | pandas / scikit-learn | Cost |
|---|---|---|
| Drop rows | df.dropna(subset=["income"]) | Loses data and may remove whole groups |
| Drop a column | df.drop(columns="x") | Sensible if mostly empty and unimportant |
| Fill with median or mode | df["x"].fillna(df["x"].median()), SimpleImputer | Shrinks variation and can hide differences |
| Fill within groups | groupby("city")["x"].transform(lambda s: s.fillna(s.median())) | Better when groups differ; still an estimate |
| Forward fill for time series | df["x"].ffill() | Assumes the last value still holds |
| Model-based imputation | KNNImputer, IterativeImputer | More accurate but more complex |
| Add an indicator | SimpleImputer(add_indicator=True) | Lets the model learn from the fact that data was missing |
Doing it without leakage
Compute fill values, such as the median, on the training data only, then apply them to validation and test data. If you compute the median on all the data before splitting, information from the test set leaks into training. Putting SimpleImputer inside a scikit-learn Pipeline handles this automatically, including during cross-validation. For categorical columns, filling with a new category such as "Unknown" is often better than the most frequent value, because it keeps the fact of missingness visible.
- Missing values can hide behind codes such as 0, -1, 999 or "NA"; convert them to NaN first, or they distort every statistic.
- Report how much data was missing and what you did; readers need this to judge your results.
- Tree-based models such as gradient boosting in some libraries can handle NaN directly, but the reasons for missingness still matter.
How it works
- Convert disguised codes such as 999 or "-" to NaN.
- Measure missingness per column and per group.
- Ask why values are missing.
- Choose drop, fill, model-based imputation or an indicator, fitted on training data only.
- Record and report what you did.
In code
import numpy as np
import pandas as pd
from sklearn.impute import SimpleImputer
df = pd.DataFrame({
"area": ["urban", "urban", "rural", "rural", "rural", "urban"],
"income": [42000, np.nan, 18000, np.nan, np.nan, 61000],
"age": [34, 51, 999, 29, 45, 38], # 999 is a code for "not given"
})
df["age"] = df["age"].replace(999, np.nan)
print("Missing share:\n", df.isna().mean().round(2))
print("Missing income by area:\n", df.groupby("area")["income"].apply(lambda s: s.isna().mean()))
print("Rows left after dropna:", len(df.dropna()), "of", len(df))
df["income_filled"] = df.groupby("area")["income"].transform(lambda s: s.fillna(s.median()))
print(df)
imp = SimpleImputer(strategy="median", add_indicator=True)
filled = imp.fit_transform(df[["income", "age"]])
print(pd.DataFrame(filled, columns=imp.get_feature_names_out()).astype(int))Output
Missing share:
area 0.00
income 0.50
age 0.17
dtype: float64
Missing income by area:
area
rural 0.666667
urban 0.333333
Name: income, dtype: float64
Rows left after dropna: 2 of 6
area income age income_filled
0 urban 42000.0 34.0 42000.0
1 urban NaN 51.0 51500.0
2 rural 18000.0 NaN 18000.0
3 rural NaN 29.0 18000.0
4 rural NaN 45.0 18000.0
5 urban 61000.0 38.0 61000.0
income age missingindicator_income missingindicator_age
0 42000 34 0 0
1 42000 51 1 0
2 18000 38 0 1
3 42000 29 1 0
4 42000 45 1 0
5 61000 38 0 0Try it yourself
The rural voices that vanished
A team analyses a health survey of 2,000 households, half urban and half rural.
Which type of missingness?
Classify each situation.
People with high debts avoid stating them
A lab machine randomly fails on some samples
Weekend readings missing because the office was closed
Older people skip an online question more often
Common mistakes
- Dropping rows without checking who is removed.
- Computing fill values on all data, leaking test information.
- Leaving disguised codes such as 999 in numeric columns.
Check yourself
Summary
- Measure missingness and ask why: completely at random, at random, or not at random.
- Drop, fill or model-impute deliberately, fitted on training data only.
- Check effects by group, consider an indicator and report what you did.
Read it. Now try it yourself.
Open the interactive activityFinished reading?
Sources: pandas: Working with missing data · Python for Data Analysis, 3rd edition (Wes McKinney)