Definition
Before you train any model on a new data set, ten minutes of exploring can reveal impossible ages, a column that is 80 per cent empty and a label that leaks the answer. What should those ten minutes look like?
Exploring a data set, often called exploratory data analysis (EDA), means systematically getting to know it before modelling: its size and structure, the types and meaning of columns, missing values, distributions, unusual values, relationships between columns and how the target is distributed. In pandas this uses methods such as info, describe, value_counts, isna, nunique, corr and groupby.
In simple words
It is like a doctor's first check-up before treatment: height, weight, temperature, a few questions. Nothing dramatic, but it catches problems early and guides everything that follows.
Overview
- Exploring a data set means understanding its size, columns, types, values and problems before analysing it.
- shape, head(), tail() and sample() show size and example rows.
- info() lists columns, types and non-missing counts; describe() summarises numeric columns.
- value_counts() and nunique() describe categorical columns.
- isna().sum() counts missing values per column.
- Write down questions and suspicions as you explore; they guide the cleaning and analysis.
Why it matters
Most modelling failures trace back to data problems that a careful look would have revealed. EDA also generates ideas: which features matter, which need transforming, and which groups behave differently.
Key terms
- Exploratory data analysis (EDA)
- Investigating data with summaries and charts before modelling.
- value_counts
- Counts how often each value appears in a column.
- Cardinality
- The number of distinct values in a column, found with nunique().
- Distribution
- How a column's values are spread: centre, spread, skew and outliers.
- Class balance
- How evenly the target classes are represented.
- Data dictionary
- A document describing what each column means, its units and allowed values.
Types
Exploration combines several kinds of check. Each type of check answers a different question about the data.
1. Structural checks
Looking at the size and layout: rows, columns, names and types.
- How it works: df.shape, df.columns, df.dtypes and df.info() show whether the file loaded as expected and whether numbers were read as text.
- Example: A price column showing dtype object reveals values like "₹1,200" that need cleaning.
- When to use it: Do this first, every time a new file is loaded.
2. Univariate summaries
Describing one column at a time.
- How it works: describe() gives count, mean, std, min, quartiles and max; value_counts() gives category frequencies; histograms show shape.
- Example: df["age"].describe() shows a minimum of -1, an obvious data-entry error.
- When to use it: Use it to spot impossible values, skew and rare categories. It is also the moment to note which columns are skewed and may need a log transform later.
3. Data-quality checks
Counting missing values, duplicates and inconsistent labels.
- How it works: isna().sum(), duplicated().sum() and unique() reveal gaps, repeated rows and spellings such as "Delhi" versus "delhi".
- Example: Finding 312 duplicate orders and 5% missing pin codes.
- When to use it: Use it to plan cleaning before trusting any number. Record every problem you find in a short list, because it becomes your cleaning plan and part of the project documentation.
4. Bivariate and group comparisons
Looking at how two columns relate, or how a value differs across groups.
- How it works: df.corr() for numeric pairs, pd.crosstab for two categories, and df.groupby("group")["value"].mean() for comparisons.
- Example: Average delivery time by city shows Bengaluru 12 minutes slower than Pune.
- When to use it: Use it to form hypotheses about what drives the target.
5. Visual checks
Quick charts that reveal patterns numbers hide.
- How it works: df.hist(), df.plot.box() and scatter plots show distributions, outliers and relationships at a glance.
- Example: A histogram of order values shows two peaks: small snacks and large family orders.
- When to use it: Use it alongside summaries, as Anscombe's quartet shows identical statistics can hide very different data.
Explained in detail
A ten-minute checklist
| Question | pandas | What to look for |
|---|---|---|
| How big is it? | df.shape | Does it match the source? Enough rows for the task? |
| What are the columns? | df.info(), df.head() | Names, types, obviously wrong types |
| What is missing? | df.isna().mean() | Share of blanks per column; columns mostly empty |
| What do numbers look like? | df.describe() | Impossible minimums or maximums, huge skew |
| What do categories look like? | df["col"].value_counts(), nunique() | Typos ("Delhi" vs "delhi "), rare categories, ID-like columns |
| Are there duplicates? | df.duplicated().sum() | Repeated records that would leak into the test set |
| How is the target spread? | df["target"].value_counts(normalize=True) | Class imbalance that makes accuracy misleading |
| What relates to the target? | df.groupby("col")["target"].mean(), df.corr(numeric_only=True) | Strong signals, and suspiciously perfect ones that may be leakage |
Univariate, bivariate and by group
Start with one column at a time: its distribution, summary and oddities. Then look at pairs: how does each feature relate to the target? For a numeric feature and a numeric target, a correlation and a scatter plot; for a category and a yes/no target, the rate of yes in each category with groupby and mean. Finally, compare groups: does the pattern hold for every city, age band or season? Differences between groups can reveal both useful features and fairness problems before any model is built.
Questions, not just numbers
- Write down every surprise: "Why are 3 per cent of ages above 100?" "Why does one branch have no returns?"
- Check meaning with the data owner; a code of 999 might mean "unknown", not a real value.
- Build or read a data dictionary, so every column's units and meaning are clear.
- Look out for leakage: a column recorded after the outcome, such as "refund_issued" when predicting complaints.
- Keep EDA in a notebook with notes, so others can follow what you found and why you made each cleaning decision.
John Tukey, who coined the term exploratory data analysis in his 1977 book, compared it to detective work: looking for clues before confirming anything. Charts, covered in the visualisation chapter, are the detective's best tool, but even the summary tables on this page reveal most problems.
Charts make most of these checks faster: a histogram shows skew, a bar chart of counts shows typos and a box plot shows outliers.
How it works
- Check shape, columns and types.
- Measure missing values and duplicates.
- Summarise numeric columns and count categories.
- Look at the target's balance and its relationship with each feature.
- Write down surprises and questions, and resolve them before modelling.
In code
import numpy as np
import pandas as pd
rng = np.random.default_rng(5)
n = 300
df = pd.DataFrame({
"age": rng.integers(18, 70, n).astype(float),
"city": rng.choice(["Pune", "Delhi", "Chennai", "delhi "], n, p=[0.4, 0.35, 0.2, 0.05]),
"monthly_spend": rng.lognormal(8, 0.6, n).round(),
"churned": rng.choice([0, 1], n, p=[0.85, 0.15]),
})
df.loc[rng.choice(n, 20, replace=False), "age"] = np.nan
df.loc[3, "age"] = 150
df = pd.concat([df, df.iloc[:5]]) # a few duplicate rows
print("Shape:", df.shape, " Duplicates:", int(df.duplicated().sum()))
print("Missing share:\n", df.isna().mean().round(3))
print(df[["age", "monthly_spend"]].describe().round(0))
print(df["city"].value_counts())
print("Churn rate:", df["churned"].mean().round(3))
print("Churn by city:\n", df.groupby("city")["churned"].mean().round(2))Output
Shape: (305, 4) Duplicates: 5
Missing share:
age 0.066
city 0.000
monthly_spend 0.000
churned 0.000
dtype: float64
age monthly_spend
count 285.0 305.0
mean 44.0 3671.0
std 17.0 2394.0
min 18.0 416.0
25% 31.0 1942.0
50% 42.0 3189.0
75% 57.0 4682.0
max 150.0 19286.0
city
Pune 135
Delhi 103
Chennai 52
delhi 15
Name: count, dtype: int64
Churn rate: 0.167
Churn by city:
city
Chennai 0.17
Delhi 0.17
Pune 0.14
delhi 0.33
Name: churned, dtype: float64Try it yourself
What ten minutes revealed
An analyst receives a customer data set and is asked to build a churn model by Friday.
Which check finds it?
Match each problem to the check that reveals it.
A negative salary
A rare category with only two rows
"Mumbai" and "Bombay" as separate values
A column that is 70 per cent blank
Common mistakes
- Jumping to modelling without exploring.
- Ignoring spelling variants in categories.
- Missing leakage hidden in a too-good feature.
Check yourself
Summary
- EDA gets to know the data before modelling.
- Check size, types, missing values, duplicates, distributions, categories and target balance.
- Write down surprises, look for leakage and resolve questions with the data owner.
Finished reading?
Sources: Python for Data Analysis, 3rd edition (Wes McKinney) · Open Government Data Platform India