I2 · Working with Data: NumPy, pandas, SQL and Visualisation

Exploring a Dataset

Exploring a data set, often called exploratory data analysis (EDA), means systematically getting to know it before modelling: its size and structure, the types and meaning of columns, missing values, distributions, unusual values, relationships between columns and how the target is distributed. In pandas this uses methods such as info, describe, value_counts, isna, nunique, corr and groupby.

10 min Intermediate · Sublevel Beginner Jump to the activity

What you need: The earlier topics of this module.

Definition

Before you train any model on a new data set, ten minutes of exploring can reveal impossible ages, a column that is 80 per cent empty and a label that leaks the answer. What should those ten minutes look like?

Exploring a data set, often called exploratory data analysis (EDA), means systematically getting to know it before modelling: its size and structure, the types and meaning of columns, missing values, distributions, unusual values, relationships between columns and how the target is distributed. In pandas this uses methods such as info, describe, value_counts, isna, nunique, corr and groupby.

In simple words

It is like a doctor's first check-up before treatment: height, weight, temperature, a few questions. Nothing dramatic, but it catches problems early and guides everything that follows.

Overview

  • Exploring a data set means understanding its size, columns, types, values and problems before analysing it.
  • shape, head(), tail() and sample() show size and example rows.
  • info() lists columns, types and non-missing counts; describe() summarises numeric columns.
  • value_counts() and nunique() describe categorical columns.
  • isna().sum() counts missing values per column.
  • Write down questions and suspicions as you explore; they guide the cleaning and analysis.

Why it matters

Most modelling failures trace back to data problems that a careful look would have revealed. EDA also generates ideas: which features matter, which need transforming, and which groups behave differently.

Key terms

Exploratory data analysis (EDA)
Investigating data with summaries and charts before modelling.
value_counts
Counts how often each value appears in a column.
Cardinality
The number of distinct values in a column, found with nunique().
Distribution
How a column's values are spread: centre, spread, skew and outliers.
Class balance
How evenly the target classes are represented.
Data dictionary
A document describing what each column means, its units and allowed values.

Types

Exploration combines several kinds of check. Each type of check answers a different question about the data.

1. Structural checks

Looking at the size and layout: rows, columns, names and types.

  • How it works: df.shape, df.columns, df.dtypes and df.info() show whether the file loaded as expected and whether numbers were read as text.
  • Example: A price column showing dtype object reveals values like "₹1,200" that need cleaning.
  • When to use it: Do this first, every time a new file is loaded.

2. Univariate summaries

Describing one column at a time.

  • How it works: describe() gives count, mean, std, min, quartiles and max; value_counts() gives category frequencies; histograms show shape.
  • Example: df["age"].describe() shows a minimum of -1, an obvious data-entry error.
  • When to use it: Use it to spot impossible values, skew and rare categories. It is also the moment to note which columns are skewed and may need a log transform later.

3. Data-quality checks

Counting missing values, duplicates and inconsistent labels.

  • How it works: isna().sum(), duplicated().sum() and unique() reveal gaps, repeated rows and spellings such as "Delhi" versus "delhi".
  • Example: Finding 312 duplicate orders and 5% missing pin codes.
  • When to use it: Use it to plan cleaning before trusting any number. Record every problem you find in a short list, because it becomes your cleaning plan and part of the project documentation.

4. Bivariate and group comparisons

Looking at how two columns relate, or how a value differs across groups.

  • How it works: df.corr() for numeric pairs, pd.crosstab for two categories, and df.groupby("group")["value"].mean() for comparisons.
  • Example: Average delivery time by city shows Bengaluru 12 minutes slower than Pune.
  • When to use it: Use it to form hypotheses about what drives the target.

5. Visual checks

Quick charts that reveal patterns numbers hide.

  • How it works: df.hist(), df.plot.box() and scatter plots show distributions, outliers and relationships at a glance.
  • Example: A histogram of order values shows two peaks: small snacks and large family orders.
  • When to use it: Use it alongside summaries, as Anscombe's quartet shows identical statistics can hide very different data.

Explained in detail

A ten-minute checklist

QuestionpandasWhat to look for
How big is it?df.shapeDoes it match the source? Enough rows for the task?
What are the columns?df.info(), df.head()Names, types, obviously wrong types
What is missing?df.isna().mean()Share of blanks per column; columns mostly empty
What do numbers look like?df.describe()Impossible minimums or maximums, huge skew
What do categories look like?df["col"].value_counts(), nunique()Typos ("Delhi" vs "delhi "), rare categories, ID-like columns
Are there duplicates?df.duplicated().sum()Repeated records that would leak into the test set
How is the target spread?df["target"].value_counts(normalize=True)Class imbalance that makes accuracy misleading
What relates to the target?df.groupby("col")["target"].mean(), df.corr(numeric_only=True)Strong signals, and suspiciously perfect ones that may be leakage

Univariate, bivariate and by group

Start with one column at a time: its distribution, summary and oddities. Then look at pairs: how does each feature relate to the target? For a numeric feature and a numeric target, a correlation and a scatter plot; for a category and a yes/no target, the rate of yes in each category with groupby and mean. Finally, compare groups: does the pattern hold for every city, age band or season? Differences between groups can reveal both useful features and fairness problems before any model is built.

Questions, not just numbers

  • Write down every surprise: "Why are 3 per cent of ages above 100?" "Why does one branch have no returns?"
  • Check meaning with the data owner; a code of 999 might mean "unknown", not a real value.
  • Build or read a data dictionary, so every column's units and meaning are clear.
  • Look out for leakage: a column recorded after the outcome, such as "refund_issued" when predicting complaints.
  • Keep EDA in a notebook with notes, so others can follow what you found and why you made each cleaning decision.

John Tukey, who coined the term exploratory data analysis in his 1977 book, compared it to detective work: looking for clues before confirming anything. Charts, covered in the visualisation chapter, are the detective's best tool, but even the summary tables on this page reveal most problems.

Charts make most of these checks faster: a histogram shows skew, a bar chart of counts shows typos and a box plot shows outliers.

How it works

  1. Check shape, columns and types.
  2. Measure missing values and duplicates.
  3. Summarise numeric columns and count categories.
  4. Look at the target's balance and its relationship with each feature.
  5. Write down surprises and questions, and resolve them before modelling.

In code

explore.py
import numpy as np
import pandas as pd

rng = np.random.default_rng(5)
n = 300
df = pd.DataFrame({
    "age": rng.integers(18, 70, n).astype(float),
    "city": rng.choice(["Pune", "Delhi", "Chennai", "delhi "], n, p=[0.4, 0.35, 0.2, 0.05]),
    "monthly_spend": rng.lognormal(8, 0.6, n).round(),
    "churned": rng.choice([0, 1], n, p=[0.85, 0.15]),
})
df.loc[rng.choice(n, 20, replace=False), "age"] = np.nan
df.loc[3, "age"] = 150
df = pd.concat([df, df.iloc[:5]])  # a few duplicate rows

print("Shape:", df.shape, " Duplicates:", int(df.duplicated().sum()))
print("Missing share:\n", df.isna().mean().round(3))
print(df[["age", "monthly_spend"]].describe().round(0))
print(df["city"].value_counts())
print("Churn rate:", df["churned"].mean().round(3))
print("Churn by city:\n", df.groupby("city")["churned"].mean().round(2))

Output

Shape: (305, 4)  Duplicates: 5
Missing share:
 age              0.066
city             0.000
monthly_spend    0.000
churned          0.000
dtype: float64
         age  monthly_spend
count  285.0          305.0
mean    44.0         3671.0
std     17.0         2394.0
min     18.0          416.0
25%     31.0         1942.0
50%     42.0         3189.0
75%     57.0         4682.0
max    150.0        19286.0
city
Pune       135
Delhi      103
Chennai     52
delhi       15
Name: count, dtype: int64
Churn rate: 0.167
Churn by city:
 city
Chennai    0.17
Delhi      0.17
Pune       0.14
delhi      0.33
Name: churned, dtype: float64

Try it yourself

What ten minutes revealed

An analyst receives a customer data set and is asked to build a churn model by Friday.

Which check finds it?

Match each problem to the check that reveals it.

  • A negative salary

  • A rare category with only two rows

  • "Mumbai" and "Bombay" as separate values

  • A column that is 70 per cent blank

Common mistakes

  • Jumping to modelling without exploring.
  • Ignoring spelling variants in categories.
  • Missing leakage hidden in a too-good feature.

Check yourself

Question 1

1. Which shows the share of missing values per column?

Question 2

2. "Delhi" and "delhi " appear as separate categories. You should…

Question 3

3. A feature predicts the target almost perfectly. What first?

Summary

  • EDA gets to know the data before modelling.
  • Check size, types, missing values, duplicates, distributions, categories and target balance.
  • Write down surprises, look for leakage and resolve questions with the data owner.

Finished reading?

Sources: Python for Data Analysis, 3rd edition (Wes McKinney) · Open Government Data Platform India