I2 · Working with Data: NumPy, pandas, SQL and Visualisation

Exploratory Data Analysis Workflow

Exploratory data analysis (EDA) is the process of examining a data set to understand its structure, quality and patterns before formal modelling, an approach championed by John Tukey in his 1977 book. A typical EDA workflow checks the size and types of the data, missing values and duplicates, summary statistics, the distribution of each variable, relationships between variables, differences between groups and outliers, and ends with a list of findings, data problems and questions to investigate. EDA combines summary tables with charts, because each reveals things the other hides.

20 min Intermediate · Sublevel Beginner Jump to the activity

What you need: The earlier topics of this module.

Definition

In 1973 the statistician Francis Anscombe published four small data sets with the same average, the same spread, the same correlation and the same regression line. Plotted, they look completely different. That is why every data project begins with exploring the data, not with a model.

Exploratory data analysis (EDA) is the process of examining a data set to understand its structure, quality and patterns before formal modelling, an approach championed by John Tukey in his 1977 book. A typical EDA workflow checks the size and types of the data, missing values and duplicates, summary statistics, the distribution of each variable, relationships between variables, differences between groups and outliers, and ends with a list of findings, data problems and questions to investigate. EDA combines summary tables with charts, because each reveals things the other hides.

In simple words

Before cooking with a new ingredient, you look at it, smell it and taste a little. EDA is getting to know your data in the same way before you trust it with a model.

Overview

  • Exploratory data analysis (EDA), named by John Tukey in 1977, means investigating data to understand it before modelling.
  • It combines summary statistics with charts and checks of data quality.
  • A good EDA starts from questions and ends with findings, problems and next steps written down.
  • Anscombe's quartet (1973) shows four data sets with the same means and correlations but very different shapes.
  • EDA guides cleaning, feature engineering and the choice of model.
  • Keep the test set aside during EDA if the data will train a model, to avoid biasing decisions.

Why it matters

EDA catches errors, reveals patterns that suggest good features, shows whether the data can answer your question at all and prevents misleading conclusions. Skipping it is one of the most common causes of failed analyses.

Key terms

Summary statistics
Count, mean, median, standard deviation, minimum, maximum and quartiles.
Univariate analysis
Looking at one variable at a time, such as its histogram.
Bivariate analysis
Looking at relationships between two variables, such as a scatter plot.
Skewness
When a distribution has a long tail on one side, such as incomes.
Anscombe's quartet
Four data sets with identical statistics but very different shapes.
Data profile
A written summary of a data set's size, types, quality issues and key patterns.

Types

EDA uses analyses of increasing depth. These types are usually done in this order.

1. Data profiling

Describing the data set's structure and quality.

  • How it works: Record rows, columns, types, missing values, duplicates and unique counts with info(), isna().sum() and nunique().
  • Example: A 50,000-row sales file with 3% missing pin codes and 120 duplicate invoices.
  • When to use it: Do it first; it decides what cleaning is needed. Save the profile so that later data can be compared with it.

2. Univariate analysis

Studying each variable on its own.

  • How it works: Numeric columns get describe(), histograms and box plots; categorical columns get value_counts() and bar charts.
  • Example: Order values are right-skewed with a long tail of large orders.
  • When to use it: Use it to find skew, impossible values and rare categories. Note every surprise as a question to follow up in the next steps.

3. Bivariate analysis

Studying pairs of variables, especially each feature against the target.

  • How it works: Use scatter plots and correlations for two numbers, grouped box plots for a number by category, and crosstabs for two categories.
  • Example: Delivery time rises with distance, and rainy days add about 8 minutes on average.
  • When to use it: Use it to find which features are likely to help predict the target.

4. Multivariate analysis

Studying three or more variables together.

  • How it works: Correlation heat maps, pair plots, colour-coded scatter plots and PCA reveal interactions and redundant features.
  • Example: The effect of discounts on sales is strong in festival months but weak otherwise.
  • When to use it: Use it to discover interactions and correlated features before modelling.

5. Hypothesis and reporting

Turning observations into questions to test and a short written summary.

  • How it works: List findings with evidence (a chart or number), note data problems and limitations, and suggest the next analyses or features.
  • Example: "Weekend orders are 30% larger; test whether a weekend feature improves the model."
  • When to use it: Use it to end every EDA, so that others can act on what you found.

Explained in detail

Find out how a row entered the table

A library kiosk records returns when someone scans a book. Ask who owns the log, which branches use the kiosk, what time zone it uses and whether the export contains a row per scan or per book. If a student scans twice, two rows can describe one return. If another returns a book at the desk, the kiosk can miss it entirely. A neat chart of scans cannot establish which branch has the most readers.

Keep the raw export and write a short data dictionary. Give each column its meaning, unit, allowed values and missing-value convention. Record the collection dates, permission to reuse the data, and the filters you applied. A classmate should be able to repeat the summary and point to the people or events it leaves out.

A step-by-step workflow

Steppandas toolsWhat to look for
1. Shape and typesdf.shape, df.dtypes, df.head()Expected rows and columns; numbers stored as text
2. Qualitydf.isna().sum(), df.duplicated().sum()Missing values, duplicates, impossible values
3. Summary statisticsdf.describe(), value_counts()Ranges, typical values, rare categories
4. DistributionsHistograms, box plotsSkew, multiple peaks, outliers
5. RelationshipsScatter plots, df.corr(), groupbyTrends, clusters, differences between groups
6. Write upA short data profileFindings, problems to fix and questions to test

Why charts and numbers are both needed

Anscombe's quartet makes the point vividly. All four sets have a mean x of 9, a mean y of about 7.5, a correlation of about 0.82 and the same regression line, y ≈ 3 + 0.5x. Yet one is a reasonable straight-line cloud, one is a smooth curve, one is a perfect line spoiled by a single outlier, and one has all points at the same x except one extreme point that creates the whole correlation. Summary statistics alone would treat them as identical; a model fitted without plotting could be badly wrong for three of the four. In real data, plots reveal problems such as a value of 999 used for "unknown", a sudden jump when a measuring method changed, or two groups mixed together that should be analysed separately.

Good EDA habits

  • Start with questions, but stay open to surprises the data shows you.
  • Look at both summary tables and charts for every important variable.
  • Check groups separately; patterns in the whole can reverse within groups, a trap known as Simpson's paradox.
  • Keep a log of findings and the code that produced them, so the analysis is reproducible.
  • Separate exploration from confirmation: patterns found by exploring should be tested on new data.

Libraries can speed up the first pass. Tools such as ydata-profiling generate a report of every column's type, missing values, distribution and correlations with one command. They are a useful starting point, but they do not know your question or your domain, so they cannot replace thinking about what the numbers mean, which groups matter and which patterns are surprising. Treat automated reports as a checklist of things to look at more closely, then follow up with your own targeted charts and summaries.

How it works

  1. Check shape, types and the first rows.
  2. Count missing values and duplicates, and look for impossible values.
  3. Compute summary statistics and category counts.
  4. Plot distributions and relationships, and compare groups.
  5. Write a short data profile with findings, problems and questions.

In code

anscombe_eda.py
import matplotlib.pyplot as plt
import numpy as np

x = [10, 8, 13, 9, 11, 14, 6, 4, 12, 7, 5]
sets = {
    "I":   (x, [8.04, 6.95, 7.58, 8.81, 8.33, 9.96, 7.24, 4.26, 10.84, 4.82, 5.68]),
    "II":  (x, [9.14, 8.14, 8.74, 8.77, 9.26, 8.10, 6.13, 3.10, 9.13, 7.26, 4.74]),
    "III": (x, [7.46, 6.77, 12.74, 7.11, 7.81, 8.84, 6.08, 5.39, 8.15, 6.42, 5.73]),
    "IV":  ([8, 8, 8, 8, 8, 8, 8, 19, 8, 8, 8], [6.58, 5.76, 7.71, 8.84, 8.47, 7.04, 5.25, 12.50, 5.56, 7.91, 6.89]),
}

fig, axes = plt.subplots(1, 4, figsize=(15, 3.4), sharey=True)
print(f"{'set':<4}{'mean x':>8}{'mean y':>8}{'sd y':>7}{'corr':>7}   regression line")
for ax, (name, (xs, ys)) in zip(axes, sets.items()):
    xs, ys = np.array(xs, float), np.array(ys)
    slope, intercept = np.polyfit(xs, ys, 1)
    r = np.corrcoef(xs, ys)[0, 1]
    print(f"{name:<4}{xs.mean():>8.2f}{ys.mean():>8.2f}{ys.std(ddof=1):>7.2f}{r:>7.2f}   y = {intercept:.2f} + {slope:.3f}x")
    ax.scatter(xs, ys)
    line_x = np.array([3, 20])
    ax.plot(line_x, intercept + slope * line_x, color="crimson")
    ax.set_title(f"Set {name}")
    ax.set_xlabel("x")
axes[0].set_ylabel("y")
fig.suptitle("Same statistics, different stories: always plot your data")
fig.tight_layout()

Output

set   mean x  mean y   sd y   corr   regression line
I       9.00    7.50   2.03   0.82   y = 3.00 + 0.500x
II      9.00    7.50   2.03   0.82   y = 3.00 + 0.500x
III     9.00    7.50   2.03   0.82   y = 3.00 + 0.500x
IV      9.00    7.50   2.03   0.82   y = 3.00 + 0.500x

Try it yourself

The EDA workflow

Put the steps in order.

  1. 1Check missing values, duplicates and impossible values
  2. 2Compute summary statistics
  3. 3Plot relationships and compare groups
  4. 4Write a data profile with findings and questions
  5. 5Check shape, types and first rows
  6. 6Plot distributions of each variable

The scheme that seemed to fail

A state education department explores data on a reading programme.

Common mistakes

  • Jumping to modelling without plotting the data.
  • Trusting summary statistics alone.
  • Ignoring differences between groups within the data.

Check yourself

Question 1

1. What does Anscombe's quartet show?

Question 2

2. Which pandas call counts missing values in each column?

Question 3

3. A pattern in the whole data reverses within every group. This is…

Summary

  • EDA examines size, types, quality, distributions, relationships and groups before modelling.
  • Combine summary statistics with plots; Anscombe's quartet shows why both are needed.
  • Check groups separately, log findings and test explored patterns on new data.

Read it. Now try it yourself.

Open the interactive activity

Finished reading?

Sources: Gebru et al., Datasheets for Datasets · John W. Tukey, Exploratory Data Analysis (1977) · pandas: chart visualization