B8 · Computational Thinking and the AI Project Cycle

Data Exploration

Data exploration is the stage of the AI project cycle where the team studies the collected data to understand it and check its quality. It involves summarising and visualising the data, and finding and fixing problems such as missing values, duplicates, errors, outliers and imbalance. Fixing these problems is called data cleaning.

10 min Beginner Jump to the activity

What you need: Nothing. This page starts from the beginning, with no programming and no formulas.

Definition

The canteen's sales register says 4,000 lunches were sold on one Tuesday. The school has 900 students. Before you train any model, something needs a closer look.

Data exploration is the stage of the AI project cycle where the team studies the collected data to understand it and check its quality. It involves summarising and visualising the data, and finding and fixing problems such as missing values, duplicates, errors, outliers and imbalance. Fixing these problems is called data cleaning.

In simple words

Before cooking rice, you spread it on a plate and pick out the small stones and husks. You do not just throw the whole bag into the pot. Data exploration is picking the stones out of your data before you feed it to a model.

Why it matters

A model trained on messy data learns the mess. Data scientists often say they spend most of their time cleaning and understanding data, because this step decides how good the final model can be.

Key terms

Data exploration
Studying data to understand it and check its quality before modelling.
Data cleaning
Fixing or removing errors, gaps and duplicates in data.
Missing value
A blank where data should be.
Duplicate
The same record appearing more than once.
Outlier
A value far from the rest, which may be an error or a real but rare event.
Imbalanced data
Data where some groups or outcomes have far more examples than others.

Explained in detail

First, get to know the data

Start by looking at the data as a whole. How many rows and columns are there? What does each column mean, and in what units? What are the smallest, largest and typical values in each column? Then draw a few charts: a line graph of daily sales over the term, a bar graph of average sales by day of the week. Patterns appear quickly: Mondays are busy, exam weeks are quiet. So do problems: a spike to 4,000 on one Tuesday stands out immediately.

Common problems and fixes

ProblemExamplePossible fix
Missing valuesNo sales recorded on three daysCheck whether the canteen was shut; if not, fill from nearby days or leave those days out
DuplicatesThe same day entered twiceRemove the extra copy
Errors4,000 instead of 400, a typing slipCorrect it from the original register, or remove it
Inconsistent formats"Mon", "Monday" and "monday" in one columnUse one standard spelling
Real outliersAnnual day, when parents also ateKeep, but mark it as a special day
ImbalanceOnly five exam days among a hundredCollect more examples, or treat the model's exam-day predictions with caution

Not every outlier is a mistake

It is tempting to delete every unusual value, but some are real and important. The annual day really did have many more diners. Deleting it would make the model unprepared for the next annual day. The rule is to investigate before deciding: find out why a value is unusual. If it is an error, fix or remove it. If it is real, keep it and, if possible, add a feature that explains it, such as "special event: yes".

Checking for fairness

Exploration is also when you check whether every group is properly represented. For a crop disease app, are there photos from every region, every crop variety, every season, every kind of phone camera? For a voice assistant, are there recordings of children, older people and every major accent? A simple count of examples per group often reveals gaps that would otherwise become unfairness in the finished model. Finding them now is far cheaper than discovering them after the app is used by millions.

How it works

  1. Count rows and columns and read what each column means.
  2. Summarise each column: smallest, largest, average, number of blanks.
  3. Draw charts to spot patterns and odd values.
  4. Investigate odd values, then fix errors, remove duplicates and handle blanks.
  5. Count examples per group to check balance and fairness.

Try it yourself

The Tuesday with 4,000 lunches

The team explores a term's worth of canteen sales.

Match the problem to the fix

How would you handle each data problem?

  • The same record appears twice
  • A typing slip adds an extra zero
  • A real but unusual event day
  • "Mon" and "Monday" in one column

Common mistakes

  • Training a model without looking at the data first.
  • Deleting every unusual value without investigating.
  • Not checking whether every group has enough examples.

Check yourself

Question 1

1. What is data cleaning?

Question 2

2. An unusually high value turns out to be a real special event. What should you do?

Question 3

3. Why count examples per group during exploration?

Summary

  • Data exploration studies and checks data before modelling.
  • Summaries and charts reveal patterns, errors and outliers.
  • Investigate before deleting, and check every group is represented.

Finished reading?

Sources: Open Government Data Platform India · Google Machine Learning Crash Course: Classification