Definition
The canteen's sales register says 4,000 lunches were sold on one Tuesday. The school has 900 students. Before you train any model, something needs a closer look.
Data exploration is the stage of the AI project cycle where the team studies the collected data to understand it and check its quality. It involves summarising and visualising the data, and finding and fixing problems such as missing values, duplicates, errors, outliers and imbalance. Fixing these problems is called data cleaning.
In simple words
Before cooking rice, you spread it on a plate and pick out the small stones and husks. You do not just throw the whole bag into the pot. Data exploration is picking the stones out of your data before you feed it to a model.
Why it matters
A model trained on messy data learns the mess. Data scientists often say they spend most of their time cleaning and understanding data, because this step decides how good the final model can be.
Key terms
- Data exploration
- Studying data to understand it and check its quality before modelling.
- Data cleaning
- Fixing or removing errors, gaps and duplicates in data.
- Missing value
- A blank where data should be.
- Duplicate
- The same record appearing more than once.
- Outlier
- A value far from the rest, which may be an error or a real but rare event.
- Imbalanced data
- Data where some groups or outcomes have far more examples than others.
Explained in detail
First, get to know the data
Start by looking at the data as a whole. How many rows and columns are there? What does each column mean, and in what units? What are the smallest, largest and typical values in each column? Then draw a few charts: a line graph of daily sales over the term, a bar graph of average sales by day of the week. Patterns appear quickly: Mondays are busy, exam weeks are quiet. So do problems: a spike to 4,000 on one Tuesday stands out immediately.
Common problems and fixes
| Problem | Example | Possible fix |
|---|---|---|
| Missing values | No sales recorded on three days | Check whether the canteen was shut; if not, fill from nearby days or leave those days out |
| Duplicates | The same day entered twice | Remove the extra copy |
| Errors | 4,000 instead of 400, a typing slip | Correct it from the original register, or remove it |
| Inconsistent formats | "Mon", "Monday" and "monday" in one column | Use one standard spelling |
| Real outliers | Annual day, when parents also ate | Keep, but mark it as a special day |
| Imbalance | Only five exam days among a hundred | Collect more examples, or treat the model's exam-day predictions with caution |
Not every outlier is a mistake
It is tempting to delete every unusual value, but some are real and important. The annual day really did have many more diners. Deleting it would make the model unprepared for the next annual day. The rule is to investigate before deciding: find out why a value is unusual. If it is an error, fix or remove it. If it is real, keep it and, if possible, add a feature that explains it, such as "special event: yes".
Checking for fairness
Exploration is also when you check whether every group is properly represented. For a crop disease app, are there photos from every region, every crop variety, every season, every kind of phone camera? For a voice assistant, are there recordings of children, older people and every major accent? A simple count of examples per group often reveals gaps that would otherwise become unfairness in the finished model. Finding them now is far cheaper than discovering them after the app is used by millions.
How it works
- Count rows and columns and read what each column means.
- Summarise each column: smallest, largest, average, number of blanks.
- Draw charts to spot patterns and odd values.
- Investigate odd values, then fix errors, remove duplicates and handle blanks.
- Count examples per group to check balance and fairness.
Try it yourself
The Tuesday with 4,000 lunches
The team explores a term's worth of canteen sales.
Match the problem to the fix
How would you handle each data problem?
- The same record appears twice
- A typing slip adds an extra zero
- A real but unusual event day
- "Mon" and "Monday" in one column
Common mistakes
- Training a model without looking at the data first.
- Deleting every unusual value without investigating.
- Not checking whether every group has enough examples.
Check yourself
Summary
- Data exploration studies and checks data before modelling.
- Summaries and charts reveal patterns, errors and outliers.
- Investigate before deleting, and check every group is represented.
Finished reading?
Sources: Open Government Data Platform India · Google Machine Learning Crash Course: Classification