B2 · How Artificial Intelligence Works: Data and Learning

Testing a Model

Testing a model means checking its predictions on new examples it did not see during training, and comparing them with the right answers. The examples kept aside for this are called the test set.

8 min Beginner Jump to the activity

What you need: Nothing. This page starts from the beginning, with no programming and no formulas.

Definition

Would you trust a model you never tested?

Testing a model means checking its predictions on new examples it did not see during training, and comparing them with the right answers. The examples kept aside for this are called the test set.

In simple words

If a teacher gave you the exact exam questions to practise, a perfect score wouldn't prove you understood. A fair exam uses new questions. In the same way, a model is tested only on examples it has never seen, so the result shows whether it really learned the pattern.

Why it matters

A model that is only checked on its training examples can look perfect and then fail in real use. Testing on new examples is how engineers find out whether a model is ready, and where it goes wrong, before people rely on it.

Key terms

Testing
Checking a trained model on examples it has never seen, to find out how well it really works.
Test set
Labelled examples kept aside from training and used only for testing.
Unseen data
Data that played no part in training the model.
Data leakage
When test examples, or hints about them, slip into training and make the model look better than it is.
Field test
A trial of the model in the real place and conditions where it will be used.

Explained in detail

Why a separate test?

Imagine a teacher who hands out a practice paper, goes through the answers, and then sets exactly the same paper as the final exam. Everyone scores full marks, and the teacher learns nothing about who understood the subject. A fair exam has new questions on the same topics. Testing a model works on the same principle. If you check it on the examples it trained on, a high score tells you only that it remembers them. To know whether it has learned the pattern, you must try it on examples it has never seen.

Splitting the data

Before training starts, the labelled examples are divided. A common choice is to use about four-fifths for training and keep one-fifth aside as the test set. The split should be made at random, so that both parts contain the same mix of cases. The test set is then locked away. The model must never be adjusted using it, and neither should the people building the model keep peeking at it and tuning their design to raise the test score, because that slowly turns the test into practice.

The mistake called leakage

Sometimes test examples reach the training set by accident. The same photo may appear twice under different file names, or several photos of the same patient may be split across both sets. The model then scores well because it has, in effect, seen the answers. This is called data leakage, and it has caused published results to collapse when others tried to repeat them. A result that looks too good is a reason to check for leakage before celebrating.

A good test looks like real use

A test set drawn from the same collection as the training data shares its blind spots. If every training photo was taken in daylight, so was every test photo, and the score says nothing about night. A careful team therefore builds a test set that matches the conditions the model will actually meet: different places, seasons, cameras, accents and kinds of people. It reports the results separately for each group, because a model that is 95 per cent accurate overall may be 99 per cent accurate for one group and 70 per cent for another.

From the lab to the field

Passing a test set is the beginning, not the end. Before a model is trusted with real decisions it should be tried in the field, in the hospital, farm or street where it will work, at first alongside people who can catch its mistakes. Testing continues after launch as well, because the world changes. A spam filter tested last year has never seen this year's scams. Models need regular re-testing, just as a vehicle needs regular servicing.

How it works

  1. Before training, put some examples aside. The model must never see them.
  2. Train the model on the rest.
  3. Ask the model about each test example.
  4. Mark each prediction right or wrong.
  5. Count the right ones and look closely at the wrong ones.

Try it yourself

Test your model with new animals

This model is already trained. Check its answer for each new animal, then test all nine and count how many it got right.

Common mistakes

  • Testing on the training examples. That measures memory, not learning.
  • Testing on too few examples to be sure.
  • Looking only at the score and never at the mistakes.

Check yourself

Question 1

1. What is the test set?

Question 2

2. Why not test on training examples?

Question 3

3. After testing, what should you study most carefully?

Summary

  • Test a model on new examples it has never seen.
  • Keep the test set aside before training starts.
  • Study the mistakes: they show what to improve.

Finished reading?

Sources: Google Machine Learning Crash Course