B2 · How Artificial Intelligence Works: Data and Learning

Measuring Accuracy

Accuracy is the share of predictions a model gets right on a test set, usually given as a percentage. If a model is right on 8 of 10 test examples, its accuracy is 80 per cent.

8 min Beginner Jump to the activity

What you need: Nothing. This page starts from the beginning, with no programming and no formulas.

Definition

The model got 8 out of 10 test pictures right. How good is that?

Accuracy is the share of predictions a model gets right on a test set, usually given as a percentage. If a model is right on 8 of 10 test examples, its accuracy is 80 per cent.

In simple words

Accuracy is like your score in a test: count the right answers, divide by the number of questions, and turn it into a percentage. 8 out of 10 is 80 per cent; 45 out of 50 is 90 per cent. But just like a test, the score only means something if the questions were fair and new.

Why it matters

Accuracy is the first number people check to decide whether a model is good enough. It also lets you compare two models, or the same model before and after you improve it.

Key terms

Accuracy
The share of a model's predictions that are correct, usually given as a percentage.
Test set
The unseen, labelled examples used to measure accuracy.
False alarm
The model says "yes" when the true answer is "no", such as marking a genuine email as spam.
Miss
The model says "no" when the true answer is "yes", such as letting a spam email through.
Baseline
The score of a very simple method, used as a comparison.
Imbalanced data
Data in which one class is far more common than another.

Explained in detail

Counting right answers

Accuracy is the simplest way to say how good a model is. Give the model a set of test examples it has never seen. Count how many it gets right. Divide by the total and express the result as a percentage. Eight right out of ten is 80 per cent. Forty-five right out of fifty is 90 per cent. Because it is a single number that everyone understands, accuracy is the figure you will see most often in news about AI.

How many test examples?

A score from a tiny test is shaky. With ten test examples, a single lucky or unlucky case moves the score by ten points. With a thousand, one case moves it by a tenth of a point. Real projects test on thousands of examples so that the score can be trusted, and they make sure those examples cover the variety the model will meet.

When accuracy misleads

Suppose a bank handles 10,000 payments and 10 of them are fraudulent. A lazy model that says "genuine" every single time is right 9,990 times out of 10,000. Its accuracy is 99.9 per cent, and it has caught no fraud at all. This happens whenever the data is imbalanced, with one class much rarer than the other, and rare classes are often the ones that matter most: fraud, disease, defects, emergencies. A high accuracy figure on its own can hide a useless model.

The defence is a baseline. Before being impressed by a score, ask what the simplest possible method would score. If always guessing the most common answer gives 99.9 per cent, then a model at 99.9 per cent has learned nothing.

Two kinds of mistake

Accuracy treats every mistake as equal. In real life they are not. A model that answers yes or no can be wrong in two ways. A false alarm is saying yes when the truth is no. A miss is saying no when the truth is yes. Which is worse depends entirely on the task.

TaskFalse alarmMissWorse mistake
Screening for a serious diseaseA healthy person is sent for more testsA sick person is told they are fineThe miss
Spam filterAn important email goes to the spam folderOne spam email reaches the inboxUsually the false alarm
Fire alarmPeople leave the building for nothingA real fire goes unnoticedThe miss

Because of this, people who evaluate models report the two kinds of error separately as well as the overall accuracy. They also report accuracy separately for different groups of people, since a model can be excellent on average and poor for one group. You will study these measures in more detail in the module on the AI project cycle.

How it works

Right answersTest examplesAccuracy
81080 per cent
455090 per cent
69About 67 per cent

Try it yourself

Mark the model's test

Ten pictures the model has never seen. How many did it get right?

Score a model on ten test pictures

For each test picture, decide whether the model was right. Then count: what is its accuracy?

  • Picture of a dog. Model said: dog

  • Picture of a cat. Model said: cat (4th)

  • Picture of a cat. Model said: dog

  • Picture of a dog. Model said: dog (4th)

  • Picture of a cat. Model said: cat

  • Picture of a cat. Model said: cat (2nd)

  • Picture of a small dog. Model said: cat

  • Picture of a cat. Model said: cat (3rd)

  • Picture of a dog. Model said: dog (2nd)

  • Picture of a dog. Model said: dog (3rd)

Common mistakes

  • Measuring accuracy on training examples instead of test examples.
  • Trusting high accuracy when one answer is far more common than the other.
  • Ignoring which mistakes were made. Some mistakes matter more than others.

Check yourself

Question 1

1. A model is right on 9 of 10 test examples. Its accuracy is…

Question 2

2. Where should accuracy be measured?

Question 3

3. 95 of 100 patients are healthy. A model says 'healthy' for everyone. What's wrong?

Summary

  • Accuracy is the percentage of correct predictions on a test set.
  • Count the right answers and compare with the total.
  • High accuracy can mislead when one answer is much more common.

Finished reading?

Sources: Google Machine Learning Crash Course