Definition
The model got 8 out of 10 test pictures right. How good is that?
Accuracy is the share of predictions a model gets right on a test set, usually given as a percentage. If a model is right on 8 of 10 test examples, its accuracy is 80 per cent.
In simple words
Accuracy is like your score in a test: count the right answers, divide by the number of questions, and turn it into a percentage. 8 out of 10 is 80 per cent; 45 out of 50 is 90 per cent. But just like a test, the score only means something if the questions were fair and new.
Why it matters
Accuracy is the first number people check to decide whether a model is good enough. It also lets you compare two models, or the same model before and after you improve it.
Key terms
- Accuracy
- The share of a model's predictions that are correct, usually given as a percentage.
- Test set
- The unseen, labelled examples used to measure accuracy.
- False alarm
- The model says "yes" when the true answer is "no", such as marking a genuine email as spam.
- Miss
- The model says "no" when the true answer is "yes", such as letting a spam email through.
- Baseline
- The score of a very simple method, used as a comparison.
- Imbalanced data
- Data in which one class is far more common than another.
Explained in detail
Counting right answers
Accuracy is the simplest way to say how good a model is. Give the model a set of test examples it has never seen. Count how many it gets right. Divide by the total and express the result as a percentage. Eight right out of ten is 80 per cent. Forty-five right out of fifty is 90 per cent. Because it is a single number that everyone understands, accuracy is the figure you will see most often in news about AI.
How many test examples?
A score from a tiny test is shaky. With ten test examples, a single lucky or unlucky case moves the score by ten points. With a thousand, one case moves it by a tenth of a point. Real projects test on thousands of examples so that the score can be trusted, and they make sure those examples cover the variety the model will meet.
When accuracy misleads
Suppose a bank handles 10,000 payments and 10 of them are fraudulent. A lazy model that says "genuine" every single time is right 9,990 times out of 10,000. Its accuracy is 99.9 per cent, and it has caught no fraud at all. This happens whenever the data is imbalanced, with one class much rarer than the other, and rare classes are often the ones that matter most: fraud, disease, defects, emergencies. A high accuracy figure on its own can hide a useless model.
The defence is a baseline. Before being impressed by a score, ask what the simplest possible method would score. If always guessing the most common answer gives 99.9 per cent, then a model at 99.9 per cent has learned nothing.
Two kinds of mistake
Accuracy treats every mistake as equal. In real life they are not. A model that answers yes or no can be wrong in two ways. A false alarm is saying yes when the truth is no. A miss is saying no when the truth is yes. Which is worse depends entirely on the task.
| Task | False alarm | Miss | Worse mistake |
|---|---|---|---|
| Screening for a serious disease | A healthy person is sent for more tests | A sick person is told they are fine | The miss |
| Spam filter | An important email goes to the spam folder | One spam email reaches the inbox | Usually the false alarm |
| Fire alarm | People leave the building for nothing | A real fire goes unnoticed | The miss |
Because of this, people who evaluate models report the two kinds of error separately as well as the overall accuracy. They also report accuracy separately for different groups of people, since a model can be excellent on average and poor for one group. You will study these measures in more detail in the module on the AI project cycle.
How it works
| Right answers | Test examples | Accuracy |
|---|---|---|
| 8 | 10 | 80 per cent |
| 45 | 50 | 90 per cent |
| 6 | 9 | About 67 per cent |
Try it yourself
Mark the model's test
Ten pictures the model has never seen. How many did it get right?
Score a model on ten test pictures
For each test picture, decide whether the model was right. Then count: what is its accuracy?
Picture of a dog. Model said: dog
Picture of a cat. Model said: cat (4th)
Picture of a cat. Model said: dog
Picture of a dog. Model said: dog (4th)
Picture of a cat. Model said: cat
Picture of a cat. Model said: cat (2nd)
Picture of a small dog. Model said: cat
Picture of a cat. Model said: cat (3rd)
Picture of a dog. Model said: dog (2nd)
Picture of a dog. Model said: dog (3rd)
Common mistakes
- Measuring accuracy on training examples instead of test examples.
- Trusting high accuracy when one answer is far more common than the other.
- Ignoring which mistakes were made. Some mistakes matter more than others.
Check yourself
Summary
- Accuracy is the percentage of correct predictions on a test set.
- Count the right answers and compare with the total.
- High accuracy can mislead when one answer is much more common.
Finished reading?
Sources: Google Machine Learning Crash Course