Definition
Is learning answers by heart the same as understanding?
Memorising means storing the exact examples seen before. Learning means finding the general pattern behind them, so that new examples can be handled. A model that only memorises does well on its training examples and badly on new ones; this problem is called overfitting.
In simple words
Two students prepare for an exam. One memorises the answers to last year's paper. The other understands the chapter. When the exam has new questions, the first student is stuck and the second does well. Good AI must be like the second student.
Why it matters
Overfitting is one of the most common reasons AI fails in the real world. It is why we always test on new examples, and why more varied data usually helps more than the same data repeated.
Key terms
- Memorising
- Storing the exact training examples and their answers, without grasping the pattern behind them.
- Generalisation
- Handling new examples correctly by using the pattern learned from old ones.
- Overfitting
- Fitting the training examples so closely, including their accidents, that performance on new examples suffers.
- Underfitting
- Learning too simple a pattern, so that even the training examples are handled poorly.
- Noise
- Random quirks and errors in data that are not part of the real pattern.
Explained in detail
Two students
Two students prepare for a mathematics test using the same twenty practice questions. The first memorises each question with its answer. The second works out why each answer is right. On the practice questions both score full marks, and you cannot tell them apart. The real test has new questions. The second student does well. The first is lost, because a memorised answer is useless for a question that has changed even slightly. A machine learning model can end up as either student, and from the training score alone you cannot tell which.
What overfitting is
Every set of training examples contains two things mixed together: the real pattern, and accidents. In a set of dog photos, the real pattern is the shape of a dog. The accidents are things like the fact that three of the photos happen to have a red sofa in the background. A model with enough capacity will keep reducing its training error by learning the accidents as well: "red sofa means dog". It now fits the training examples perfectly, and it has become worse at new photos. This is overfitting. The model has learned the examples and not the idea.
The opposite problem
A model can also fail by learning too little. If it settles on a pattern as crude as "everything with four legs is a dog", it will be wrong about cats, cows and tables, on the training examples and on new ones alike. This is underfitting. It happens when the model is too simple for the task, is given too few useful features, or has not been trained for long enough.
| Score on training examples | Score on new examples | What it means | |
|---|---|---|---|
| Underfitting | Low | Low | The pattern learned is too simple |
| Good fit | High | High, and close to the training score | The real pattern was learned |
| Overfitting | Very high | Much lower | The examples were memorised |
How to spot it and prevent it
The sign of overfitting is a gap: a high score on the training examples and a clearly lower score on the test set. That is the whole reason for keeping a test set. Several things reduce overfitting.
- More examples, and more varied ones, so that accidents do not repeat and only the real pattern survives.
- A simpler model, which has less room to store accidents.
- Stopping training at the point where the score on unseen examples stops improving.
- Cleaner data, with wrong labels corrected.
How it works
| Memorising model | Learning model | |
|---|---|---|
| Training examples | Nearly perfect | Very good |
| New examples | Poor | Good |
| What it stored | Every example exactly | The general pattern |
Try it yourself
The memoriser and the learner
The student studies the practice questions. Move the slider to change how they study, then give the exam.
Practice questions compared with a real exam
Discuss this with a friend or think it through on your own.
- Write five maths questions and their answers on cards.
- Partner A memorises the five answers. Partner B learns the method.
- Test both with the same five questions, then with five new ones.
- Compare their scores on old and new questions.
Think about it
- Which partner is more like a useful AI model?
- How could you tell, without new questions, that someone only memorised?
Learning or memorising?
Which does each behaviour show?
Gets every training example right but fails on new ones
Recognises a cat only in the exact photos it saw
Recognises a cat in a new photo taken at night
Does about as well on new examples as on old ones
Common mistakes
- Celebrating a perfect score on training examples.
- Thinking the fix is always more of the same data. Varied examples help more.
- Forgetting to keep test examples aside.
Check yourself
Summary
- Memorising stores examples; learning finds the pattern behind them.
- A memorising model overfits: great on training data, poor on new data.
- Testing on new examples reveals the difference.
Finished reading?