B2 · How Artificial Intelligence Works: Data and Learning

Learning Compared with Memorising

Memorising means storing the exact examples seen before. Learning means finding the general pattern behind them, so that new examples can be handled. A model that only memorises does well on its training examples and badly on new ones; this problem is called overfitting.

6 min Beginner Jump to the activity

What you need: Nothing. This page starts from the beginning, with no programming and no formulas.

Definition

Is learning answers by heart the same as understanding?

Memorising means storing the exact examples seen before. Learning means finding the general pattern behind them, so that new examples can be handled. A model that only memorises does well on its training examples and badly on new ones; this problem is called overfitting.

In simple words

Two students prepare for an exam. One memorises the answers to last year's paper. The other understands the chapter. When the exam has new questions, the first student is stuck and the second does well. Good AI must be like the second student.

Why it matters

Overfitting is one of the most common reasons AI fails in the real world. It is why we always test on new examples, and why more varied data usually helps more than the same data repeated.

Key terms

Memorising
Storing the exact training examples and their answers, without grasping the pattern behind them.
Generalisation
Handling new examples correctly by using the pattern learned from old ones.
Overfitting
Fitting the training examples so closely, including their accidents, that performance on new examples suffers.
Underfitting
Learning too simple a pattern, so that even the training examples are handled poorly.
Noise
Random quirks and errors in data that are not part of the real pattern.

Explained in detail

Two students

Two students prepare for a mathematics test using the same twenty practice questions. The first memorises each question with its answer. The second works out why each answer is right. On the practice questions both score full marks, and you cannot tell them apart. The real test has new questions. The second student does well. The first is lost, because a memorised answer is useless for a question that has changed even slightly. A machine learning model can end up as either student, and from the training score alone you cannot tell which.

What overfitting is

Every set of training examples contains two things mixed together: the real pattern, and accidents. In a set of dog photos, the real pattern is the shape of a dog. The accidents are things like the fact that three of the photos happen to have a red sofa in the background. A model with enough capacity will keep reducing its training error by learning the accidents as well: "red sofa means dog". It now fits the training examples perfectly, and it has become worse at new photos. This is overfitting. The model has learned the examples and not the idea.

The opposite problem

A model can also fail by learning too little. If it settles on a pattern as crude as "everything with four legs is a dog", it will be wrong about cats, cows and tables, on the training examples and on new ones alike. This is underfitting. It happens when the model is too simple for the task, is given too few useful features, or has not been trained for long enough.

Score on training examplesScore on new examplesWhat it means
UnderfittingLowLowThe pattern learned is too simple
Good fitHighHigh, and close to the training scoreThe real pattern was learned
OverfittingVery highMuch lowerThe examples were memorised

How to spot it and prevent it

The sign of overfitting is a gap: a high score on the training examples and a clearly lower score on the test set. That is the whole reason for keeping a test set. Several things reduce overfitting.

  • More examples, and more varied ones, so that accidents do not repeat and only the real pattern survives.
  • A simpler model, which has less room to store accidents.
  • Stopping training at the point where the score on unseen examples stops improving.
  • Cleaner data, with wrong labels corrected.

How it works

Memorising modelLearning model
Training examplesNearly perfectVery good
New examplesPoorGood
What it storedEvery example exactlyThe general pattern

Try it yourself

The memoriser and the learner

The student studies the practice questions. Move the slider to change how they study, then give the exam.

Practice questions compared with a real exam

Discuss this with a friend or think it through on your own.

  1. Write five maths questions and their answers on cards.
  2. Partner A memorises the five answers. Partner B learns the method.
  3. Test both with the same five questions, then with five new ones.
  4. Compare their scores on old and new questions.

Think about it

  • Which partner is more like a useful AI model?
  • How could you tell, without new questions, that someone only memorised?

Learning or memorising?

Which does each behaviour show?

  • Gets every training example right but fails on new ones

  • Recognises a cat only in the exact photos it saw

  • Recognises a cat in a new photo taken at night

  • Does about as well on new examples as on old ones

Common mistakes

  • Celebrating a perfect score on training examples.
  • Thinking the fix is always more of the same data. Varied examples help more.
  • Forgetting to keep test examples aside.

Check yourself

Question 1

1. What is overfitting?

Question 2

2. How do you catch a memorising model?

Question 3

3. Which student is like a good model?

Summary

  • Memorising stores examples; learning finds the pattern behind them.
  • A memorising model overfits: great on training data, poor on new data.
  • Testing on new examples reveals the difference.

Finished reading?