B2 · How Artificial Intelligence Works: Data and Learning

When AI Learns the Wrong Clue

A model learns the wrong clue when it picks up something that happens to go with the right answer in its training examples, such as the colour of the background, instead of the thing that truly decides the answer. It then works on examples like its training data and fails when that accidental clue changes. This is often called shortcut learning.

10 min Beginner Jump to the activity

What you need: Nothing. This page starts from the beginning, with no programming and no formulas.

Definition

A model learns to tell mangoes from apples and scores perfectly on its test. Then someone puts a mango on a white plate, and the model says 'apple'. What did it really learn?

A model learns the wrong clue when it picks up something that happens to go with the right answer in its training examples, such as the colour of the background, instead of the thing that truly decides the answer. It then works on examples like its training data and fails when that accidental clue changes. This is often called shortcut learning.

In simple words

Imagine a friend who finds your house by remembering 'the one with the red car outside'. It works until the car is moved. A model can do the same: it remembers an easy clue that happened to be there, instead of the thing that really matters.

Why it matters

A model that learned a shortcut can look excellent in testing and then fail badly in real use, without any warning. Knowing how to test for wrong clues, by changing one thing at a time, helps you trust a model only when it has learned the right thing.

Key terms

Clue (feature)
A piece of information a model can use to decide, such as colour, shape or background.
Shortcut
An easy clue that matches the answer in the training examples by accident, but not in real life.
Background
Everything in a picture around the main object, such as the plate, the table or the sky.
Swap test
A test that changes one thing, such as the plate colour, while keeping the object the same, to see what the model really uses.
Varied examples
Training examples that show each object in many settings, so no accidental clue goes with only one answer.

Explained in detail

How the plate became the clue

In this activity, every training photo of a mango shows it on a green plate, and every photo of an apple shows it on a white plate. To the model, a photo is just colours and shapes. The plate fills much of the picture, and its colour separates the two groups perfectly, so the easiest thing to learn is 'green plate means mango, white plate means apple'. On test photos taken the same way, this is always right, so the model scores 100 per cent. Nobody told it to look at the plate; the examples simply made the plate the easiest clue.

Testing what the model really learned

To find out, change one thing at a time. Put a mango on a white plate and an apple on a green plate. If the model now gets both wrong, it was reading the plate. If it still gets both right, it learned something about the fruit itself. This is called a swap test, and it is one of the simplest ways to check a model. Good testers also try photos with no plate, a different light or a half-eaten fruit, because each change can reveal another hidden clue.

Test photoModel saysWhat it shows
Mango on a green plateMangoLooks fine, but proves nothing
Mango on a white plateAppleThe model was reading the plate
Apple on a green plateMangoConfirms that the plate was the clue
Apple with no plateUnsureIt never learned the fruit's shape

It happens in real research too

In 2016 the researchers Marco Tulio Ribeiro, Sameer Singh and Carlos Guestrin studied a model that told wolves from husky dogs, trained on photos in which the wolves stood in snow. The model did well on similar photos, but a tool that showed which parts of each picture it used revealed that it was looking at the snow, not the animal. They used it to show that a good score is not enough; we also need to know what a model is paying attention to.

The fix is better examples: show each fruit on both plates, on tables, in hands and in different light, so that only the fruit itself separates the groups. Then test again with swapped backgrounds before trusting the model.

How it works

  1. Look at the training examples and list anything that goes with only one answer, such as plate colour.
  2. Make swap tests that change that thing but keep the object the same.
  3. Run the model on the swap tests and compare with the normal tests.
  4. If results drop, add varied examples and train again.
  5. Repeat the swap tests before using the model.

Worked example

Training: 20 mangoes on green plates and 20 apples on white plates. Normal test: 10 new photos taken the same way, and all 10 are right. Swap test: 5 mangoes on white plates and 5 apples on green plates, and the model gets all 10 wrong, calling every mango on a white plate an apple. The plate was the clue. After retraining with each fruit on both plates, the swap test gives 9 of 10 right, so now the model is using the fruit.

Try it yourself

Fruit or plate?

Follow the swap test that reveals the model's shortcut.

Real clue or possible shortcut?

For each model, decide whether the clue truly decides the answer.

  • Cat or dog: the shape of the ears and face

  • Ripe banana: the colour of the skin

  • Ripe banana: a shop's logo in the corner of every ripe photo

  • Handwritten number: the shape of the strokes

  • Cat or dog: whether the photo was taken indoors

  • Handwritten number: the colour of the paper used for each number

A new case: cats and dogs

Plan a test on paper.

  1. A model tells cats from dogs. All its dog photos were taken in a park and all its cat photos indoors. What shortcut could it learn?
  2. Describe two swap-test photos that would reveal the shortcut.
  3. Describe three new training photos that would remove the shortcut.

Think about it

  • Why did the model score well in testing even though it learned the wrong clue?

Common mistakes

  • Trusting a perfect score on test photos that look just like the training photos.
  • Assuming the model looks at the same thing a person does.
  • Trying to fix the problem by adding more photos of the same kind.

Check yourself

Question 1

1. All training mangoes were on green plates. A mango on a white plate is called 'apple'. What did the model learn?

Question 2

2. What is a swap test?

Question 3

3. Which training set best prevents the plate shortcut?

Question 4

4. New case: a model spots ripe tomatoes. Every ripe tomato photo was taken in sunlight and every unripe one in shade. What should you test?

Summary

  • A model can learn an accidental clue, such as the background, instead of the real one.
  • Swap tests change one thing at a time to reveal what the model really uses.
  • Varied training examples remove shortcuts; test again before trusting the model.

Finished reading?

Sources: Ribeiro, Singh and Guestrin · "Why should I trust you?" Explaining the predictions of any classifier (2016)