B4 · Domains of AI: Vision, Language, Speech and Robotics

Why Computer Vision Makes Mistakes

A computer vision model makes mistakes when the images it meets differ from the images it learned from: different lighting, angle, distance or background, objects that are partly hidden, unusual examples, or things it was never shown. It can also learn the wrong clue, such as the background, and not the object itself.

8 min Beginner Jump to the activity

What you need: Nothing. This page starts from the beginning, with no programming and no formulas.

Definition

A model was 99% accurate in the lab and then failed badly on the first rainy day. What went wrong?

A computer vision model makes mistakes when the images it meets differ from the images it learned from: different lighting, angle, distance or background, objects that are partly hidden, unusual examples, or things it was never shown. It can also learn the wrong clue, such as the background, and not the object itself.

In simple words

Suppose you learned what a cow looks like only from photos of brown cows standing in green fields at noon. Then someone shows you a white cow lying on a road at night. You might hesitate. A vision model does not hesitate. It answers anyway, and often wrongly.

Why it matters

Vision models are trusted with driving, medical screening and security. Knowing how they fail tells you when to be careful, and it explains why a good team tests in rain, at night and with unusual cases before launch.

Key terms

Distribution shift
A difference between the data a model was trained on and the data it meets in use.
Occlusion
An object being partly hidden behind something else.
Shortcut learning
A model relying on an easy clue that happens to work in training but is not the real reason.
Adversarial example
An input changed slightly and deliberately so that a model gets it wrong.
Robustness
A model's ability to keep working when conditions change.
Data augmentation
Making extra training images by flipping, rotating, darkening or cropping the ones you have.

Explained in detail

The world is larger than any dataset

A vision model is tested on images held back from the same collection it was trained on. A high score shows that it handles images of that kind. The world then presents images of other kinds. In 2019 researchers built a test set of ordinary household objects photographed in unusual positions: a chair lying on its side, a hammer seen from the end, a kettle on a bed. Models that scored very well on the standard test lost roughly forty percentage points of accuracy on these photos. Nothing was wrong with the objects. They simply did not look like the training pictures.

Shortcuts: right answers for wrong reasons

A model is rewarded only for getting the training answers right, so it uses whatever clue works most easily. Sometimes that clue has nothing to do with the task. In one study, a model for detecting pneumonia in chest X-rays turned out to rely partly on marks that showed which hospital the X-ray came from. Some hospitals treated sicker patients, so the hospital was a good predictor in training and useless anywhere else. The model had learned where the picture was taken, not what illness looked like.

Texture over shape

People recognise objects mainly by shape. A cat drawn in outline is still a cat. Researchers discovered in 2019 that vision models lean heavily on surface texture. When they made a picture with the clear shape of a cat but the wrinkled grey skin texture of an elephant, people said "cat" and the models said "elephant". This shows that a model can reach the same answers as people on ordinary photos while using quite different evidence, which is why its mistakes can seem so strange.

Deliberate attacks

In 2014 researchers showed something more unsettling. They took a photo that a model correctly labelled "panda" and added a faint pattern of changes to its pixels, too small for a person to see. The model then labelled the unchanged-looking photo "gibbon", with very high confidence. Such inputs are called adversarial examples. Later work produced printed stickers that make a model misread a road sign, and patterned glasses that confuse face recognition. A system used for security or safety has to be designed with people in mind who will try to fool it.

How engineers make models sturdier

MethodWhat it does
Wider dataCollect images from many places, seasons, cameras and kinds of people
Data augmentationFlip, rotate, blur and darken training images so the model sees more variety
Hard-case collectionDeliberately gather the cases the model gets wrong and add them
An "unsure" answerLet the model pass doubtful cases to a person
Field testingTry the system in the real place before relying on it
MonitoringKeep checking accuracy after launch, because conditions drift

How it works

CauseExampleFix
LightingTrained in daylight, used at nightAdd night, dusk and rainy photos
Angle and distanceTrained on front views, sees objects from aboveAdd photos from many angles and distances
Hidden objectsA person half behind a pillar is missedAdd partly hidden examples
Unusual examplesA cycle carrying a huge load is not recognised as a cycleCollect rare cases on purpose
Wrong clueLearns "snow means wolf" from the backgroundsVary the backgrounds; check what the model looks at
Never seenA fruit model shown a shoe still names a fruitAdd an "other" group; show low confidence

The model that learned the background

In a well-known teaching experiment, researchers trained a model to tell wolves from husky dogs. It scored well. Then they checked which parts of each photo the model was using, and found it was looking at the snow. Most of the wolf photos had snow in the background, so the model had learned "snow means wolf". It had found a pattern that worked on the training photos and meant nothing about the animals.

Try it yourself

Lab day and rainy day

A model was trained to spot potholes from photos taken on sunny afternoons.

What went wrong?

Match each failure to its most likely cause.

  • A model calls any animal on grass a 'cow'

  • A model says 'doctor' whenever it sees a white coat, even on a painter

  • A helmet detector works at noon but fails at dusk

  • A skin-check model trained on one skin tone does poorly on others

  • A fruit sorter calls a tennis ball a 'lemon'

  • A number-plate reader fails on plates covered in mud

Common mistakes

  • Thinking a high test score proves the model works everywhere.
  • Thinking the model sees objects the way people do. It sees patterns of pixels.
  • Fixing errors by training longer on the same photos. The fix is usually different photos.

Check yourself

Question 1

1. A model trained only on daytime photos is used at night. Why may it fail?

Question 2

2. In the wolf and husky experiment, what had the model really learned?

Question 3

3. What is the usual way to fix a model that fails in rain?

Summary

  • Vision models fail when real images differ from their training images.
  • Lighting, angle, hidden objects, rare cases and wrong clues are the common causes.
  • The cure is wider, more varied data and testing in real conditions.

Finished reading?

Sources: Google Machine Learning Crash Course · AI4K12: Five Big Ideas in AI