Definition
A model was 99% accurate in the lab and then failed badly on the first rainy day. What went wrong?
A computer vision model makes mistakes when the images it meets differ from the images it learned from: different lighting, angle, distance or background, objects that are partly hidden, unusual examples, or things it was never shown. It can also learn the wrong clue, such as the background, and not the object itself.
In simple words
Suppose you learned what a cow looks like only from photos of brown cows standing in green fields at noon. Then someone shows you a white cow lying on a road at night. You might hesitate. A vision model does not hesitate. It answers anyway, and often wrongly.
Why it matters
Vision models are trusted with driving, medical screening and security. Knowing how they fail tells you when to be careful, and it explains why a good team tests in rain, at night and with unusual cases before launch.
Key terms
- Distribution shift
- A difference between the data a model was trained on and the data it meets in use.
- Occlusion
- An object being partly hidden behind something else.
- Shortcut learning
- A model relying on an easy clue that happens to work in training but is not the real reason.
- Adversarial example
- An input changed slightly and deliberately so that a model gets it wrong.
- Robustness
- A model's ability to keep working when conditions change.
- Data augmentation
- Making extra training images by flipping, rotating, darkening or cropping the ones you have.
Explained in detail
The world is larger than any dataset
A vision model is tested on images held back from the same collection it was trained on. A high score shows that it handles images of that kind. The world then presents images of other kinds. In 2019 researchers built a test set of ordinary household objects photographed in unusual positions: a chair lying on its side, a hammer seen from the end, a kettle on a bed. Models that scored very well on the standard test lost roughly forty percentage points of accuracy on these photos. Nothing was wrong with the objects. They simply did not look like the training pictures.
Shortcuts: right answers for wrong reasons
A model is rewarded only for getting the training answers right, so it uses whatever clue works most easily. Sometimes that clue has nothing to do with the task. In one study, a model for detecting pneumonia in chest X-rays turned out to rely partly on marks that showed which hospital the X-ray came from. Some hospitals treated sicker patients, so the hospital was a good predictor in training and useless anywhere else. The model had learned where the picture was taken, not what illness looked like.
Texture over shape
People recognise objects mainly by shape. A cat drawn in outline is still a cat. Researchers discovered in 2019 that vision models lean heavily on surface texture. When they made a picture with the clear shape of a cat but the wrinkled grey skin texture of an elephant, people said "cat" and the models said "elephant". This shows that a model can reach the same answers as people on ordinary photos while using quite different evidence, which is why its mistakes can seem so strange.
Deliberate attacks
In 2014 researchers showed something more unsettling. They took a photo that a model correctly labelled "panda" and added a faint pattern of changes to its pixels, too small for a person to see. The model then labelled the unchanged-looking photo "gibbon", with very high confidence. Such inputs are called adversarial examples. Later work produced printed stickers that make a model misread a road sign, and patterned glasses that confuse face recognition. A system used for security or safety has to be designed with people in mind who will try to fool it.
How engineers make models sturdier
| Method | What it does |
|---|---|
| Wider data | Collect images from many places, seasons, cameras and kinds of people |
| Data augmentation | Flip, rotate, blur and darken training images so the model sees more variety |
| Hard-case collection | Deliberately gather the cases the model gets wrong and add them |
| An "unsure" answer | Let the model pass doubtful cases to a person |
| Field testing | Try the system in the real place before relying on it |
| Monitoring | Keep checking accuracy after launch, because conditions drift |
How it works
| Cause | Example | Fix |
|---|---|---|
| Lighting | Trained in daylight, used at night | Add night, dusk and rainy photos |
| Angle and distance | Trained on front views, sees objects from above | Add photos from many angles and distances |
| Hidden objects | A person half behind a pillar is missed | Add partly hidden examples |
| Unusual examples | A cycle carrying a huge load is not recognised as a cycle | Collect rare cases on purpose |
| Wrong clue | Learns "snow means wolf" from the backgrounds | Vary the backgrounds; check what the model looks at |
| Never seen | A fruit model shown a shoe still names a fruit | Add an "other" group; show low confidence |
The model that learned the background
In a well-known teaching experiment, researchers trained a model to tell wolves from husky dogs. It scored well. Then they checked which parts of each photo the model was using, and found it was looking at the snow. Most of the wolf photos had snow in the background, so the model had learned "snow means wolf". It had found a pattern that worked on the training photos and meant nothing about the animals.
Try it yourself
Lab day and rainy day
A model was trained to spot potholes from photos taken on sunny afternoons.
What went wrong?
Match each failure to its most likely cause.
A model calls any animal on grass a 'cow'
A model says 'doctor' whenever it sees a white coat, even on a painter
A helmet detector works at noon but fails at dusk
A skin-check model trained on one skin tone does poorly on others
A fruit sorter calls a tennis ball a 'lemon'
A number-plate reader fails on plates covered in mud
Common mistakes
- Thinking a high test score proves the model works everywhere.
- Thinking the model sees objects the way people do. It sees patterns of pixels.
- Fixing errors by training longer on the same photos. The fix is usually different photos.
Check yourself
Summary
- Vision models fail when real images differ from their training images.
- Lighting, angle, hidden objects, rare cases and wrong clues are the common causes.
- The cure is wider, more varied data and testing in real conditions.
Finished reading?
Sources: Google Machine Learning Crash Course · AI4K12: Five Big Ideas in AI