B8 · Computational Thinking and the AI Project Cycle

When a Model Stops Working Well

A model can stop working well when the data it sees in real use becomes different from the data it was trained and tested on. This change is often called data drift. Teams watch for it by testing the model on fresh examples from current conditions, and they decide in advance when a person should check the model's decisions.

10 min Beginner Jump to the activity

What you need: Nothing. This page starts from the beginning, with no programming and no formulas.

Definition

A waste sorter learned to read clean labels on bottles and boxes and worked well for months. Then the monsoon arrived, and now the labels are wet, torn and muddy. Is the sorter still as good as before?

A model can stop working well when the data it sees in real use becomes different from the data it was trained and tested on. This change is often called data drift. Teams watch for it by testing the model on fresh examples from current conditions, and they decide in advance when a person should check the model's decisions.

In simple words

A learner who practised only with neat printed question papers may struggle with a handwritten one. Nothing is wrong with what they know; the questions just look different. A model is the same: when what it sees changes, its performance can drop.

Why it matters

Models are tested once but used for months or years while the world changes: seasons, new products, new cameras, new habits. If no one checks, a model can quietly get worse and make many mistakes before anyone notices. Testing with fresh examples and planning human checks keeps a system trustworthy.

Key terms

Data drift
A change in the kind of data a model sees in real use, compared with its training data.
Fresh test examples
New examples collected from today's conditions, used to check the model again.
Monitoring
Regularly checking how a model is doing after it has been put to use.
Human check
A person reviewing a model's decision before it is acted on, for uncertain or important cases.
Retraining
Teaching the model again with new examples, including the new conditions.

Explained in detail

Why the sorter got worse

The sorter was trained and tested on photos of clean, dry labels, and that test showed it was right for most items. In the monsoon, labels become wet, wrinkled, torn and partly covered with mud. Some look nothing like the training photos. The model has never learned what a torn, muddy label looks like, so it makes more mistakes, often without any warning and sometimes with high confidence.

Test setCorrectWhat it means
50 clean, dry labels: the original test47Works well in the old conditions
50 wet or torn labels: a fresh test31Much worse in today's conditions

Choosing fresh test examples

To find out how the model is doing now, collect new test examples that match today's conditions: wet labels, torn labels, labels covered in mud and items photographed in the poor light at the collection point. Include the cases that are hardest and the ones that are most common. Do not reuse the old test photos, because they only tell you how the model did in the old conditions. Keep checking regularly, not just once, because conditions keep changing.

Deciding when a person should check

A good plan says, before problems appear, when a person must look. For example: a person checks any item the sorter is unsure about; a person checks every item of a new kind that the sorter was never trained on; and a person checks a sample every day, so the team notices if accuracy is falling. If mistakes are costly, such as sending batteries, which can be dangerous, to the wrong bin, human checks matter even more.

Fixing the model

The lasting fix is retraining: add many examples of wet, torn and muddy labels, with the correct answers, and train the model again. Then test it on fresh examples once more. This cycle of using, watching, testing and retraining is why the AI project cycle does not end at deployment.

Data drift is not rare. Shops change their packaging, phones get new cameras, habits shift with festivals and seasons, and new kinds of items appear. That is why teams that run AI systems keep a small set of fresh, recently labelled examples and check the model against it every week or every month.

How it works

  1. Notice what has changed in real use, such as weather, light or new kinds of items.
  2. Collect fresh test examples from current conditions.
  3. Test the model and compare with the original test.
  4. Set rules for human checks: unsure cases, new kinds of items and a daily sample.
  5. Retrain with new examples, then test again.

Worked example

On the original test of 50 clean labels, the sorter got 47 right. The team collects 50 fresh photos from the monsoon, with wet, torn and muddy labels, and now it gets only 31 right. The drop shows data drift. The team decides that a worker checks every item the sorter is unsure about and every item with a badly damaged label, and checks 20 random items each day. Meanwhile they collect 500 labelled monsoon photos, retrain the sorter and test it again on fresh examples.

Try it yourself

The monsoon sorter

Follow the team as it notices and fixes a drop in performance.

Is this a good fresh test example?

The sorter now works in the monsoon. Sort each example.

  • A torn box label half covered in mud

  • A wet, wrinkled bottle label photographed this week

  • A label photographed in the dim light at the collection point

  • A perfect studio photo of a brand-new bottle

  • The same clean label photos used in the original test

A new case: the attendance camera

Plan on paper.

  1. A face-recognition attendance camera was tested in bright morning light. In winter, learners arrive in caps and scarves, and the mornings are darker. What has changed?
  2. List three fresh test examples you would collect.
  3. Write two rules for when a teacher should check the camera's decision.

Think about it

  • What could go wrong for a learner if no one checks the camera?

Common mistakes

  • Assuming a model that passed its test once will work forever.
  • Testing again with the old test examples instead of fresh ones.
  • Having no plan for when a person should check.

Check yourself

Question 1

1. The sorter got 47 of 50 clean labels right but only 31 of 50 wet labels. What does this show?

Question 2

2. Which is the best fresh test example?

Question 3

3. When should a person check the sorter's decision?

Question 4

4. New case: a crop-disease app was trained on photos from one state. Farmers in a different region, with other crop varieties, start using it. What should happen first?

Summary

  • When real-world data changes, a model can get worse without warning: this is data drift.
  • Test regularly with fresh examples from today's conditions, not the old test set.
  • Plan human checks for unsure, new and important cases, and retrain with new examples.

Finished reading?