Definition
A waste sorter learned to read clean labels on bottles and boxes and worked well for months. Then the monsoon arrived, and now the labels are wet, torn and muddy. Is the sorter still as good as before?
A model can stop working well when the data it sees in real use becomes different from the data it was trained and tested on. This change is often called data drift. Teams watch for it by testing the model on fresh examples from current conditions, and they decide in advance when a person should check the model's decisions.
In simple words
A learner who practised only with neat printed question papers may struggle with a handwritten one. Nothing is wrong with what they know; the questions just look different. A model is the same: when what it sees changes, its performance can drop.
Why it matters
Models are tested once but used for months or years while the world changes: seasons, new products, new cameras, new habits. If no one checks, a model can quietly get worse and make many mistakes before anyone notices. Testing with fresh examples and planning human checks keeps a system trustworthy.
Key terms
- Data drift
- A change in the kind of data a model sees in real use, compared with its training data.
- Fresh test examples
- New examples collected from today's conditions, used to check the model again.
- Monitoring
- Regularly checking how a model is doing after it has been put to use.
- Human check
- A person reviewing a model's decision before it is acted on, for uncertain or important cases.
- Retraining
- Teaching the model again with new examples, including the new conditions.
Explained in detail
Why the sorter got worse
The sorter was trained and tested on photos of clean, dry labels, and that test showed it was right for most items. In the monsoon, labels become wet, wrinkled, torn and partly covered with mud. Some look nothing like the training photos. The model has never learned what a torn, muddy label looks like, so it makes more mistakes, often without any warning and sometimes with high confidence.
| Test set | Correct | What it means |
|---|---|---|
| 50 clean, dry labels: the original test | 47 | Works well in the old conditions |
| 50 wet or torn labels: a fresh test | 31 | Much worse in today's conditions |
Choosing fresh test examples
To find out how the model is doing now, collect new test examples that match today's conditions: wet labels, torn labels, labels covered in mud and items photographed in the poor light at the collection point. Include the cases that are hardest and the ones that are most common. Do not reuse the old test photos, because they only tell you how the model did in the old conditions. Keep checking regularly, not just once, because conditions keep changing.
Deciding when a person should check
A good plan says, before problems appear, when a person must look. For example: a person checks any item the sorter is unsure about; a person checks every item of a new kind that the sorter was never trained on; and a person checks a sample every day, so the team notices if accuracy is falling. If mistakes are costly, such as sending batteries, which can be dangerous, to the wrong bin, human checks matter even more.
Fixing the model
The lasting fix is retraining: add many examples of wet, torn and muddy labels, with the correct answers, and train the model again. Then test it on fresh examples once more. This cycle of using, watching, testing and retraining is why the AI project cycle does not end at deployment.
Data drift is not rare. Shops change their packaging, phones get new cameras, habits shift with festivals and seasons, and new kinds of items appear. That is why teams that run AI systems keep a small set of fresh, recently labelled examples and check the model against it every week or every month.
How it works
- Notice what has changed in real use, such as weather, light or new kinds of items.
- Collect fresh test examples from current conditions.
- Test the model and compare with the original test.
- Set rules for human checks: unsure cases, new kinds of items and a daily sample.
- Retrain with new examples, then test again.
Worked example
On the original test of 50 clean labels, the sorter got 47 right. The team collects 50 fresh photos from the monsoon, with wet, torn and muddy labels, and now it gets only 31 right. The drop shows data drift. The team decides that a worker checks every item the sorter is unsure about and every item with a badly damaged label, and checks 20 random items each day. Meanwhile they collect 500 labelled monsoon photos, retrain the sorter and test it again on fresh examples.
Try it yourself
The monsoon sorter
Follow the team as it notices and fixes a drop in performance.
Is this a good fresh test example?
The sorter now works in the monsoon. Sort each example.
A torn box label half covered in mud
A wet, wrinkled bottle label photographed this week
A label photographed in the dim light at the collection point
A perfect studio photo of a brand-new bottle
The same clean label photos used in the original test
A new case: the attendance camera
Plan on paper.
- A face-recognition attendance camera was tested in bright morning light. In winter, learners arrive in caps and scarves, and the mornings are darker. What has changed?
- List three fresh test examples you would collect.
- Write two rules for when a teacher should check the camera's decision.
Think about it
- What could go wrong for a learner if no one checks the camera?
Common mistakes
- Assuming a model that passed its test once will work forever.
- Testing again with the old test examples instead of fresh ones.
- Having no plan for when a person should check.
Check yourself
Summary
- When real-world data changes, a model can get worse without warning: this is data drift.
- Test regularly with fresh examples from today's conditions, not the old test set.
- Plan human checks for unsure, new and important cases, and retrain with new examples.
Finished reading?