Definition
Team Swachh's classifier works well on the six bins near the market. Would it work as well on the older, rusty bins in the poorer part of town? If it did not, which neighbourhoods would stay dirty?
A fairness and safety review is a structured check of an AI prototype before it is proposed for real use. It asks whether the system works equally well for all the groups and places it will serve, whether it protects privacy, what happens when it is wrong, whether people can understand and challenge its decisions, and whether any group could be harmed.
In simple words
Before a new bridge opens, inspectors check it carefully: is it strong at both ends, in rain and heat, for heavy trucks and for people walking? A fairness and safety review is the inspection for your AI idea, checking it works for everyone it is meant to serve.
Why it matters
An AI that works well on average can still fail the people who need it most, and a helpful tool can still invade privacy or be misused. Reviewing fairness and safety deliberately is what separates a responsible project from a merely clever one.
Key terms
- Fairness
- Working equally well for, and not disadvantaging, any group or place.
- Group-wise testing
- Measuring results separately for each group or setting.
- Feedback loop harm
- When an AI's mistakes cause a situation that makes its future mistakes worse.
- Transparency
- Telling people that AI is used, what it does and its limits.
- Contestability
- A way for people to question or appeal a decision the system made.
- Review checklist
- A set of questions used to check a system systematically.
Explained in detail
The review checklist
| Area | Question | Team Swachh's finding |
|---|---|---|
| Fairness | Does it work equally well for every area and group? | Only 70 per cent accurate on old rusty bins, versus 90 per cent on new ones |
| Access | Can everyone who should use it, use it? | Areas with fewer smartphones send fewer photos |
| Privacy | Does it collect only what is needed, and protect it? | Face filter works; no names collected |
| Errors | What happens when it is wrong, and who is affected? | Missed bins stay full; supervisor can still check manually |
| Transparency | Do users know it is AI and what it can't do? | Added a short note in Hindi and English |
| Misuse | Could anyone use it to cause harm? | Someone might send fake photos; supervisor confirms before sending trucks far |
The fairness problem they found
When the team tested their model on 20 photos of older bins from a poorer neighbourhood, accuracy fell to 70 per cent, and most errors were misses: rust and dents made full bins look unfamiliar. Combined with fewer smartphones in that area, this created a double disadvantage. Fewer photos would be sent, and the ones that were sent would more often be misread. The very neighbourhoods with the least resources would get the least benefit, and might even get worse service if trucks were redirected to areas the app flagged.
How they responded
- Collected 60 photos of older bins, with help from a shopkeeper in that area, and retrained: accuracy on old bins rose to 86 per cent.
- Proposed that trucks keep a minimum visit to every area, whatever the app says, so no area loses service.
- Added a phone-call reporting option through local shopkeepers.
- Recorded the remaining gap honestly in the proposal.
The feedback loop danger
The team spotted a subtle risk using their system map. If an area sends fewer reports, it gets fewer pickups; garbage builds up; residents give up reporting; the area gets even fewer pickups. AI systems that decide where resources go can create feedback loops like this, quietly deepening inequality. The minimum-visit rule was their safeguard. A good review looks not only at today's accuracy but at what the system would do to the world over time.
How it works
- Go through the checklist: fairness, access, privacy, errors, transparency and misuse.
- Test your prototype separately on each group or setting.
- Look for feedback loops that could deepen inequality over time.
- Fix what you can, and plan safeguards for the rest.
- Record remaining risks honestly for the proposal.
Try it yourself
Who does the model work for?
The model stays the same; only the mix of training examples changes. Slide and watch which group it serves well.
The neighbourhood the model could not see
Team Swachh runs a group-wise test.
Common mistakes
- Testing only on the easiest or most familiar setting.
- Ignoring how the system could change the world over time.
- Leaving remaining risks out of the proposal.
Check yourself
Summary
- Review fairness, access, privacy, errors, transparency and misuse with a checklist.
- Test by group and look for feedback loops that could deepen inequality.
- Fix what you can, add safeguards and report remaining risks honestly.
Finished reading?
Sources: UNESCO Recommendation on the Ethics of Artificial Intelligence · Digital Personal Data Protection Act, 2023 (MeitY)