Definition
In 2018, researchers found that commercial gender-classification systems were wrong for under 1 per cent of lighter-skinned men but for up to about 35 per cent of darker-skinned women. Cameras are everywhere, so who vision systems fail, and who they watch, matters.
Responsible computer vision means designing, testing and deploying vision systems so that they are accurate for everyone they affect, respect privacy and consent, are used only for legitimate purposes with human oversight, and cannot easily be misused. Key issues include demographic bias, surveillance and facial recognition, consent for collecting images, deepfakes and manipulated media, and accountability when systems make mistakes.
In simple words
A camera that is wrong more often for some people treats them unfairly, and a camera that recognises everyone everywhere can take away the freedom to go about life unwatched. Responsible vision asks: is it fair, is it wanted and who is in control?
Overview
- Computer vision can harm people through bias, surveillance, misuse and errors.
- The Gender Shades study (2018) found far higher error rates for darker-skinned women in commercial face analysis.
- Images of people are personal data under laws such as India's Digital Personal Data Protection Act, 2023.
- Consent, data minimisation and clear purpose limit misuse.
- Models must be tested across skin tones, ages, genders and conditions.
- High-stakes decisions need human review and a way to appeal.
Why it matters
Vision systems are used in policing, airports, exams, hiring interviews, shops, schools and phones. Mistakes or misuse can lead to wrongful arrests, exclusion and loss of privacy, so builders and users must understand the risks and safeguards.
Key terms
- Disaggregated evaluation
- Measuring accuracy separately for each group of people rather than only overall.
- False match rate
- How often a face recognition system wrongly says two different people are the same person.
- Surveillance
- Systematic monitoring of people, which facial recognition can make automatic and large-scale.
- Consent
- A person's informed, free agreement to the collection and use of their images.
- Deepfake
- Synthetic image or video, made with AI, showing people doing or saying things they never did.
- Human oversight
- A trained person reviewing and able to overrule a system's decisions.
Types
Risks in computer vision fall into several types. Each needs its own safeguards.
1. Bias and unequal performance
Models working worse for some groups of people.
- How it works: Training data that under-represents some skin tones, ages or regions leads to higher error rates for them.
- Example: A skin-disease model trained mostly on light skin missing conditions on darker skin.
- When to use it: Measure performance per group, collect representative data and publish the results in a model card.
2. Surveillance and privacy
Tracking people without their knowledge or consent.
- How it works: Face recognition and person tracking in public spaces can identify and follow people across cameras.
- Example: Recognising protesters from CCTV footage.
- When to use it: Minimise data, blur faces where identity is not needed, follow the law and avoid mass identification of the public.
3. Deepfakes and manipulation
Synthetic or altered images and videos used to deceive.
- How it works: Generative models can create realistic fake faces and videos of real people.
- Example: Fake videos of public figures spread before an election.
- When to use it: Label synthetic media, use watermarks and detection tools, and never create media of real people without consent.
4. Safety-critical errors
Mistakes that can cause physical harm.
- How it works: Misreading a road sign or missing a pedestrian can be catastrophic, especially in rare conditions such as fog or night.
- Example: A driver-assistance system failing to detect a pedestrian at night.
- When to use it: Test extensively in realistic conditions, add fail-safes and keep humans in control.
5. Accountability and transparency
Being clear about what a system does and who answers for its decisions.
- How it works: Model cards, dataset documentation, logs and human review make it possible to explain and correct decisions.
- Example: A school attendance system publishing its accuracy and offering a manual correction process.
- When to use it: Use them for any vision system that affects people's rights, money or safety.
Explained in detail
Bias in vision systems
The Gender Shades study by Joy Buolamwini and Timnit Gebru tested three commercial gender classifiers in 2018. Error rates were at most 0.8 per cent for lighter-skinned men but reached 34.7 per cent for darker-skinned women. Later evaluations by the US National Institute of Standards and Technology (NIST) found that many face recognition algorithms had higher false match rates for some demographic groups, though the best algorithms reduced these gaps. Causes include unbalanced training data, cameras and exposure settings tuned for lighter skin and test sets that do not represent the people the system will meet. In India, with its great diversity of skin tones, languages and clothing, testing across groups is especially important.
Facial recognition and surveillance
Face recognition can verify a passenger at an airport gate, which many people choose for convenience, or identify anyone walking past a street camera, which they never agreed to. Wrongful arrests based on false facial-recognition matches have been reported in the United States. Several cities, including San Francisco in 2019, banned police use of the technology, and the European Union's AI Act, adopted in 2024, prohibits most real-time remote biometric identification in public spaces by police, with narrow exceptions. India's Digital Personal Data Protection Act, 2023, requires a lawful basis, usually consent, for processing personal data such as facial images, though it allows exemptions for certain government purposes. Policy debates in India continue over facial recognition in policing and public services.
A checklist for builders
| Question | Good practice |
|---|---|
| Is vision needed at all? | Prefer less intrusive options, such as counting without identifying |
| Who is in the data? | Collect varied, consented images; document them in a datasheet |
| Does it work for everyone? | Report accuracy and error rates for each group |
| Who decides? | Keep a trained person in the loop for consequential decisions |
| What is stored? | Store as little as possible, for as short a time as possible, securely |
| Can people object? | Post clear notices and provide a way to appeal or opt out |
- Deepfakes can harm reputations and spread misinformation; check sources, look for watermarks or provenance labels such as C2PA content credentials, and never create sexual or defamatory fakes of real people.
- Even well-meant uses, such as monitoring pupils' attention in class, can harm trust and wellbeing; involve the people affected before deploying.
How it works
- State the purpose and ask whether a less intrusive method would do.
- Collect images with consent and represent everyone the system will affect.
- Evaluate error rates for each group and fix large gaps before launch.
- Set up human review, notices, appeals and data deletion schedules.
- Monitor after launch and publish what the system does and how well.
In code
import pandas as pd
# Results of a face-verification test: each row is one attempt
data = pd.DataFrame({
"group": ["A"] * 400 + ["B"] * 400,
"same_person": [True] * 200 + [False] * 200 + [True] * 200 + [False] * 200,
"score": [0.90] * 190 + [0.55] * 10 + [0.20] * 196 + [0.70] * 4
+ [0.88] * 170 + [0.50] * 30 + [0.25] * 180 + [0.68] * 20,
})
for threshold in [0.6, 0.75]:
data["match"] = data["score"] >= threshold
rows = []
for group, g in data.groupby("group"):
false_match = (g["match"] & ~g["same_person"]).sum() / (~g["same_person"]).sum()
false_non_match = (~g["match"] & g["same_person"]).sum() / g["same_person"].sum()
rows.append({"group": group, "false match rate": round(false_match, 3), "false non-match rate": round(false_non_match, 3)})
print(f"Threshold {threshold}")
print(pd.DataFrame(rows).to_string(index=False))
overall = ((data["score"] >= 0.6) == data["same_person"]).mean()
print(f"Overall accuracy at 0.6 hides the gap: {overall:.3f}")Output
Threshold 0.6
group false match rate false non-match rate
A 0.02 0.05
B 0.10 0.15
Threshold 0.75
group false match rate false non-match rate
A 0.0 0.05
B 0.0 0.15
Overall accuracy at 0.6 hides the gap: 0.920Try it yourself
An attendance system rethought
A college plans face-recognition cameras to mark attendance automatically.
Lower risk or higher risk?
Classify each use of computer vision.
Identifying protesters from street cameras
Sorting recyclable waste on a conveyor
Blurring faces in published street photos
Counting cars at a junction without storing plates
Screening job interview videos for "personality"
Common mistakes
- Reporting only overall accuracy and hiding group differences.
- Collecting face images without consent because they are "public".
- Letting a system make consequential decisions with no human review or appeal.
Check yourself
Summary
- Vision systems can be less accurate for some groups; evaluate each group separately.
- Face recognition enables surveillance; laws such as India's DPDP Act and the EU AI Act set limits.
- Ask if vision is needed, get consent, minimise data, keep humans in charge and allow appeals.
Finished reading?
Sources: Buolamwini and Gebru (2018), Gender Shades · NIST Face Recognition Technology Evaluation (FRTE) · Digital Personal Data Protection Act, 2023 (MeitY)