Definition
Our air-quality model is accurate on average. But what if every monitoring station sits in a wealthy neighbourhood, so the alerts never reflect the smoky industrial area where most child labourers live? An ethics and risk review asks such questions before launch, not after harm.
An ethics and risk review is a structured check of how an AI system could cause harm and what will be done about it. It identifies stakeholders, lists potential harms (unfair outcomes, privacy breaches, safety failures, misuse, over-reliance and environmental cost), rates each by likelihood and severity, assigns mitigations and owners, and decides whether the system is acceptable to launch. Frameworks such as the US NIST AI Risk Management Framework (2023), organised around govern, map, measure and manage, and NITI Aayog's Responsible AI principles guide the process.
In simple words
Before a bridge opens, engineers check what could make it fail and how bad that would be, then strengthen the weak points. A risk review does the same for an AI system: imagine the ways it could hurt people, and fix the worst ones first.
Overview
- An ethics and risk review checks a project for possible harms before it is shared or used.
- It asks who could be harmed, how badly and how likely, and what can reduce the risk.
- Reviews cover data, model, deployment and misuse.
- A risk matrix rates each risk by likelihood and impact.
- Every high risk needs an owner and a mitigation plan.
- The review is repeated when the project changes or is used in a new setting.
Why it matters
Risk reviews catch problems while they are cheap to fix, protect the people affected, build trust with users and are increasingly expected by organisations and regulators. They also make a capstone project stronger: examiners value students who think about consequences.
Key terms
- Risk register
- A table listing each risk, its likelihood, severity, mitigation, owner and status.
- Likelihood and severity
- How probable a harm is and how serious it would be; together they set priority.
- Mitigation
- An action that reduces a risk's likelihood or severity.
- Automation bias
- People trusting an automated output too much and ignoring other evidence.
- Residual risk
- The risk that remains after mitigations, which someone must accept or reject.
- Go/no-go decision
- The documented decision whether to launch, pilot or stop.
Types
There are several types of risk to review in an AI project. Each needs its own questions.
1. Fairness and bias risks
The chance that the system works worse or decides unfairly for some groups.
- How it works: Check group representation in the data and compare metrics across groups; document gaps you cannot fix.
- Example: A speech app with much higher error rates for older speakers.
- When to use it: Review it in every project that makes predictions about people.
2. Privacy and security risks
The chance that personal data is exposed, misused or stolen.
- How it works: List personal fields, check consent, remove what is unneeded, secure storage and keep API keys out of shared code.
- Example: Accidentally pushing a CSV of survey responses with phone numbers to a public GitHub repository.
- When to use it: Review it before collecting data and again before publishing anything.
3. Safety and reliability risks
The chance that wrong outputs cause harm.
- How it works: Ask what happens when the model is wrong, test unusual inputs and add warnings, limits or human review.
- Example: A plant-disease app that tells a farmer a sick crop is healthy, so treatment is delayed.
- When to use it: Review it most carefully for health, safety, money and children.
4. Misuse and dual-use risks
The chance that someone uses the system for a harmful purpose.
- How it works: Imagine bad actors, limit features, add usage rules and avoid publishing details that mainly help misuse.
- Example: A face-matching tool built for finding lost pets being used to track people.
- When to use it: Review it whenever your system could identify, deceive or target people.
5. Social and environmental risks
Wider effects on jobs, trust, the environment and communities.
- How it works: Consider who gains and loses, energy use, over-reliance on AI and whether affected communities were consulted.
- Example: Choosing a smaller model to cut a project's energy use by 90% with little loss in accuracy.
- When to use it: Review it in the final report to show responsible thinking.
Explained in detail
A risk register for the air-quality project
| Risk | Likelihood / severity | Mitigation and owner |
|---|---|---|
| Missed alert on a dangerous day | Medium / high | Lower threshold to 80; publish live readings as a backup; data lead |
| Uneven coverage: few sensors in poorer areas | High / high | Report errors by area; state clearly where forecasts are less reliable; advocate for more sensors; project lead |
| Alerts only in English | High / medium | Send SMS in Hindi and the local language; communications lead |
| Over-reliance: schools ignore visible smog because no alert came | Low / high | Alert message reminds staff to use judgement and check the sky; project lead |
| Personal data from the principal survey | Low / medium | Collect only names of schools, not individuals; delete after the pilot; data lead |
Running the review
Hold the review as a meeting with the project team and, ideally, people from outside it, including someone who represents those affected. Walk through the system step by step, from data collection to the message a user receives, and ask at each step: who could be harmed, how, and how would we know? Check the five pillars: fairness (are errors worse for some areas or groups?), transparency (do users know how alerts are made and how reliable they are?), accountability (who answers if an alert is wrong?), privacy (what personal data is involved?) and safety (what happens if the data feed fails?). Rate each risk by likelihood and severity, focus mitigation on the high-high ones and write down the residual risk.
Deciding and monitoring
The review ends with a go/no-go decision: launch, run a limited pilot with extra monitoring, redesign, or stop. For our project a sensible decision is a one-month pilot with five schools, alerts in two languages, a clear note on which areas the forecast covers poorly and a weekly check of missed alerts. After launch, keep monitoring: errors by area, complaints, and whether schools act on alerts. Repeat the review when anything important changes, such as new data sources, a new model or a new group of users. Store the risk register with the project documentation so that examiners, partners and future team members can see what was considered.
How it works
- List stakeholders, including people affected but not using the system.
- Walk through each step of the system and brainstorm possible harms.
- Rate each risk by likelihood and severity and record it in a risk register.
- Assign mitigations and owners, and state the residual risk.
- Make a documented go/no-go decision and set up monitoring.
In code
The per-area numbers here are illustrative, chosen to show the kind of table a team should build from its own test results. The pattern they show is common in real monitoring networks: errors and missed alerts grow where sensors are scarce, and those areas often hold many people.
import pandas as pd
# Forecast errors by area of the city in the test year, and how many sensors each area has
areas = pd.DataFrame({
"area": ["Central", "University", "Old City", "Industrial East", "Riverside settlements"],
"sensors": [4, 2, 1, 1, 0],
"population_lakh": [6, 3, 8, 7, 5],
"mae": [7.9, 8.4, 11.6, 15.2, 18.9], # the last area is estimated from the nearest sensor
"missed_bad_days": [1, 1, 3, 6, 9],
})
areas["people_per_sensor_lakh"] = (areas["population_lakh"] / areas["sensors"].replace(0, float("nan"))).round(1)
print(areas.to_string(index=False))
worst = areas.sort_values("mae", ascending=False).iloc[0]
print(f"\nHighest error: {worst['area']} (MAE {worst['mae']}) with {worst['sensors']} sensors.")
share = areas.loc[areas["sensors"] <= 1, "population_lakh"].sum() / areas["population_lakh"].sum()
print(f"{share:.0%} of the population lives in areas with one sensor or none.")
risk = pd.DataFrame([
("Missed alert on a dangerous day", 2, 3), ("Uneven sensor coverage", 3, 3),
("Alerts only in English", 3, 2), ("Over-reliance on alerts", 1, 3), ("Survey personal data", 1, 2),
], columns=["risk", "likelihood", "severity"])
risk["priority"] = risk["likelihood"] * risk["severity"]
print("\nRisk register sorted by priority (likelihood x severity, each 1-3):")
print(risk.sort_values("priority", ascending=False).to_string(index=False))Output
area sensors population_lakh mae missed_bad_days people_per_sensor_lakh
Central 4 6 7.9 1 1.5
University 2 3 8.4 1 1.5
Old City 1 8 11.6 3 8.0
Industrial East 1 7 15.2 6 7.0
Riverside settlements 0 5 18.9 9 NaN
Highest error: Riverside settlements (MAE 18.9) with 0 sensors.
69% of the population lives in areas with one sensor or none.
Risk register sorted by priority (likelihood x severity, each 1-3):
risk likelihood severity priority
Uneven sensor coverage 3 3 9
Missed alert on a dangerous day 2 3 6
Alerts only in English 3 2 6
Over-reliance on alerts 1 3 3
Survey personal data 1 2 2Try it yourself
The neighbourhood the sensors missed
During the risk review, a community health worker joins the meeting.
Which pillar does each risk touch?
Classify each risk in the air-quality project.
Users are not told how accurate alerts are
The data feed fails and no alert is sent on a bad day
Survey responses store principals' phone numbers
Poorer areas have fewer sensors and larger errors
Common mistakes
- Holding the review after launch instead of before.
- Only the builders taking part, with no outside or affected voices.
- Listing risks without owners, mitigations or a decision.
Check yourself
Summary
- An ethics and risk review lists harms, rates likelihood and severity, assigns mitigations and owners and decides go or no-go.
- Check fairness, transparency, accountability, privacy and safety at every step, with affected people involved.
- Pilot carefully, monitor after launch and repeat the review when things change.
Finished reading?
Sources: NIST AI Risk Management Framework (AI RMF 1.0) · NITI Aayog: Responsible AI for All (2021)