I11 · Intermediate Capstone: An End-to-End AI Project

Ethics and Risk Review

An ethics and risk review is a structured check of how an AI system could cause harm and what will be done about it. It identifies stakeholders, lists potential harms (unfair outcomes, privacy breaches, safety failures, misuse, over-reliance and environmental cost), rates each by likelihood and severity, assigns mitigations and owners, and decides whether the system is acceptable to launch. Frameworks such as the US NIST AI Risk Management Framework (2023), organised around govern, map, measure and manage, and NITI Aayog's Responsible AI principles guide the process.

12 min Intermediate · Sublevel Advanced Jump to the activity

What you need: The earlier topics of this module.

Definition

Our air-quality model is accurate on average. But what if every monitoring station sits in a wealthy neighbourhood, so the alerts never reflect the smoky industrial area where most child labourers live? An ethics and risk review asks such questions before launch, not after harm.

An ethics and risk review is a structured check of how an AI system could cause harm and what will be done about it. It identifies stakeholders, lists potential harms (unfair outcomes, privacy breaches, safety failures, misuse, over-reliance and environmental cost), rates each by likelihood and severity, assigns mitigations and owners, and decides whether the system is acceptable to launch. Frameworks such as the US NIST AI Risk Management Framework (2023), organised around govern, map, measure and manage, and NITI Aayog's Responsible AI principles guide the process.

In simple words

Before a bridge opens, engineers check what could make it fail and how bad that would be, then strengthen the weak points. A risk review does the same for an AI system: imagine the ways it could hurt people, and fix the worst ones first.

Overview

  • An ethics and risk review checks a project for possible harms before it is shared or used.
  • It asks who could be harmed, how badly and how likely, and what can reduce the risk.
  • Reviews cover data, model, deployment and misuse.
  • A risk matrix rates each risk by likelihood and impact.
  • Every high risk needs an owner and a mitigation plan.
  • The review is repeated when the project changes or is used in a new setting.

Why it matters

Risk reviews catch problems while they are cheap to fix, protect the people affected, build trust with users and are increasingly expected by organisations and regulators. They also make a capstone project stronger: examiners value students who think about consequences.

Key terms

Risk register
A table listing each risk, its likelihood, severity, mitigation, owner and status.
Likelihood and severity
How probable a harm is and how serious it would be; together they set priority.
Mitigation
An action that reduces a risk's likelihood or severity.
Automation bias
People trusting an automated output too much and ignoring other evidence.
Residual risk
The risk that remains after mitigations, which someone must accept or reject.
Go/no-go decision
The documented decision whether to launch, pilot or stop.

Types

There are several types of risk to review in an AI project. Each needs its own questions.

1. Fairness and bias risks

The chance that the system works worse or decides unfairly for some groups.

  • How it works: Check group representation in the data and compare metrics across groups; document gaps you cannot fix.
  • Example: A speech app with much higher error rates for older speakers.
  • When to use it: Review it in every project that makes predictions about people.

2. Privacy and security risks

The chance that personal data is exposed, misused or stolen.

  • How it works: List personal fields, check consent, remove what is unneeded, secure storage and keep API keys out of shared code.
  • Example: Accidentally pushing a CSV of survey responses with phone numbers to a public GitHub repository.
  • When to use it: Review it before collecting data and again before publishing anything.

3. Safety and reliability risks

The chance that wrong outputs cause harm.

  • How it works: Ask what happens when the model is wrong, test unusual inputs and add warnings, limits or human review.
  • Example: A plant-disease app that tells a farmer a sick crop is healthy, so treatment is delayed.
  • When to use it: Review it most carefully for health, safety, money and children.

4. Misuse and dual-use risks

The chance that someone uses the system for a harmful purpose.

  • How it works: Imagine bad actors, limit features, add usage rules and avoid publishing details that mainly help misuse.
  • Example: A face-matching tool built for finding lost pets being used to track people.
  • When to use it: Review it whenever your system could identify, deceive or target people.

5. Social and environmental risks

Wider effects on jobs, trust, the environment and communities.

  • How it works: Consider who gains and loses, energy use, over-reliance on AI and whether affected communities were consulted.
  • Example: Choosing a smaller model to cut a project's energy use by 90% with little loss in accuracy.
  • When to use it: Review it in the final report to show responsible thinking.

Explained in detail

A risk register for the air-quality project

RiskLikelihood / severityMitigation and owner
Missed alert on a dangerous dayMedium / highLower threshold to 80; publish live readings as a backup; data lead
Uneven coverage: few sensors in poorer areasHigh / highReport errors by area; state clearly where forecasts are less reliable; advocate for more sensors; project lead
Alerts only in EnglishHigh / mediumSend SMS in Hindi and the local language; communications lead
Over-reliance: schools ignore visible smog because no alert cameLow / highAlert message reminds staff to use judgement and check the sky; project lead
Personal data from the principal surveyLow / mediumCollect only names of schools, not individuals; delete after the pilot; data lead

Running the review

Hold the review as a meeting with the project team and, ideally, people from outside it, including someone who represents those affected. Walk through the system step by step, from data collection to the message a user receives, and ask at each step: who could be harmed, how, and how would we know? Check the five pillars: fairness (are errors worse for some areas or groups?), transparency (do users know how alerts are made and how reliable they are?), accountability (who answers if an alert is wrong?), privacy (what personal data is involved?) and safety (what happens if the data feed fails?). Rate each risk by likelihood and severity, focus mitigation on the high-high ones and write down the residual risk.

Deciding and monitoring

The review ends with a go/no-go decision: launch, run a limited pilot with extra monitoring, redesign, or stop. For our project a sensible decision is a one-month pilot with five schools, alerts in two languages, a clear note on which areas the forecast covers poorly and a weekly check of missed alerts. After launch, keep monitoring: errors by area, complaints, and whether schools act on alerts. Repeat the review when anything important changes, such as new data sources, a new model or a new group of users. Store the risk register with the project documentation so that examiners, partners and future team members can see what was considered.

How it works

  1. List stakeholders, including people affected but not using the system.
  2. Walk through each step of the system and brainstorm possible harms.
  3. Rate each risk by likelihood and severity and record it in a risk register.
  4. Assign mitigations and owners, and state the residual risk.
  5. Make a documented go/no-go decision and set up monitoring.

In code

The per-area numbers here are illustrative, chosen to show the kind of table a team should build from its own test results. The pattern they show is common in real monitoring networks: errors and missed alerts grow where sensors are scarce, and those areas often hold many people.

risk_by_area.py
import pandas as pd

# Forecast errors by area of the city in the test year, and how many sensors each area has
areas = pd.DataFrame({
    "area": ["Central", "University", "Old City", "Industrial East", "Riverside settlements"],
    "sensors": [4, 2, 1, 1, 0],
    "population_lakh": [6, 3, 8, 7, 5],
    "mae": [7.9, 8.4, 11.6, 15.2, 18.9],          # the last area is estimated from the nearest sensor
    "missed_bad_days": [1, 1, 3, 6, 9],
})
areas["people_per_sensor_lakh"] = (areas["population_lakh"] / areas["sensors"].replace(0, float("nan"))).round(1)
print(areas.to_string(index=False))

worst = areas.sort_values("mae", ascending=False).iloc[0]
print(f"\nHighest error: {worst['area']} (MAE {worst['mae']}) with {worst['sensors']} sensors.")
share = areas.loc[areas["sensors"] <= 1, "population_lakh"].sum() / areas["population_lakh"].sum()
print(f"{share:.0%} of the population lives in areas with one sensor or none.")

risk = pd.DataFrame([
    ("Missed alert on a dangerous day", 2, 3), ("Uneven sensor coverage", 3, 3),
    ("Alerts only in English", 3, 2), ("Over-reliance on alerts", 1, 3), ("Survey personal data", 1, 2),
], columns=["risk", "likelihood", "severity"])
risk["priority"] = risk["likelihood"] * risk["severity"]
print("\nRisk register sorted by priority (likelihood x severity, each 1-3):")
print(risk.sort_values("priority", ascending=False).to_string(index=False))

Output

                 area  sensors  population_lakh  mae  missed_bad_days  people_per_sensor_lakh
              Central        4                6  7.9                1                     1.5
           University        2                3  8.4                1                     1.5
             Old City        1                8 11.6                3                     8.0
      Industrial East        1                7 15.2                6                     7.0
Riverside settlements        0                5 18.9                9                     NaN

Highest error: Riverside settlements (MAE 18.9) with 0 sensors.
69% of the population lives in areas with one sensor or none.

Risk register sorted by priority (likelihood x severity, each 1-3):
                           risk  likelihood  severity  priority
         Uneven sensor coverage           3         3         9
Missed alert on a dangerous day           2         3         6
         Alerts only in English           3         2         6
        Over-reliance on alerts           1         3         3
           Survey personal data           1         2         2

Try it yourself

The neighbourhood the sensors missed

During the risk review, a community health worker joins the meeting.

Which pillar does each risk touch?

Classify each risk in the air-quality project.

  • Users are not told how accurate alerts are

  • The data feed fails and no alert is sent on a bad day

  • Survey responses store principals' phone numbers

  • Poorer areas have fewer sensors and larger errors

Common mistakes

  • Holding the review after launch instead of before.
  • Only the builders taking part, with no outside or affected voices.
  • Listing risks without owners, mitigations or a decision.

Check yourself

Question 1

1. What does a risk register record?

Question 2

2. Schools ignoring visible smog because no alert came is an example of…

Question 3

3. Which risk should get attention first?

Summary

  • An ethics and risk review lists harms, rates likelihood and severity, assigns mitigations and owners and decides go or no-go.
  • Check fairness, transparency, accountability, privacy and safety at every step, with affected people involved.
  • Pilot carefully, monitor after launch and repeat the review when things change.

Finished reading?

Sources: NIST AI Risk Management Framework (AI RMF 1.0) · NITI Aayog: Responsible AI for All (2021)