I10 · Responsible AI in Practice

Moral Dilemmas and AI

A moral dilemma is a situation in which every available choice violates some important value, so there is no clearly right answer. AI raises dilemmas in two ways: designers must decide in advance how a system will act in hard cases, such as collisions, triage or content removal, and AI changes who decides, how fast and at what scale. Classic thought experiments such as the trolley problem help clarify values, but most real AI dilemmas are less dramatic and more common: trading accuracy against fairness, safety against freedom, or personalisation against privacy.

10 min Intermediate · Sublevel Advanced Jump to the activity

What you need: The earlier topics of this module.

Definition

A self-driving car's brakes fail. It can stay on course and hit three pedestrians, or swerve and hit one. Millions of people around the world have answered this question online, and their answers differed by country, age and culture. Should a machine make this choice at all?

A moral dilemma is a situation in which every available choice violates some important value, so there is no clearly right answer. AI raises dilemmas in two ways: designers must decide in advance how a system will act in hard cases, such as collisions, triage or content removal, and AI changes who decides, how fast and at what scale. Classic thought experiments such as the trolley problem help clarify values, but most real AI dilemmas are less dramatic and more common: trading accuracy against fairness, safety against freedom, or personalisation against privacy.

In simple words

Sometimes every option has a cost. A dilemma is when you must choose anyway. With AI, these choices are made once by designers and then repeated millions of times, so they deserve careful thought and open discussion.

Overview

  • A moral dilemma is a situation where every option breaks some value or causes some harm.
  • AI raises new dilemmas because machines now make or support decisions about people.
  • The "trolley problem" is often used to discuss self-driving cars.
  • MIT's Moral Machine experiment collected about 40 million decisions from 233 countries and territories (published 2018).
  • Real AI dilemmas are usually about everyday trade-offs: privacy versus safety, accuracy versus fairness.
  • Structured reasoning and diverse voices help reach decisions people can accept.

Why it matters

Thinking through dilemmas helps you notice hidden value choices in everyday AI design, explain your decisions to others and recognise when a decision should be left to people rather than automated.

Key terms

Moral dilemma
A choice where every option sacrifices something of value.
Trolley problem
A thought experiment about whether to divert harm from many people to fewer.
Value alignment
Making an AI system's behaviour match human values and intentions.
Moral pluralism
The fact that reasonable people and cultures hold different values.
Meaningful human control
Humans retain real understanding and authority over important decisions.
Dual use
Technology that can be used both to help and to harm.

Types

AI moral dilemmas fall into several types. Recognising the type helps choose how to reason about it.

1. Life-and-death trade-offs

Choices where an AI system's action could save some lives while risking others.

  • How it works: Designers set rules in advance, such as minimising total harm and never targeting a group, because there is no time to decide in the moment.
  • Example: A self-driving car that cannot avoid a crash and must choose how to steer.
  • When to use it: Use these cases to discuss values publicly; real systems mostly focus on avoiding such situations.

2. Privacy versus safety or security

Dilemmas where more monitoring could prevent harm but intrudes on people's lives.

  • How it works: Weigh the seriousness of the risk, the intrusion, alternatives and safeguards such as time limits and oversight.
  • Example: Using face recognition at a railway station to find missing children versus tracking every traveller.
  • When to use it: Use proportionality: choose the least intrusive option that achieves the goal.

3. Accuracy versus fairness

Cases where the most accurate model gives worse results for some groups.

  • How it works: Measure performance per group, test fairness constraints and decide openly how much overall accuracy to trade for equity.
  • Example: A hiring model that is 2% less accurate overall after removing a strong bias against women.
  • When to use it: Use it whenever groups have different base rates or data quality.

4. Autonomy versus helpfulness

Dilemmas about how much an AI should nudge, persuade or decide for people.

  • How it works: Respect informed choice: show options and reasons, avoid manipulation and let people override suggestions.
  • Example: A health app nudging a user to exercise versus sending constant guilt-inducing alerts.
  • When to use it: Use it in recommendation, health, finance and education apps that shape behaviour.

5. Responsibility and automation

Dilemmas about who is to blame and when humans should stay in the loop.

  • How it works: Map who decides at each step, decide which decisions need human approval and keep humans able to understand and override.
  • Example: Whether a doctor, a hospital or a software company is responsible when an AI misses a tumour.
  • When to use it: Use it for high-stakes systems where errors seriously harm people.

Explained in detail

The Moral Machine experiment

Researchers at MIT built an online game, the Moral Machine, that presented variations of the self-driving dilemma: spare passengers or pedestrians, more lives or fewer, young or old, people crossing legally or illegally. The study, published in Nature in 2018, collected about 40 million decisions from people in 233 countries and territories. Some preferences were widely shared, such as sparing more lives and sparing humans over animals, but others varied by culture; for example, the preference for sparing the young over the old was weaker in several East Asian countries. The lesson is not that cars should follow a popular vote, but that value choices built into AI are contested and should be made openly, with public input and legal limits, not quietly by engineers.

Everyday dilemmas in AI

SituationValues in tension
Content moderation on a social platformFree expression versus protection from hate and misinformation
Predicting which students may drop outEarly help versus labelling and stigma
Facial recognition to find missing childrenChild safety versus mass surveillance of everyone
Medical AI that is accurate overall but weaker for one groupBenefit to many now versus fairness to all
Releasing a powerful open modelAccess and innovation versus possible misuse

Handling dilemmas responsibly

There is rarely a formula, but there is a good process. Name the values in tension explicitly. Ask the people affected, especially those with least power. Look for creative options that reduce the conflict, such as using face matching only on a court-approved list and deleting all other data. Prefer reversible decisions and keep a human in the loop where the stakes are high, which is often called meaningful human control. Write down the decision, the reasons and the evidence that would change it. Finally, recognise that some decisions, such as whether to use lethal force, many argue should not be delegated to machines at all; international discussions on autonomous weapons at the United Nations continue for this reason.

Dilemmas also arise in how AI products are designed for attention. Recommendation systems that maximise watch time can push sensational or harmful content, especially to young people, because engagement and wellbeing are not the same goal. Designers face a real choice between metrics that grow the business and metrics that protect users. Naming that conflict openly is the first step towards better choices.

How it works

  1. Describe the dilemma and every option, including doing nothing.
  2. List the values and the people affected by each option.
  3. Look for options that reduce the conflict.
  4. Decide who should make the final call, and keep humans in control of high-stakes cases.
  5. Document the reasoning and review it as evidence and values change.

Try it yourself

Finding missing children

A state police department wants facial recognition on railway-station cameras to find missing children.

Should a human make the final decision?

Decide for each case.

  • Sorting emails into folders

  • Denying someone a welfare benefit

  • Recommending a song

  • Removing a journalist's post as "misinformation"

  • Deciding whether a patient needs surgery

Common mistakes

  • Assuming there is always one technically correct answer.
  • Letting engineers settle value questions alone and quietly.
  • Automating high-stakes, irreversible decisions without human control.

Check yourself

Question 1

1. What did the Moral Machine experiment show?

Question 2

2. Which is an everyday AI dilemma?

Question 3

3. "Meaningful human control" means…

Summary

  • Moral dilemmas force a choice between important values; AI makes such choices at scale.
  • Thought experiments clarify values; everyday dilemmas involve fairness, safety, privacy and expression.
  • Handle them openly: involve those affected, reduce conflicts, keep humans in control and document reasons.

Finished reading?

Sources: Awad et al. (2018), The Moral Machine experiment, Nature · UNESCO Recommendation on the Ethics of Artificial Intelligence (2021)