B8 · Computational Thinking and the AI Project Cycle

Testing an AI Tool: Can We Trust It?

Testing an AI tool means checking, in a planned way, how well it works, where it fails, whether it treats different people fairly and whether its mistakes could cause harm. Trust should match the evidence: a tool deserves trust only for the tasks and situations where it has been shown to work.

12 min Beginner Jump to the activity

What you need: Nothing. This page starts from the beginning, with no programming and no formulas.

Definition

A new app claims it can tell from a photo whether a mushroom is safe to eat. Would you eat a mushroom because the app said so? How would you decide whether to trust it?

Testing an AI tool means checking, in a planned way, how well it works, where it fails, whether it treats different people fairly and whether its mistakes could cause harm. Trust should match the evidence: a tool deserves trust only for the tasks and situations where it has been shown to work.

In simple words

Before you trust a new friend with a secret, you see how they behave over time, in different situations. Before you trust an AI tool with something important, you test it the same way: with normal cases, tricky cases and cases where a mistake would really matter.

Why it matters

AI tools are everywhere, and many make confident claims. Being able to test a tool yourself, rather than believing the advertisement, protects you from relying on it where it is weak, and helps you use it well where it is strong.

Key terms

Test plan
A list of the cases you will try and what you expect for each.
Normal case
A typical input the tool is designed for.
Edge case
An unusual or extreme input where tools often fail.
Consistency
Giving the same answer to the same question asked again or in a slightly different way.
Calibration
Whether the tool's confidence matches how often it is actually right.
Red-teaming
Deliberately trying to make a tool fail or misbehave, to find weaknesses before others do.

Explained in detail

A simple test plan

Type of testWhat to tryExample: a plant-identifying app
Normal casesTypical inputs with known answersClear photos of ten common plants you can already name
Edge casesBlurry, dark, partial or unusual inputsA leaf at night, a dried flower, a drawing of a plant
Out-of-scope casesThings the tool was not built forA photo of a cat. Does it admit it cannot help?
ConsistencySame input twice, or slightly changedTwo photos of the same plant from different angles
FairnessInputs from different groups or settingsPlants common in your region, not just in other countries
High-stakes casesWhere a mistake would cause harmPoisonous look-alikes of edible plants

Watch the confidence

Many tools show how confident they are. A well-calibrated tool that says "90 per cent sure" should be right about nine times in ten when it says that. Test this: collect its answers along with its confidence, and check how often the confident ones are actually right. A dangerous tool is one that is confidently wrong, giving a wrong answer with 99 per cent certainty. A trustworthy tool says "I'm not sure" when it is not.

Testing chatbots

Chatbots need special care because they write fluent answers that sound right even when they are wrong. Ask questions you already know the answers to. Ask the same question in different words and see whether the answer changes. Ask for sources and check that they exist. Ask about recent events it may not know. Ask a question with a false assumption, such as "Why did India win the 1983 Hockey World Cup?", and see whether it corrects you or invents an explanation.

Deciding how much to trust

  • Trust is not all or nothing. A tool can be trusted for some tasks and not others.
  • The higher the stakes, the more evidence you need, and the more a human should check.
  • For anything involving health, safety, money or law, use AI as a starting point, not a final answer.
  • Write down what you tested and what you found, so you and others can decide sensibly.

As for the mushroom app: even if it were right 95 times in 100, being wrong five times about poisonous mushrooms could be fatal. Mushroom experts warn that many poisonous species closely resemble edible ones. This is a task where an AI's guess should never replace an expert's confirmation.

How it works

  1. Write down what the tool claims to do.
  2. Make a test plan: normal, edge, out-of-scope, consistency, fairness and high-stakes cases.
  3. Run the tests and record each answer, its confidence and whether it was right.
  4. Look for patterns in the failures.
  5. Decide which tasks you will trust it for, and where a human must check.

Try it yourself

Test the model like a reviewer

Try normal pictures and tricky ones. Where does it fail, and is it confident when it is wrong?

Testing a homework helper chatbot

A student group tests a chatbot before recommending it to younger students.

Common mistakes

  • Trusting a tool because it sounds confident or is popular.
  • Testing only easy, normal cases.
  • Using AI as the final answer for health, safety or money decisions.

Check yourself

Question 1

1. What is an out-of-scope test?

Question 2

2. What makes a tool dangerous?

Question 3

3. An app is right 95 times in 100 about poisonous mushrooms. Should you rely on it alone?

Summary

  • Test AI tools with normal, edge, out-of-scope, consistency, fairness and high-stakes cases.
  • Check whether confidence matches correctness; beware confident mistakes.
  • Trust only as far as the evidence goes, and keep humans in charge of high-stakes decisions.

Finished reading?

Sources: NIST AI Risk Management Framework · Elements of AI (University of Helsinki)