Definition
A prompt looks perfect on the three examples you tried, so you launch it. A week later users report wrong answers on a kind of question you never tested. Prompts need tests, just like code.
Testing prompts, often called evaluation or "evals", means running a prompt on a fixed, representative set of test inputs and measuring the outputs against clear criteria. Checks can be exact (the answer equals the expected label), rule-based (the output is valid JSON, under 100 words, mentions no prices), model-graded (another LLM scores the answer against a rubric) or human-graded. Improving a prompt then becomes a loop: change one thing, re-run the whole test set, compare scores and look at failures, keeping a record of versions.
In simple words
A teacher does not judge a new textbook by reading one page. They try it with the whole class for a few weeks and check test results. Evaluating a prompt is the same: try it on many varied cases and measure.
Overview
- Prompts should be tested like software, on a fixed set of example inputs.
- An evaluation set includes typical, tricky and adversarial cases with expected outputs or criteria.
- Change one thing at a time and compare scores, keeping a version history of prompts.
- Metrics can be exact matches, rubric scores from people or LLM-as-judge ratings.
- Testing should cover safety: prompt injection, harmful requests and privacy leaks.
- Model updates can change behaviour, so tests are rerun regularly.
Why it matters
LLM outputs vary and can fail in surprising ways. Without tests, changes that fix one problem silently break another. Evaluations let you improve with confidence, compare models and catch regressions before users do.
Key terms
- Test set
- A fixed collection of inputs, ideally with expected outputs, used to evaluate prompts.
- Rubric
- Written criteria describing what a good answer contains.
- LLM-as-judge
- Using a model to grade outputs against a rubric; it must itself be checked against human grades.
- Regression
- Something that used to work and breaks after a change.
- Edge case
- An unusual input, such as an empty message, mixed languages or an off-topic question.
- Prompt version
- A saved, labelled copy of a prompt so results can be compared and changes undone.
Types
There are several types of prompt testing and improvement methods. These are the main ones.
1. Reference-based (exact or fuzzy match) tests
Comparing outputs with known correct answers.
- How it works: For extraction and classification, compare fields exactly; for text, use similarity scores or keyword checks.
- Example: Checking that 95 of 100 test emails are routed to the right department.
- When to use it: Use them whenever a correct answer exists; they are cheap and repeatable.
2. Rubric-based human evaluation
People scoring outputs against criteria such as accuracy, helpfulness, tone and safety.
- How it works: A written rubric with examples keeps raters consistent; two raters per item reveal disagreement.
- Example: Teachers rating AI explanations of science topics on a 1–5 scale for correctness and clarity.
- When to use it: Use it for open-ended outputs and to calibrate automatic judges.
3. LLM-as-judge evaluation
Using another LLM to score or compare outputs.
- How it works: The judge receives the question, the answer and a rubric, and returns a score with reasons; pairwise comparisons are often more reliable than absolute scores.
- Example: Automatically checking 1,000 chatbot answers each night for faithfulness to the source documents.
- When to use it: Use it to scale evaluation, while checking its judgements against people because judges have biases.
4. A/B testing and online metrics
Comparing prompt versions with real users.
- How it works: Users are randomly shown version A or B, and outcomes such as task completion, ratings or escalation rates are compared statistically.
- Example: A new support prompt cutting escalations to human agents from 18% to 12%.
- When to use it: Use it after offline tests pass, to confirm real-world improvement.
5. Red-teaming and regression testing
Trying to break the prompt, and rerunning all tests after every change.
- How it works: Red-teamers try prompt injection, jailbreaks and requests for private data; a regression suite catches cases that a change breaks.
- Example: Finding that "ignore previous instructions" inside a pasted document made the bot reveal its system prompt.
- When to use it: Use them before every release and whenever the underlying model changes.
Explained in detail
Building a test set
Collect 30 to 200 inputs that represent real use: common cases, difficult cases and edge cases such as Hinglish, spelling mistakes, very long messages, empty messages, off-topic requests and attempts to make the model break its rules. For each input, write the expected answer or the criteria a good answer must meet. Keep the test set separate from the examples inside your prompt, otherwise the scores will be falsely high. Add every real failure reported by users to the set so it can never silently return.
Kinds of checks
| Check | Example | Best for |
|---|---|---|
| Exact match | Predicted category equals the expected category | Classification and extraction |
| Rule-based | Valid JSON; under 80 words; no phone numbers | Format, length, safety rules |
| LLM-as-judge | "Score 1–5: is the explanation correct and suitable for a 12-year-old?" | Open-ended answers at scale |
| Human review | Teachers rate 50 sampled answers | Final quality and checking the judge |
The improvement loop
Run the current prompt on the full test set and record the scores; this is your baseline. Read the failures and group them: missing context, wrong format, unclear instruction, an ambiguous category. Make one change aimed at the biggest group, such as adding an example or clarifying a rule, and re-run everything. Keep the change only if the overall score improves and no important case gets worse. Track cost and latency too, since a longer prompt that gains one per cent of accuracy may not be worth doubling the cost. When a new model version is released, re-run the same tests before switching. LLM-as-judge scores are convenient but can favour longer answers or their own style, so compare a sample of judge scores with human ratings before trusting them.
Evaluation continues after launch. Log real inputs and outputs, with consent and privacy protections, and sample some each week for review. Track simple signals such as how often users rephrase a question, ask for a human, or give a thumbs-down, because rising numbers often reveal a new kind of request. Turn the worst examples into new test cases. Over time the test set becomes a valuable asset that captures what your users actually need, and it lets you switch to a newer or cheaper model with confidence, because you can prove the new one performs at least as well on the cases that matter.
How it works
- Write a representative test set, including edge cases and past failures.
- Define checks: exact, rule-based, LLM-graded or human.
- Measure a baseline for the current prompt and model.
- Change one thing, re-run all tests and compare scores, cost and speed.
- Save each version and re-test whenever the model changes.
In code
# Two versions of a "model" stand in for an LLM, so the harness runs offline.
TESTS = [
("Light nahi hai subah se", "electricity"), ("Power cut every evening", "electricity"),
("Nal se ganda paani", "water"), ("No water supply since Monday", "water"),
("Kachra 3 din se nahi utha", "sanitation"), ("Drain overflowing near school", "sanitation"),
("Bijli ka bill galat aaya", "electricity"), ("Tanker nahi aaya", "water"),
("What is the capital of France?", "other"), ("", "other"),
]
def prompt_v1(msg): # English keywords only, no "other" category
m = msg.lower()
if "power" in m or "electric" in m: return "electricity"
if "water" in m: return "water"
return "sanitation"
def prompt_v2(msg): # adds Hinglish examples and an explicit "other" option
m = msg.lower()
if any(w in m for w in ["power", "electric", "light", "bijli"]): return "electricity"
if any(w in m for w in ["water", "paani", "nal", "tanker"]): return "water"
if any(w in m for w in ["kachra", "drain", "garbage", "sanitation"]): return "sanitation"
return "other"
for name, model in [("v1", prompt_v1), ("v2", prompt_v2)]:
results = [(msg, exp, model(msg)) for msg, exp in TESTS]
score = sum(exp == got for _, exp, got in results) / len(results)
print(f"Prompt {name}: {score:.0%} correct")
for msg, exp, got in results:
if exp != got:
print(f" FAIL {msg!r:<34} expected {exp:<12} got {got}")Output
Prompt v1: 40% correct FAIL 'Light nahi hai subah se' expected electricity got sanitation FAIL 'Nal se ganda paani' expected water got sanitation FAIL 'Bijli ka bill galat aaya' expected electricity got sanitation FAIL 'Tanker nahi aaya' expected water got sanitation FAIL 'What is the capital of France?' expected other got sanitation FAIL '' expected other got sanitation Prompt v2: 100% correct
Try it yourself
The update that broke Hinglish
A team improves its support bot's prompt to give shorter answers.
The prompt improvement loop
Put the steps in order.
- 1Make one targeted change
- 2Re-run all tests and compare
- 3Read and group the failures
- 4Build a representative test set
- 5Run the current prompt to get a baseline
- 6Save the version if it is better overall
Common mistakes
- Testing only on the examples already inside the prompt.
- Changing several things at once, so you cannot tell what helped.
- Trusting LLM-as-judge scores without comparing them with human ratings.
Check yourself
Summary
- Evaluate prompts on a fixed, representative test set with edge cases and past failures.
- Use exact, rule-based, LLM-graded and human checks as suits each criterion.
- Change one thing at a time, compare scores, cost and speed, and keep versions.
Finished reading?
Sources: Anthropic: prompt engineering overview · Jurafsky and Martin, Speech and Language Processing (3rd ed. draft): Large Language Models