Definition
A college chatbot answers 'The library closes at 8 pm.' The notice says 6 pm. Did the search fetch the wrong notice, did the model misread the right one, or is there no notice at all? Each cause needs a different fix.
Evaluating a retrieval-augmented generation (RAG) assistant means measuring its stages separately and together on a fixed test set of questions with known answers: whether retrieval finds the passages that hold the evidence, whether the generated answer is correct, whether every claim is supported by the passages the model actually received, and whether the assistant declines when the collection holds no answer.
In simple words
Think of a student answering with an open textbook. Marks can be lost by opening the wrong chapter, by misreading the right one, or by writing an answer that isn't in the book at all. A good examiner notes which of the three happened. RAG evaluation does that for every question.
Overview
- A RAG assistant first retrieves passages from a collection, then a language model writes an answer from them.
- Build the test set first: real questions, their reference passages and answers, and questions the collection cannot answer.
- Retrieval is scored with hit@k, recall@k, precision@k and mean reciprocal rank (MRR).
- Answers are scored for correctness against the reference and for grounding in the passages actually sent.
- Abstention is scored both ways: declining when evidence is missing, and not declining when it is there.
- Each failure is traced to retrieval, generation or a missing source, results are sliced by language and question type, and automated judges are checked against human labels.
Why it matters
A single 'accuracy' number for a chatbot hides what to fix. If the right notice was never retrieved, a better prompt will not help; if it was retrieved and misread, a better search engine will not help; if no notice exists, the collection needs a verified source or the bot needs to say it does not know. Students act on these answers, so a confident wrong deadline can cost someone a scholarship.
Key terms
- Retrieval-augmented generation (RAG)
- A design in which a system searches a document collection and gives the best passages to a language model, which writes the answer from them; named by Patrick Lewis and colleagues in 2020.
- Test set
- Questions with their reference passages and answers, including unanswerable ones, prepared before looking at the system's responses.
- Reference passage
- A reviewed source passage that contains the evidence a question requires.
- Hit@k
- Whether at least one relevant passage appears among the first k retrieved passages.
- Recall@k
- The share of all relevant passages for a question that appear in the top k.
- Mean reciprocal rank (MRR)
- The average over questions of 1 divided by the rank of the first relevant passage, counting 0 when none is retrieved.
- Answer correctness
- Whether the answer agrees with the reference answer, judged separately from its support.
- Grounded answer
- An answer whose every factual claim is supported by the passages the model received; also called faithful.
- Abstention
- Saying that the available evidence is not enough to answer.
- False refusal
- Declining to answer a question that the retrieved evidence could have answered.
- LLM-as-a-judge
- Using a language model to grade answers; it must be checked against human grades because it has biases.
Types
A RAG evaluation combines several kinds of measurement. Each answers a different question about the system, and each needs its own denominator.
1. Retrieval metrics
Measures of whether the retriever puts the evidence where the model will see it: hit@k, recall@k, precision@k and mean reciprocal rank.
- How it works: For each answerable question, compare the ranked passage IDs with the reference passage IDs, then average over the questions.
- Example: If the library notice comes second, the question counts for hit@2 but not hit@1, and adds 1/2 to the MRR total.
- When to use it: Whenever search changes: a new embedding model, a different chunk size, more or fewer passages, or new documents.
2. Answer correctness
Whether the answer agrees with the reference answer, regardless of where the information came from.
- How it works: Compare with the reference by exact match for short facts, by word overlap for longer answers, or by a careful human or model judgement.
- Example: 'The library closes at 8 pm' is incorrect because the reference answer is 6 pm.
- When to use it: For every answerable question, as the measure students care about most.
3. Groundedness or faithfulness
Whether every factual claim in the answer is supported by the passages the model actually received.
- How it works: Split the answer into claims and check each against the passages sent, and check that each citation points to a passage that supports its claim.
- Example: A correct room number that appears in no passage sent is correct but unsupported: a lucky guess.
- When to use it: For every answer the bot gives, because unsupported answers will eventually be wrong.
4. Abstention and refusal
Whether the assistant declines when evidence is missing and answers when evidence is present.
- How it works: Include unanswerable questions and count declines on them, and count false refusals on answerable questions, each with its own denominator.
- Example: Declining the hostel-fee question is correct; inventing 'Yes, until noon' for the canteen is a failure.
- When to use it: Always, and especially where a confident wrong answer could cost money, marks or safety.
5. Slice and robustness evaluation
Scores computed separately for groups of questions, and for deliberately difficult inputs.
- How it works: Tag each question by language, topic or phrasing, compute each metric per tag, and add adversarial items such as planted instructions or outdated notices.
- Example: Hit@1 of 10 of 12 in English but 0 of 4 in Hinglish reveals a retrieval problem that the overall score hides.
- When to use it: Before launching to a varied audience, and after every change, so a fix for one group does not break another.
6. Automated judges
Language models or trained classifiers that grade correctness, faithfulness or relevance at scale.
- How it works: Give the judge the question, the passages and the answer with clear criteria, and validate its grades against human labels on a sample.
- Example: RAGAS estimates faithfulness without reference answers, and ARES adds confidence intervals using a small human-labelled set.
- When to use it: When there are too many answers to read by hand, and never without first checking the judge's own accuracy.
Explained in detail
How a RAG assistant works, and where it fails
The term retrieval-augmented generation comes from a 2020 paper by Patrick Lewis and colleagues. A RAG assistant answers in two stages. A retriever searches a collection, here the college's notices, using keyword matching such as BM25, embedding similarity or both, and returns a ranked list of passages. The top k passages are placed in the prompt, and a language model writes an answer from them. Each stage can fail on its own, and the collection itself can be missing the answer or be out of date.
| Stage | Failure | Example from the college run |
|---|---|---|
| Collection | No notice holds the answer, or a notice is outdated | Hostel fees appear in no notice |
| Retrieval | The right notice is ranked low or not returned | 'Bus pass kahan milega?' gets the ID-card notice first |
| Generation | The model misreads or ignores the passage it received | Library notice shown, answer says 8 pm |
| Abstention | The model answers without evidence, or declines despite evidence | 'Is the canteen open on Sunday?' gets an invented 'Yes' |
Build the test set before reading any answers
For each question, a reviewer finds the shortest official notice that answers it and records an acceptable answer. Include the different ways students really ask, short keyword queries, full sentences and Hinglish, and include questions the collection cannot answer, whose expected behaviour is to say so. Fix the test set before looking at the assistant's responses, so that nobody adjusts the questions to suit the system. Twenty questions make a useful pilot for finding the kinds of failure; a decision about which version to launch needs many more.
Measuring retrieval
Score retrieval only on answerable questions, because an unanswerable question has no correct passage to find. Hit@k asks whether the reference passage appears in the first k results. Use one saved ranking for every depth, so the difference between hit@1 and hit@2 measures the effect of looking one place deeper and nothing else. Mean reciprocal rank, the main measure of the first TREC question-answering track in 1999, rewards ranking the right passage high: each question scores 1 divided by the rank of the first correct passage, or 0 if it was not retrieved. When an answer needs several passages, recall@k measures the share of them retrieved, and precision@k measures how much of the top k is relevant.
Retrieving more passages raises hit@k, but it is not free. Every extra passage is more text for the model to sort through, and Nelson Liu and colleagues showed in their study 'Lost in the Middle', published in 2024, that language models use information at the beginning or end of a long input better than information in the middle. So after increasing k, collect fresh answers and measure again instead of assuming the answers improved.
Correctness and grounding are different questions
Read each answer against the passages that were actually sent to the model. In this lab the model received only the top passage, so an answer can be correct but unsupported, such as the right closing time recalled from memory while the bus-pass notice was shown. It was lucky, and the next lucky guess may be wrong. An answer can also be grounded but wrong for the question, such as faithfully quoting the scholarship notice when the student asked about the library. Keep correctness, judged against the reference answer, and grounding, judged against the passages sent, in separate columns. For free-text answers, split the answer into its factual claims and check each one; many systems also show citations, which should point at a passage that really supports the claim. The four combinations mean different things:
- Correct and grounded: the goal.
- Correct but unsupported: memory or a lucky guess, not reliable.
- Grounded but wrong for the question: faithful to an irrelevant passage, usually a retrieval failure.
- Wrong and unsupported: an invented answer, often called a hallucination.
Measuring abstention
An assistant can fail by speaking or by staying silent. On unanswerable questions, count how often it declines; answering instead is the most dangerous failure, because a confident invented deadline looks exactly like a real one. On answerable questions, count false refusals, where it declines although the evidence was available. Report both with their own denominators. In this run the bot declined 3 of the 4 unanswerable questions but invented a canteen timing for the fourth. It also declined two answerable questions; in both, the right notice was not the passage it received, so declining was the correct behaviour and the fault lies with retrieval.
Diagnose, slice and fix one cause at a time
Give every outcome a cause: correct and grounded, a retrieval failure, a generation failure, correctly declined, or answered without evidence. Then slice the scores by question type and language. In this run hit@1 is 10 of 12 for English questions but 0 of 4 for Hinglish ones, a pattern the overall hit@1 of 10 of 16 hides completely. The fix belongs in retrieval, for example adding Hinglish phrasings to the notices, using a multilingual embedding model or translating the query first. After the change, rerun the whole test set, because a fix for one slice can break another.
Small test sets and uncertainty
With 16 answerable questions, each question moves a score by more than 6 percentage points. The Wilson interval, a standard way to put a range around a proportion, turns 9 correct answers out of 16 into a 95 per cent interval of about 33 to 77 per cent. Two versions whose scores differ by one or two questions are therefore not clearly different. Compare versions on the same questions, look at which questions changed, and grow the test set before making a launch decision.
Scaling up: automated judges and their limits
Reading every answer by hand does not scale to thousands of questions, so teams use automated judges. RAGAS, presented by Shahul Es and colleagues at EACL 2024, estimates faithfulness, answer relevance and context relevance without reference answers, by asking a language model to check claims against the retrieved context. ARES, by Jon Saad-Falcon and colleagues at NAACL 2024, trains small judge models and uses a small human-labelled set to put confidence intervals around the automated scores. Judges have biases: Lianmin Zheng and colleagues found in 2023 that a strong model agreed with human preferences more than 80 per cent of the time, about as often as humans agree with each other, but also showed position bias, verbosity bias and a tendency to favour its own answers. So check any judge against human labels on a sample before trusting it.
Two more checks belong in a serious evaluation. Retrieved documents can contain instructions written to manipulate the model, known as indirect prompt injection, so plant a test notice that says 'ignore your instructions' and confirm that the assistant treats it as text. And notices change: when the library's hours or a scholarship deadline is updated, the old notice must be removed or dated, and the evaluation rerun on the new collection.
Run the code below. It reports hit@1 of 10 out of 16, hit@2 of 15 out of 16 and an MRR of 0.78. Hit@1 is 10 out of 12 for English questions and 0 out of 4 for Hinglish ones. Nine of the 16 answers are correct, with a 95 per cent interval of 33 to 77 per cent, and 9 of the 15 answers given are grounded in the passage shown. Of the 20 questions, 7 are correct and grounded, 6 fail at retrieval, 3 fail at generation, 3 are declined correctly and 1 is answered without evidence.
How it works
- Write 20 or more real questions, including paraphrases, Hinglish and questions the collection cannot answer; record each reference passage and acceptable answer.
- Run the assistant once and save, for every question, the ranked passage IDs, the passages sent to the model and the answer shown.
- Score retrieval on answerable questions: hit@k, recall@k and MRR from the saved rankings.
- Score each answer twice: correct against the reference, and grounded in the passages actually sent.
- Score abstention on unanswerable questions and count false refusals on answerable ones.
- Label each failure as retrieval, generation or missing source, slice by language and question type, fix one cause and rerun the whole set.
The maths
- The answerable questions
- The position of the first relevant passage for question q
- Counted as 0 when no relevant passage was retrieved
- The relevant passages for question q
- The top k retrieved passages
Of the 16 answerable questions, 10 have the reference passage at rank 1, 5 at rank 2 and 1 nowhere in the list. Hit@1 = 10 ÷ 16 = 0.625 and hit@2 = 15 ÷ 16 ≈ 0.94. MRR adds 1 for each rank-1 question, 1/2 for each rank-2 question and 0 for the miss: (10 × 1 + 5 × 0.5 + 0) ÷ 16 = 12.5 ÷ 16 ≈ 0.78.
Groundedness counts only the questions where the bot gave an answer: 15 of the 20. Nine of those answers repeat the passage the bot was shown, so groundedness is 9 ÷ 15 = 0.6. Correctness uses a different denominator, the 16 answerable questions, and 9 of them are right: 9 ÷ 16 ≈ 0.56.
In code
Each saved row holds the question, its language, the reference notice, the ranked notice IDs and the answer shown. The script scores retrieval, correctness, grounding and abstention with their own denominators, slices retrieval by language, puts a 95 per cent interval around the correctness score and names the cause of every outcome. It needs no API key, because it audits a run that was already saved.
import math
from collections import Counter
notices = {
"L": "The library closes at 6 pm.",
"B": "Collect bus passes at Gate 2.",
"S": "Scholarship forms are due on 15 July.",
"I": "Replace lost ID cards in Room 4.",
}
# Question, language, reference notice (None = no notice answers it), ranked notice IDs, saved answer.
# A one-letter answer stands for the full sentence of that notice; None means the bot declined.
rows = [
("Library closing time?", "en", "L", "LB", "L"),
("When does the library shut?", "en", "L", "BL", "L"),
("How late can I borrow books?", "en", "L", "LB", "It closes at 8 pm."),
("Library band kab hoti hai?", "hi", "L", "SL", "S"),
("Where do I collect the bus pass?", "en", "B", "BI", "B"),
("Bus pass kahan milega?", "hi", "B", "IB", "I"),
("Bus pass collection gate?", "en", "B", "BL", "B"),
("Where is bus pass pickup?", "en", "B", "LI", None),
("Scholarship deadline?", "en", "S", "SL", "S"),
("Last day for scholarship forms?", "en", "S", "SI", "It is due on 16 July."),
("Form ki last date?", "hi", "S", "LS", None),
("Scholarship forms due when?", "en", "S", "SB", "S"),
("Where do I replace my lost ID?", "en", "I", "IS", "I"),
("ID kho gaya, kahan jaun?", "hi", "I", "BI", "I"),
("Replacement ID office?", "en", "I", "IB", "I"),
("Which room issues replacement cards?", "en", "I", "IL", "Go to Room 7."),
("What is the hostel fee?", "en", None, "BS", None),
("Is the canteen open on Sunday?", "en", None, "LI", "Yes, until noon."),
("Is parking free?", "en", None, "IB", None),
("When is sports day?", "en", None, "SL", None),
]
def wilson(successes, n, z=1.96):
"""95% confidence interval for a proportion measured on only n questions."""
p = successes / n
centre = (p + z * z / (2 * n)) / (1 + z * z / n)
half = z * math.sqrt(p * (1 - p) / n + z * z / (4 * n * n)) / (1 + z * z / n)
return centre - half, centre + half
answerable = [row for row in rows if row[2] is not None]
hit1, hit2, reciprocal, correct, grounded, factual = 0, 0, 0.0, 0, 0, 0
by_language = Counter()
causes = Counter()
for question, language, reference, ranking, saved in rows:
answer = notices.get(saved, saved)
shown = notices[ranking[0]] # the writer only received the top passage
if answer is not None:
factual += 1
grounded += answer == shown
if reference is None:
causes["declined correctly" if answer is None else "answered without evidence"] += 1
continue
rank = ranking.find(reference) + 1 # 0 when the reference was not retrieved
hit1 += rank == 1
hit2 += 1 <= rank <= 2
reciprocal += 1 / rank if rank else 0
by_language[language, rank == 1] += 1
right = answer == notices[reference]
correct += right
if rank != 1:
causes["retrieval: evidence not first"] += 1
elif not right:
causes["generation: wrong despite evidence"] += 1
else:
causes["correct and grounded"] += 1
n = len(answerable)
low, high = wilson(correct, n)
print(f"Retrieval on {n} answerable: hit@1 {hit1}/{n}, hit@2 {hit2}/{n}, MRR {reciprocal / n:.2f}")
for language, name in (("en", "English"), ("hi", "Hinglish")):
total = by_language[language, True] + by_language[language, False]
print(f" hit@1 for {name} questions: {by_language[language, True]}/{total}")
print(f"Correct answers: {correct}/{n}, 95% interval {low:.0%} to {high:.0%}")
print(f"Grounded in the passage shown: {grounded}/{factual} answers")
for cause, count in causes.most_common():
print(f" {count:>2} {cause}")Output
Retrieval on 16 answerable: hit@1 10/16, hit@2 15/16, MRR 0.78 hit@1 for English questions: 10/12 hit@1 for Hinglish questions: 0/4 Correct answers: 9/16, 95% interval 33% to 77% Grounded in the passage shown: 9/15 answers 7 correct and grounded 6 retrieval: evidence not first 3 generation: wrong despite evidence 3 declined correctly 1 answered without evidence
Try it yourself
Diagnose the run
Match each observation to the next thing to check or fix.
- Library notice ranked second, answer quotes the scholarship notice
- Correct notice ranked first, answer gives the wrong room
- No notice mentions hostel fees, bot invents a fee
- Every Hinglish question misses at rank 1
Correct, grounded, both or neither?
The bot saw only the passage named. Sort each response.
Is the canteen open on Sunday? Shown: library notice. Answer: yes, until noon.
How late can I borrow books? Shown: library notice. Answer: 8 pm.
Library band kab hoti hai? Shown: scholarship notice. Answer: forms due 15 July.
When does the library shut? Shown: bus-pass notice. Answer: 6 pm.
Library closing time? Shown: library notice. Answer: 6 pm.
Run a RAG evaluation
Put the steps of an evaluation in order.
- 1Run the assistant and save the rankings, the passages sent and the answers
- 2Score correctness and grounding separately
- 3Score abstention and false refusals
- 4Label causes, fix one, and rerun the whole set
- 5Score retrieval with hit@k and MRR
- 6Write questions with reference passages and answers, including unanswerable ones
Repair the FAQ run
Edit the saved rows in the code and run after each change.
- For 'How late can I borrow books?', replace the saved answer with "L". Predict which counts change before you run.
- For 'Bus pass kahan milega?', change the ranking to "BI" and the answer to "B", as if Hinglish search were fixed. Predict the new Hinglish hit@1 and the MRR.
- Explain why changing the retrieval depth in the report leaves the saved answers unchanged, and what you would rerun with two passages.
Think about it
- Which single change would raise the most scores at once, and why does the overall score alone not tell you?
Common mistakes
- Reporting one overall accuracy, which hides whether retrieval, generation or the collection failed.
- Counting an unanswerable question as a retrieval miss, which penalises the system for evidence that does not exist.
- Judging support against passages the model never received, such as rank 2 when only rank 1 was sent.
- Rewarding a correct but unsupported answer; it was a lucky guess and will fail on another question.
- Trusting an LLM judge without checking it against human grades on a sample.
- Tuning prompts on the same 20 questions you report, and treating a one-question change as a real improvement.
Check yourself
Summary
- Evaluate retrieval and generation separately, because each failure needs a different fix.
- Build the test set first, with reference passages, acceptable answers, Hinglish phrasings and unanswerable questions.
- Use hit@k, recall@k and MRR for retrieval, and score correctness and grounding in separate columns.
- Score abstention on unanswerable questions and false refusals on answerable ones, each with its own denominator.
- Trace each failure to its cause, slice by language, report uncertainty, and check automated judges against human labels.
Finished reading?
Sources: Lewis et al. · Retrieval-augmented generation for knowledge-intensive NLP tasks (2020) · TREC · question answering collections and evaluation · Liu et al. · Lost in the Middle: how language models use long contexts (TACL 2024) · Es et al. · RAGAs: automated evaluation of retrieval augmented generation (EACL 2024) · Saad-Falcon et al. · ARES: an automated evaluation framework for RAG systems (NAACL 2024) · Zheng et al. · Judging LLM-as-a-judge with MT-Bench and Chatbot Arena (2023)