Definition
In 2023 a lawyer in New York submitted a court filing citing six earlier cases. None of them existed: an AI chatbot had invented them, complete with realistic names and quotations. The judge fined the lawyers. This is a hallucination, and grounding is the main defence.
A hallucination is output from a generative model that is fluent and confident but false or unsupported, such as invented facts, citations, quotes, statistics or code functions. It happens because an LLM generates likely-sounding text rather than looking facts up, and it may lack the information, misremember it or fill gaps plausibly. Grounding means tying the model's answer to trusted sources supplied at run time, for example through retrieval-augmented generation (RAG), tool use such as search or a calculator, and instructions to answer only from the sources and cite them.
In simple words
A student who has not studied may write a confident-sounding answer full of made-up details. An open-book exam with the instruction "answer only from the book and give page numbers" makes such invention much harder. Grounding gives the model an open book.
Overview
- A hallucination is fluent, confident output that is false or not supported by the sources.
- LLMs predict likely text, not verified facts, so they can invent citations, numbers and events.
- Hallucinations are more likely for rare topics, recent events, precise numbers and long reasoning.
- Grounding ties answers to trusted sources such as retrieved documents, databases or tools.
- Asking for citations and allowing "I don't know" reduces invented answers.
- High-stakes outputs in health, law and finance must be checked by people.
Why it matters
Hallucinations are the biggest obstacle to using LLMs where accuracy matters: law, medicine, finance, education and government. Knowing why they happen and how to reduce and detect them is essential for anyone building or using these tools.
Key terms
- Hallucination
- Confident output that is false or not supported by any source.
- Grounding
- Basing answers on trusted information provided at run time.
- Retrieval-augmented generation (RAG)
- Retrieving relevant passages and giving them to the model to answer from.
- Citation
- A pointer to the exact source passage supporting a claim.
- Abstention
- The model saying it does not know or that the sources do not answer the question.
- Faithfulness
- Whether every claim in an answer is supported by the provided sources.
Types
There are several types of hallucination and several types of grounding technique to reduce them.
1. Factual (world-knowledge) hallucinations
False statements about the world, such as wrong dates, people or statistics.
- How it works: The model fills gaps in its knowledge with plausible-sounding text, especially for obscure or recent facts.
- Example: Stating the wrong year a bridge opened, with a made-up architect's name.
- When to use it: Watch for them in any factual answer; verify important claims with reliable sources.
2. Faithfulness (source-contradicting) hallucinations
Answers that contradict or go beyond the documents the model was given.
- How it works: Even with RAG, the model may mix in outside knowledge or misread a chunk.
- Example: A summary saying a policy allows refunds within 30 days when the document says 14.
- When to use it: Check them by comparing each claim with the cited passage, manually or with an LLM judge.
3. Fabricated citations and references
Invented sources, links, case names or paper titles.
- How it works: The model produces text in the format of a citation without retrieving a real source.
- Example: A lawyer in New York was sanctioned in 2023 for filing a brief citing cases invented by a chatbot.
- When to use it: Never trust a citation until you open it; require citations to come from retrieved documents.
4. Retrieval and tool grounding
Giving the model trusted information at answer time.
- How it works: RAG supplies relevant documents; tools such as calculators, search engines and databases provide exact, current values.
- Example: Calling the railway database for today's train timings instead of relying on memory.
- When to use it: Use them for any answer that depends on specific, private or changing facts.
5. Prompting, verification and abstention
Instructions and checks that make the model cautious and self-correcting.
- How it works: Prompts say to answer only from sources and to say "I don't know"; a second pass checks claims; low-confidence answers are routed to people.
- Example: A medical-information bot that says "Please consult a doctor" when its sources do not cover the question.
- When to use it: Use them together with grounding in every high-stakes application.
Explained in detail
Why hallucinations happen
An LLM is trained to produce probable continuations of text, not to check facts. When the answer is well represented in its training data, probable and true usually coincide. When it is not, for example a rare fact, a recent event after the knowledge cutoff, a precise number or an exact citation, the model still produces something that looks like a good answer, because it has seen countless real citations and numbers and can imitate their form. Training that rewards confident, complete answers can make this worse, while training that rewards admitting uncertainty helps. Long chains of reasoning can also compound small errors. Hallucination rates have fallen in newer models, but no model is free of them.
How to reduce them
| Technique | How it helps |
|---|---|
| Retrieval (RAG) | Supplies the actual source text, so the model copies facts instead of recalling them |
| Answer-only-from-sources instruction | Tells the model not to use outside knowledge |
| Permission to abstain | "If the sources do not say, reply 'not found'" reduces guessing |
| Citations | Each claim points to a passage that can be checked |
| Tools | A calculator, database query or search returns exact values |
| Verification step | A second pass or program checks claims against the sources |
Retrieval-augmented generation was described in a 2020 paper by Patrick Lewis and colleagues at Facebook AI Research. It is now the standard way to build chatbots over company documents, government schemes or textbooks. But grounding is not a guarantee: retrieval can return the wrong passage, the model can misread or over-generalise a correct one, and sources themselves can be outdated or wrong. Check faithfulness by verifying that each sentence in an answer is supported by a cited passage, sample answers regularly for human review, and design interfaces that show sources so users can check them. In high-stakes settings, treat AI output as a draft that a qualified person verifies.
Users can protect themselves too. Ask the model for its sources and open them; be most suspicious of precise details such as numbers, dates, quotations, legal sections and references, because these are the easiest to invent and the hardest to notice. Ask the same question in a different way and see whether the answer stays the same; inconsistent answers are a warning sign. For anything important, confirm with an official website, a textbook or a qualified person. Treat an LLM as a fast, helpful first draft or a study partner, not as an authority.
How it works
- Retrieve relevant, up-to-date passages from trusted sources.
- Instruct the model to answer only from them and cite each claim.
- Allow and encourage "not found" when the sources do not answer.
- Use tools for calculations, dates and database facts.
- Verify claims against the sources and keep humans reviewing important outputs.
In code
import re
sources = {
"S1": "PM-KISAN provides income support of Rs 6,000 per year to eligible farmer families, paid in three equal instalments.",
"S2": "Farmers must link their Aadhaar with the bank account to receive PM-KISAN instalments.",
}
answer = ("PM-KISAN gives eligible farmer families Rs 6,000 per year [S1]. "
"It is paid in three equal instalments [S1]. "
"Aadhaar must be linked to the bank account [S2]. "
"Farmers also get a free tractor every five years [S1].")
def words(text):
return set(re.findall(r"[a-z0-9,]+", text.lower())) - {"the", "a", "of", "to", "is", "it", "in", "per", "and", "also", "be", "must"}
for sentence in re.split(r"(?<=\.)\s+", answer.strip()):
cited = re.findall(r"\[(S\d)\]", sentence)
claim = re.sub(r"\[S\d\]", "", sentence)
if not cited:
print(f"NO CITATION {claim}")
continue
support = max(len(words(claim) & words(sources[c])) / max(1, len(words(claim))) for c in cited)
verdict = "supported " if support >= 0.6 else "UNSUPPORTED"
print(f"{verdict} ({support:.0%}) {claim.strip()}")Output
supported (78%) PM-KISAN gives eligible farmer families Rs 6,000 per year . supported (100%) It is paid in three equal instalments . supported (75%) Aadhaar must be linked to the bank account . UNSUPPORTED (0%) Farmers also get a free tractor every five years .
Try it yourself
A scheme helpdesk that cites its sources
A district office builds a chatbot to answer farmers' questions about government schemes.
Grounded or hallucinated?
Source: "The library opens at 9 a.m. and closes at 6 p.m. on weekdays."
"The library opens at 9 a.m. on weekdays."
"The library is open until 8 p.m. on Saturdays."
"Entry costs Rs 20."
"Weekday closing time is 6 p.m."
Common mistakes
- Trusting fluent, confident answers without checking sources.
- Assuming RAG eliminates all hallucinations.
- Not allowing the model to say "I don't know".
Check yourself
Summary
- Hallucinations are confident but false or unsupported outputs, from invented facts to fake citations.
- Ground answers with retrieval, tools, citations and permission to abstain.
- Verify faithfulness and keep qualified people reviewing high-stakes outputs.
Finished reading?
Sources: Lewis et al. (2020), Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks · Jurafsky and Martin, Speech and Language Processing (3rd ed. draft)