Definition
A chatbot gives you the title, author and page number of a book that supports your argument. The book does not exist. How can that happen?
A hallucination is content produced by a generative AI tool that is false or invented, but is presented fluently and confidently as if it were true. It happens because the model generates likely-sounding text and has no built-in way to check facts.
In simple words
Imagine a student who has been taught never to leave an answer blank. Faced with a question they cannot answer, they write something that sounds right, in neat handwriting, with a date and a name for good measure. They are not trying to cheat. Producing an answer is simply what they do. A language model behaves like that student: it always produces text, and text that sounds right is not always text that is right.
Why it matters
Hallucination is the single most important weakness of text-generating AI to understand, because it is invisible. A hallucinated answer looks exactly like a correct one. People have lost court cases, failed assignments and spread false information by trusting one.
Key terms
- Hallucination
- False or invented content that an AI tool presents as fact.
- Fabrication
- Something made up, such as a source, a quotation or a statistic that does not exist.
- Plausible
- Sounding reasonable and likely, whether or not it is true.
- Citation
- A reference to the source of a piece of information.
- Verify
- To check that something is true using a reliable, independent source.
- Grounding
- Connecting a model to real documents or search results so that its answers are based on them.
Explained in detail
Why a language model invents things
A language model is trained to continue text in the most likely way. It has learned the shape of a good answer extremely well: a confident opening, supporting details, a source at the end. When it knows the subject, the details it fills in are usually correct, because the correct details are what appeared in its training text. When it does not, it still produces the shape of a good answer, and fills the gaps with details that are merely typical. A reference to a book needs an author, so it supplies a likely-sounding name. Nothing in the process marks the difference between a remembered fact and a plausible invention.
The word, and what it hides
"Hallucination" is the common name, though it can mislead. A person who hallucinates sees something that is not there; a model sees nothing at all. It is also not lying, because lying requires knowing the truth and intending to deceive. The model is doing what it always does, producing likely text, and on this occasion the likely text is false. Some researchers prefer the word "confabulation", borrowed from medicine, where it describes a person who fills gaps in memory with invented details without realising it.
Real consequences
In 2023 two lawyers in New York submitted a legal document to a court that cited six earlier court decisions. A chatbot had supplied them. None of the six cases existed. When the judge asked for copies, the lawyers asked the chatbot whether the cases were real, and it assured them that they were. The court fined the lawyers. In the same year a company demonstrated its new chatbot in an advertisement; the chatbot's answer about a space telescope contained a factual error, and when the mistake was noticed, the company's value on the stock market fell by about a hundred billion dollars in a day.
When hallucination is most likely
| Situation | Why the risk is high |
|---|---|
| Exact details: dates, figures, page numbers, addresses | The model learned the general pattern, not each precise value |
| Sources, quotations and links | A plausible reference is easy to generate and hard to tell from a real one |
| Obscure subjects | Little training text, so the gaps are filled with guesses |
| Local matters: your town, your school, a small company | Almost nothing about them was in the training data |
| Recent events | They happened after the model's training ended |
| Questions built on a false assumption | Asked "Why did X win the prize?", the model may explain a win that never happened |
Reducing the risk
- Use tools that search and cite. A tool connected to a search engine can show the pages it used. You must still open them: the tool can misreport what a page says.
- Ask for sources, then check that each one exists and says what is claimed.
- Ask neutral questions. "Did X win the prize?" is safer than "Why did X win?"
- Give it the material. Paste in the document and ask questions about that, instead of relying on its memory.
- Match the checking to the stakes. A bedtime story needs no checking. A medicine dose, a legal step or a fact in an assignment does.
Models have improved, and hallucinations are less frequent than they were. They have not disappeared, and because they are rarer, people check less, so each one is more likely to slip through. The habit of verifying matters more as the tools get better, not less.
How it works
| Kind of hallucination | Example |
|---|---|
| Invented fact | A river is said to flow through a city it does not reach |
| Invented source | A book, article or court case that does not exist |
| Invented quotation | Words attributed to a famous person who never said them |
| Wrong number | A date, a population or a sum that is slightly or wildly off |
| Mixed-up people | The achievements of two people with similar names are combined |
| False confidence | "Yes, I have checked, and this is correct", when nothing was checked |
Try it yourself
How an invented book appears
A student asks a chatbot: "Give me a book about the history of kite flying in Gujarat, with its author."
How carefully should you check?
Think about what happens if the answer is wrong.
A date and a quotation for a history assignment
A rhyme for the word "monsoon"
Five ideas for a birthday party game
The last date to apply for a scholarship
A friendlier way to word your own message
The dose of a medicine for a child
Common mistakes
- Trusting an answer because it is detailed and confident. Detail is exactly what a hallucination looks like.
- Using a source a chatbot gave you without finding it yourself.
- Asking the chatbot to confirm its own answer and treating that as a check.
Check yourself
Summary
- A hallucination is false content stated fluently and confidently.
- It happens because the model produces likely text and cannot check facts.
- Exact details and sources are the riskiest; verify anything that matters with an independent source.
Finished reading?
Sources: UNESCO: Guidance for generative AI in education and research · AI4K12: Five Big Ideas in AI