Definition
How can a chatbot answer questions on your company's private documents without being trained on them?
Retrieval-Augmented Generation (RAG) is a technique in which a system first searches a knowledge source for passages relevant to a question, then gives them to an LLM along with the question, so the answer is based on that retrieved material.
In simple words
It's an open-book exam. The LLM is a smart student with a fuzzy memory. Without the book, the student may guess. With RAG, a helper finds the right pages first and says, "Answer from these." The answers get more accurate, and can say where they came from.
Overview
- Retrieval-augmented generation (RAG) gives an LLM relevant documents at question time so it can answer from them.
- It was introduced by Lewis et al. in 2020.
- A RAG pipeline has two phases: indexing documents and answering questions.
- Indexing splits documents into chunks, embeds them and stores them in a vector database.
- Answering retrieves the most relevant chunks and puts them in the prompt with the question.
- RAG keeps answers current, cites sources and reduces hallucination without retraining the model.
Why it matters
LLMs don't know your private data or recent events, and retraining them is expensive. RAG adds both cheaply, and updating it is as simple as updating the documents. It's one of the most common ways that companies put LLMs to work today.
Key terms
- Retrieval-augmented generation (RAG)
- Retrieving relevant passages and giving them to an LLM so its answer is grounded in them.
- Chunk
- A short piece of a document, such as a few paragraphs, stored for retrieval.
- Embedding
- A vector representing the meaning of a piece of text.
- Vector database
- A store that finds the embeddings most similar to a query quickly.
- Grounding
- Basing an answer on supplied sources rather than the model's memory alone.
- Citation
- A reference showing which retrieved passage supports a statement.
Types
RAG systems come in several types, from simple to advanced.
1. Naive (basic) RAG
The simplest pipeline: embed the question, retrieve the top-k chunks, generate an answer.
- How it works: Documents are chunked and embedded once; each question is embedded, the nearest chunks are found by cosine similarity and pasted into the prompt.
- Example: A school-policy bot answering "When are fees due?" from the fee policy chunk.
- When to use it: Use it as the first version of any document question-answering system.
2. Hybrid search RAG
Combining keyword search (such as BM25) with embedding search.
- How it works: Keyword search finds exact terms such as product codes and names; embeddings find paraphrases; the two ranked lists are merged.
- Example: Finding the manual for model "XR-220" that embeddings alone ranked too low.
- When to use it: Use it when questions contain codes, names or rare terms.
3. RAG with re-ranking
Retrieving many candidates and reordering them with a more accurate model.
- How it works: A fast search returns perhaps 50 chunks; a cross-encoder re-ranker scores each against the question and keeps the best 5.
- Example: Improving answer accuracy on a large legal document collection.
- When to use it: Use it when the right chunk is often retrieved but not ranked at the top.
4. Query-transforming RAG
Rewriting or expanding the question before searching.
- How it works: The LLM rewrites vague or conversational questions, splits multi-part questions or generates a hypothetical answer to search with (HyDE).
- Example: Turning "what about for kids?" into "What is the ticket price for children at the museum?".
- When to use it: Use it in chat interfaces where follow-up questions depend on earlier turns.
5. Agentic and graph RAG
RAG where an agent decides when and where to search, or where knowledge is stored as a graph.
- How it works: An agent can search several times, use different tools and check its answer; graph RAG links entities and relationships so connected facts can be followed.
- Example: Answering "Which suppliers of our top product also supply competitors?" by following links.
- When to use it: Use them for complex, multi-step questions across many sources; they cost more and are harder to test.
Explained in detail
The problem RAG solves
An LLM knows only what was in its training data, which stops at a cutoff date and never included your school's rulebook, your company's policies or yesterday's news. Asked about them, it may guess and hallucinate. Retraining the model whenever documents change is slow and expensive. RAG, described in a 2020 paper by Patrick Lewis and colleagues at Facebook AI Research, takes a simpler route: at question time, search a collection of documents for relevant passages, paste them into the prompt, and instruct the model to answer from them. The model becomes a reader of your documents rather than a memoriser of the internet.
The pipeline
- Prepare: split documents into chunks of a few hundred words, with some overlap.
- Embed: turn each chunk into an embedding vector and store it in a vector index.
- Retrieve: embed the user's question and find the most similar chunks, often combined with keyword search.
- Augment: put the top chunks into the prompt with instructions such as "answer only from these sources and cite them".
- Generate: the LLM writes an answer grounded in, and citing, the retrieved passages.
What makes RAG work well
| Factor | Why it matters |
|---|---|
| Chunk size | Too small loses context; too large dilutes relevance and wastes the context window |
| Retrieval quality | If the right passage is not retrieved, the model cannot use it; most RAG failures are retrieval failures |
| Hybrid search and re-ranking | Combining keyword and meaning-based search, then re-ordering results, improves accuracy |
| Clear instructions | Telling the model to say "I don't know" when sources lack the answer reduces hallucination |
| Evaluation | Test with real questions and check both whether the right passages were found and whether answers are faithful to them |
- RAG keeps knowledge current: update the documents, not the model.
- Citations let users verify answers, which builds trust.
- Access control matters: retrieval must respect who is allowed to see which documents.
RAG is now one of the most common ways organisations use LLMs: customer-support bots answer from help-centre articles, legal assistants search case files, and government help desks answer from scheme guidelines. In each case, the quality of the document collection matters as much as the model. Out-of-date, contradictory or poorly written documents produce poor answers, however capable the LLM. A RAG project therefore usually begins with cleaning and organising the knowledge base, deciding who owns each document and how it will be kept current.
How it works
Test the search and the answer separately
Write ten questions about the school handbook. For each answerable question, mark the paragraph that contains the answer. If the top three search results contain a supporting paragraph on seven of ten answerable questions, the hit rate at three is 70%. Now inspect the seven answers. A citation counts as evidence only if its passage supports the claim. A correct passage can still lead to an invented date or an incomplete answer.
Add questions the handbook cannot answer and check whether the assistant asks for clarification or admits the gap. Fix a retrieval failure by checking chunk boundaries and query terms. For generation errors, require a supporting passage, inspect the answer against it and rerun the same held-out questions after each change. Keep private documents outside a learner's searchable collection.
Step 1: index, once ahead of time
- Split documents into small chunks.
- Turn each chunk into an embedding vector.
- Store the vectors in a vector database.
Step 2: answer, on every question
- Turn the question into a vector too.
- Find the chunks with the closest vectors.
- Put those chunks into the prompt with the question.
- The LLM writes an answer, ideally with citations.
The maths
- The question's embedding
- A document chunk's embedding
In code
The retrieval half of RAG, using TF-IDF for simplicity. Real systems use neural embeddings and a vector database, but the idea is identical.
from sklearn.feature_extraction.text import TfidfVectorizer
from sklearn.metrics.pairwise import cosine_similarity
chunks = [
"Refunds are allowed within 30 days of purchase with a receipt.",
"Our office is open Monday to Friday, 9 am to 6 pm.",
"Shipping to Mumbai takes 3 to 5 working days.",
]
question = "Are refunds allowed after purchase?"
vectorizer = TfidfVectorizer(stop_words="english")
vectors = vectorizer.fit_transform(chunks + [question])
scores = cosine_similarity(vectors[-1], vectors[:-1])[0]
best = scores.argmax()
print("Scores:", scores.round(2))
print("Prompt for the LLM:")
print(f"Answer only from this text: {chunks[best]}")
print(f"Question: {question}")Output
Scores: [0.64 0. 0. ] Prompt for the LLM: Answer only from this text: Refunds are allowed within 30 days of purchase with a receipt. Question: Are refunds allowed after purchase?
Common mistakes
- Making chunks too big or too small. Too big dilutes the match, and too small loses the context.
- Skipping evaluation. Test with real questions and check that it finds the right passages.
- Ignoring access control. Users must only get answers from documents they're allowed to see.
Check yourself
Summary
- RAG first retrieves relevant text, then asks the LLM to answer from it.
- It adds private and fresh knowledge without retraining.
- Retrieval quality decides answer quality.
Finished reading?
Sources: Es et al., evaluating retrieval and generation · Lewis et al., RAG (2020)