What is retrieval-augmented generation (RAG)?
Retrieval-augmented generation (RAG) has AI search relevant documents before it answers and base its reply on what it finds, so answers can cite sources.
After reading this article you will be able to:
- Explain what retrieval-augmented generation (RAG) is
- Describe the steps RAG takes to find documents and answer a question
- Recognize what RAG does well and where it falls short
On this page
Retrieval-augmented generation (RAG) is a way for an AI system to look up relevant documents before it answers a question, then write its answer based on what it found. The name comes from a 2020 research paper, Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks: the generation step, where the AI writes text, is augmented by a retrieval step, where it searches a collection of documents. Today, RAG is a common way to build AI features that answer from a specific set of documents, such as a help chat on a website or a search tool for internal manuals.
Knowing how RAG works helps you judge how far to trust an AI answer and how to check it. It’s also useful background if you’re thinking about adding an AI question box to your own site.
- Question“How do I cancel?”
- Search documentsBy meaning and by keyword
- Matching passagesThe “Cancel your plan” article
- LLM writes the answerUsing only what the passages say
- Answer with sourcesLinks to the articles used
Why does AI need to look things up?
Chat-style AI tools are built on a large language model (LLM), a model that has learned patterns of language from a huge amount of text. An LLM answers from what it learned during training, which leaves some gaps:
- It doesn’t know about anything that happened after its training data was collected. Its knowledge has a cutoff.
- It has never seen your private documents, such as your price list or your internal procedures.
- When it doesn’t know something, it can still produce a confident, plausible-sounding answer that is wrong. This is often called a hallucination.
Say you run an online course site and add an AI chat where visitors can ask questions. Someone asks, “When does registration close for next month’s course?” On its own, the LLM has no way of knowing your deadline. If it guesses a date, someone who wanted to sign up could miss out.
RAG addresses this by having the AI read the relevant material before it answers. Think of it as turning a closed-book exam into an open-book exam: the answer is backed by the pages the AI just looked up, not only by what it remembers.
How does RAG work?
RAG has two phases: preparation, which happens ahead of time, and search and answer, which happens each time someone asks a question.
Preparation: make your documents searchable
- Collect the documents the answers should come from, such as help articles, FAQs, terms of service, and course descriptions.
- Split each document into smaller pieces of a few paragraphs, called chunks. If you passed along whole documents, the AI would also have to read a lot of text that has nothing to do with the question.
- Turn each chunk into an embedding, a list of numbers that represents its meaning, and store it in a searchable index. Many systems also build a keyword index alongside it, so the same chunks can be found by their exact words.
When a question arrives: search, hand over, answer
- Turn the question into an embedding too, and look up the chunks with the closest meaning.
- Pick the few chunks that are most relevant.
- Send those chunks to the LLM along with the question and an instruction such as “Answer using only the documents below. If they don’t contain the answer, say you don’t know.”
- The LLM writes the answer. If it also lists which documents it used, readers can check the sources later.
When a document changes, you update its chunks in the index, and the next answer reflects the new content. There’s no need to retrain the LLM.
How does the search step find documents?
The quality of a RAG answer depends heavily on whether the search step finds the right material. There are two main ways to search.
| Method | How it works | Good at | Weak at |
|---|---|---|---|
| Search by meaning (vector search) | Compares how close embeddings are | Matching “I want to stop paying” to the “Cancel your plan” article, even though the words differ | Exact terms such as product codes, names, and numbers |
| Search by words (keyword search) | Counts how often the same words appear | Product codes, proper nouns, and fixed terms | Paraphrases and synonyms |
Many systems combine the two, which is called hybrid search. Some also use reranking, where a second model reorders the candidates so the best matches end up at the top.
Language matters too. Japanese, for example, doesn’t put spaces between words, so keyword search in Japanese often splits text into short runs of a few characters before indexing it. Settings that work well for one language may not suit another.
What does RAG improve?
- It can answer from your own, up-to-date information. That suits details that differ from business to business and change often, such as prices and deadlines.
- It can show its sources. Because each answer is tied to specific articles or pages, readers can check them.
- It makes mistakes less likely. You can tell the AI to stick to the documents, so it guesses less often. It doesn’t eliminate mistakes, though.
- It lets you decide what the AI draws on. If you keep separate sets of documents, you can, for example, build a public question box that answers only from articles you’ve published.
For example, say someone on a piano school’s website asks, “How many days’ notice do I need to give to reschedule a lesson?” With RAG, the AI can find the part of the school’s policies that covers rescheduling, answer from it, and link to the policy page. When you update the policies, the next answer follows the new rules.
What are RAG’s limits, and how do you work with them?
RAG is useful, but it doesn’t solve everything. Here are common problems and ways to handle them.
| Problem | What happens | What helps |
|---|---|---|
| The answer isn’t in the documents | The AI guesses | Tell it to say it doesn’t know and to point people to a way to contact you |
| Documents are outdated or wrong | The AI repeats the old information | Update the index whenever documents change, and decide who keeps each document current |
| The search misses | The AI answers from loosely related chunks | Rephrase the question in shorter words and search again; adjust how documents are split and searched |
| A document contains instructions aimed at the AI | The AI follows them | Treat document text as data and don’t follow instructions inside it |
| Too much text is passed along | Costs rise and answers slow down | Cap the number and length of chunks |
As for the first row, answering “I don’t know” isn’t a failure. Pointing someone to a way to contact you helps them more than a confident guess at the wrong date or price.
The fourth row is a problem called prompt injection. It matters most when the AI can also take actions with tools. What is an AI agent? covers it in more detail.
Summary
- RAG is a way for AI to search documents before it answers and to base its answer on what it finds.
- In preparation, documents are split into chunks and indexed. For each question, the most relevant chunks are found and passed to the LLM.
- RAG can answer from your own, current information, and it can show its sources.
- Answers are only as good as the documents and the search. Being able to say “I don’t know” matters too.
How does mukumo use RAG?
When you ask how to use mukumo, including settings, pricing, or troubleshooting, the AI assistant, muku, first searches the help center with Cloudflare AI Search. It is instructed to answer only from what the articles it finds say, and not to fill in screen names, steps, or numbers by guessing.
In chat, muku ends its reply with links to the articles it used. If nothing matches, it rephrases your question in shorter words and searches once more. If that still finds nothing, it tells you the help center doesn't cover it and points you to the help center and the contact form.
Available nowthe AI assistant, muku
FAQs
How is RAG different from fine-tuning?
Fine-tuning gives a model extra training to change its style or make it better at a specific task. RAG leaves the model as it is and hands it documents each time it answers. RAG is usually the better fit for information that changes often or answers that need sources.
Will RAG still be needed as AI models can read more text at once?
If you only have a few documents, you can pass all of them every time. But the more text you pass, the higher the cost and the longer the wait, and unrelated parts can pull the answer off course. With many documents, passing only the relevant parts is still useful.
What kinds of documents can RAG use?
Text works best, such as help articles, FAQs, manuals, and web pages. PDFs and tables are usually converted to text first.
Do I need a special database to build RAG?
Searching by meaning usually relies on a system that stores embeddings and finds close matches, such as a vector database. With a small set of documents, you can also start with keyword search alone.