RAG, Explained — Building AI Apps: Retrieval

A large language model can write fluent answers, but fluency is not the same as knowing the current facts, the company policy, or the contents of a private document. Retrieval-augmented generation, usually shortened to RAG, is a practical pattern for giving a model relevant source material at the moment it answers. The model still generates the prose, but it is asked to reason from retrieved evidence rather than from its internal memory alone.

Start with the problem: words are not facts

A large language model is software trained to predict likely text. During training, it absorbs patterns from huge collections of writing: grammar, style, common facts, programming idioms, and many relationships between ideas. That makes it useful, but it does not make it a database.

The model’s internal knowledge is frozen at the time of training. It may not know what changed yesterday. It also may not know anything that was never in its training data, such as a company’s internal runbook, a legal contract, or a customer’s support history. Even when it has seen something related, it may blend facts together and produce an answer that sounds confident but is wrong.

RAG addresses this gap by separating two jobs. One system finds relevant information. The language model then uses that information to produce a readable answer. The aim is not to make the model omniscient. The aim is to make the answer depend on evidence that can be inspected, updated, and governed.

📌

The key shift. RAG turns the question from “What does the model remember?” into “What evidence should the model read before answering?”

The basic RAG pipeline

A RAG system begins when a person asks a question. The system uses that question to search a collection of documents, records, web pages, tickets, or other sources. It selects a small amount of relevant material and places it into the model’s input along with the user’s question.

The model then generates an answer from both the question and the retrieved material. If the system is designed well, the answer can include citations or links back to the source passages. This makes the response easier to verify and easier to correct when the underlying information changes.

The simplest mental model is open-book answering. A closed-book model answers from memory. A RAG system opens the right pages first, then asks the model to answer from those pages.

A question drives a search, the retrieved material joins the prompt, and the model answers from both.

Before retrieval, the knowledge has to be prepared

RAG usually depends on a preparation step called indexing. Indexing is the process of turning source material into a form that can be searched quickly. Without it, the system would have to read every document for every question, which is too slow and too expensive for most uses.

First, source documents are collected and cleaned. Then they are split into smaller pieces, often called chunks. A chunk might be a paragraph, a section, a table row, or a few hundred words. The right chunk size matters: too small, and the system loses context; too large, and irrelevant text crowds out the useful part.

Many RAG systems then create embeddings. An embedding is a list of numbers that represents the meaning of a piece of text in a way software can compare. Texts with similar meanings tend to have embeddings that are close together. These embeddings are stored in a search index, often a vector database, so the system can find passages that are semantically related to a question even if they do not use the exact same words.

What happens at answer time

At answer time, the system must decide what evidence to send to the model. This is harder than it sounds. A user’s question may be vague, may use different words than the source documents, or may require information from several places. The retrieval step often combines keyword search, semantic search using embeddings, filters such as date or department, and sometimes a reranker that reorders candidate passages by likely usefulness.

The selected passages are placed into the model’s context window, which is the amount of text the model can consider at one time. The prompt usually includes instructions such as “answer only from the provided sources” or “say you do not know if the sources are insufficient.” These instructions help, but they are not magic. If the retrieved material is weak, stale, contradictory, or irrelevant, the model may still produce a poor answer.

Good RAG design treats the prompt as the final assembly point, not the whole solution. The quality of the answer depends on the quality of the content, the retrieval method, the ranking logic, the prompt, and the user interface that shows what the answer is based on.

Keyword search is strong when exact terms, names, or identifiers matter.
Semantic search is strong when the question and the source use different wording for the same idea.
Filters are essential when permission, date, product, region, or document type changes the correct answer.
Reranking helps when the first search finds many plausible passages but only a few are truly useful.

RAG reduces hallucination, but it does not eliminate it

A hallucination is an answer that is not supported by the facts, even if it is written fluently. RAG can reduce hallucination by grounding the model in retrieved sources. Grounding means tying the answer to specific evidence rather than to the model’s unaudited memory.

But RAG can fail in several ordinary ways. The system may retrieve the wrong passage. It may retrieve the right passage plus several distracting ones. The source may be outdated. The model may overstate what the source says. The user may ask for a conclusion that no source supports. These are engineering and governance problems, not just model problems.

This is why source visibility matters. When users can inspect the passages behind an answer, they can notice weak evidence, missing documents, or stale policies. For high-stakes work, RAG should be treated as decision support, not as an unquestioned authority.

⚠️

Citations are not proof. A citation only shows where the system pointed; someone still has to check whether the cited source truly supports the claim.

Grounding helps, but RAG still fails through wrong passages, distractors, stale sources, and overstated claims.

RAG is not the same as training the model

Training changes the model’s internal parameters, which are the learned numerical settings that shape how it predicts text. Fine-tuning is a narrower form of training that adapts a model to a style, format, or task using additional examples. RAG usually does not change the model at all. It changes what the model is shown at the moment of answering.

That distinction matters. If the information changes often, RAG is usually easier to update: re-index the source, and future answers can use the new material. If the goal is to make the model follow a specialized output format, classify records consistently, or use a particular tone, fine-tuning may help. Many mature systems use both: retrieval for facts, training or fine-tuning for behavior.

A useful rule of thumb is this: put changing knowledge in retrievable sources, and put repeated behavior in the model or surrounding application logic.

RAG changes what the model is shown at answer time; training changes the model itself.

The decisions leaders and practitioners actually face

A RAG project is not only a model selection exercise. The central decisions are about content, access, risk, and workflow. The system can only retrieve what has been indexed, and it should only retrieve what the user is allowed to see. If the organization’s documents are messy, duplicated, poorly labeled, or full of obsolete versions, RAG will expose those problems rather than hide them.

Teams also need to decide where the answer belongs in the work. A support agent may need a drafted reply with citations. An engineer may need a runbook excerpt and a command suggestion. A compliance reviewer may need a list of source passages, not a polished summary. The same retrieval engine can support different interfaces, but each use case has a different tolerance for error.

Cost and latency matter too. Searching more sources, reranking more passages, and using a larger context window can improve answers, but they can also slow the system and raise costs. The right design is usually the smallest one that reliably answers the important questions within the required risk boundary.

Which sources are authoritative, and who owns keeping them current?
How will permissions be enforced at retrieval time, not just in the user interface?
What questions should the system refuse to answer without enough evidence?
Where should citations, confidence signals, or review steps appear in the workflow?
What level of latency and cost is acceptable for this use case?

⚠️

Permission matters. A RAG system that retrieves restricted information for the wrong user is a data leak, even if the model summarizes it politely.

How to know whether RAG is working

A RAG system should be evaluated in parts and as a whole. Retrieval quality asks whether the right evidence was found. Answer quality asks whether the response was accurate, complete, clear, and supported by the evidence. User outcome asks whether the system helped someone finish a real task safely and efficiently.

Good evaluation starts with a test set of realistic questions and expected source material. Some questions should have clear answers. Some should require several sources. Some should be unanswerable from the available content, because a trustworthy system must know when to say it does not know. Human review is still important, especially for sensitive domains where a plausible answer can be more dangerous than an obviously broken one.

Production monitoring is the next step. Teams should watch for unanswered questions, low-rated answers, missing sources, retrieval drift, stale content, and repeated user corrections. RAG is not a one-time feature. It is a living information system wrapped around a language model.

✅

A healthy sign. A good RAG system sometimes refuses to answer because the retrieved evidence is not strong enough.

Key takeaways

RAG means retrieval-augmented generation: retrieve relevant evidence, then ask a language model to answer from it.
RAG is useful because language models are fluent but not reliable databases of current or private facts.
Indexing, chunking, embeddings, retrieval, ranking, prompts, and citations all affect answer quality.
RAG reduces hallucination by grounding answers in sources, but weak retrieval or stale sources can still produce bad results.
RAG is different from training: use RAG for changing knowledge and training or fine-tuning for repeated behavior.
Successful RAG projects depend as much on content ownership, permissions, evaluation, and workflow design as on model choice.

Judge retrieval quality, answer quality, user outcome, and whether the system knows when to abstain.

Glossary

Retrieval-augmented generation: A pattern in which software retrieves relevant source material and gives it to a language model before the model generates an answer.
Large language model: A model trained on large amounts of text to predict and generate language.
Embedding: A numerical representation of text that helps software compare meanings rather than only exact words.
Vector database: A database designed to store and search embeddings efficiently.
Chunk: A smaller piece of a document, such as a paragraph or section, used for indexing and retrieval.
Context window: The amount of text a language model can consider at one time when generating an answer.
Grounding: The practice of tying a model’s answer to specific evidence or source material.
Hallucination: A fluent model output that is not supported by the facts or by the provided evidence.
Fine-tuning: Additional training that adapts a model toward a particular behavior, style, or task.