What Is RAG? Retrieval-Augmented Generation Explained

A language model knows what was in its training data and nothing else. It cannot read your company's internal documentation, this morning's support tickets, or a contract signed last week. Asked about any of them, it will either decline or produce something plausible and wrong.

Retrieval-augmented generation is the standard fix:

RAG is a pattern where relevant documents are retrieved from an external source and inserted into the model's prompt, so the model answers from that supplied material rather than from memory alone.

The term comes from Lewis et al. (2020), which described combining a retrieval component with a generation component. What the industry now calls RAG is usually simpler than that paper's architecture — the retriever is typically not trained jointly with the generator — but the core idea carried over: fetch relevant material first, then generate conditioned on it.

This article covers the actual pipeline, the decisions that determine whether it works, where it fails, and the genuine argument about whether long-context models make it unnecessary.

Why the Alternative Approaches Fall Short

RAG exists because the obvious alternatives have specific problems.

Fine-tuning adjusts model weights on your data. It is effective for teaching format, tone, or a specialized task, but poorly suited to teaching facts that change. Every update requires retraining, the model cannot cite where an answer came from, and there is no clean way to remove a document that should no longer be used.

Putting everything in the prompt works when "everything" is small. It stops working when the corpus exceeds the context window, and becomes expensive well before that, since you pay for every token on every request.

Traditional keyword search finds documents but does not answer questions. The user still has to read and synthesize.

RAG combines search's freshness and attributability with the model's ability to synthesize an answer.

How the Pipeline Actually Works

RAG has two phases: one that runs ahead of time, and one that runs per query.

Indexing (Ahead of Time)

1. Load. Pull documents from their sources — files, databases, wikis, ticketing systems — and extract text. This step is consistently underestimated. PDFs with multi-column layouts, scanned images, and tables routinely produce garbled text, and no downstream cleverness recovers from a bad extraction.

2. Chunk. Split documents into passages. Whole documents are usually too large to retrieve usefully; individual sentences lack context. Chunk size and boundary placement are among the highest-leverage decisions in the pipeline.

3. Embed. Convert each chunk into a vector using an embedding model. Semantically similar text produces vectors that are close together, which is what makes meaning-based retrieval possible.

4. Store. Write vectors and their source text into a vector store — a dedicated database such as Pinecone, Qdrant, Weaviate, or Milvus, or an extension to an existing one such as pgvector for PostgreSQL.

Retrieval and Generation (Per Query)

1. Embed the query using the same model used for indexing. Using a different model produces meaningless comparisons.

2. Search for the nearest chunks by vector similarity, usually cosine similarity, typically retrieving somewhere between three and twenty candidates.

3. Optionally rerank. A cross-encoder scores each candidate against the query directly. This is slower than vector similarity but more accurate, so a common pattern is retrieving broadly then reranking down to the few that actually go in the prompt.

4. Assemble the prompt — the retrieved passages, the user's question, and instructions telling the model to answer from the provided context and to say so when the context is insufficient.

5. Generate, and return the answer along with citations to the source chunks.

A Concrete Example

A support engineer asks: "What is our refund window for annual plans?"

The system embeds that question, searches the indexed policy documents, and retrieves three chunks — one from the refund policy, one from the annual billing terms, one from a support FAQ. Those three passages plus the question go into the prompt, and the model answers from them, citing the policy document.

The model was never trained on that policy. It read it at query time.

The Decisions That Determine Whether It Works

Chunk size. Too small and passages lack the context needed to be understood alone; too large and each retrieved chunk carries irrelevant text that dilutes the prompt. Splitting on semantic boundaries — sections, paragraphs — generally beats splitting on a fixed character count, and overlapping chunks slightly reduces the risk of a boundary landing mid-explanation.

How many chunks to retrieve. Too few and the answer may not be present; too many and the relevant passage competes with noise, while cost rises on every query.

Which embedding model. This determines what "similar" means. Domain-specific corpora with heavy jargon often retrieve poorly under general-purpose embeddings. Changing this model requires re-indexing the entire corpus.

Hybrid search. Vector search handles meaning but can miss exact strings — error codes, product SKUs, names. Combining vector similarity with keyword search covers both, and is frequently worth the added complexity.

Metadata filtering. Restricting retrieval by date, department, or access permissions before similarity search improves relevance and is usually mandatory for access control.

Where RAG Fails

Most RAG failures are retrieval failures wearing a generation costume.

The right document was never retrieved. If the answer is not in the retrieved chunks, no prompt engineering recovers it. This is the most common failure, and it is why measuring retrieval separately from answer quality is essential. If retrieval recall is poor, tuning the generation prompt is wasted effort.

The chunk boundary split the answer. A procedure divided across two chunks, only one of which is retrieved, produces a confidently incomplete answer.

The model ignored the context. Given retrieved passages, models sometimes answer from training data instead — particularly when the retrieved material contradicts what they learned. Explicit instructions to rely only on provided context reduce this without eliminating it.

Conflicting sources. When retrieval returns an outdated policy and a current one, the model has no reliable way to know which governs. Recency metadata and filtering matter here.

Questions retrieval cannot serve. "Summarize every complaint we received last quarter" is an aggregation, not a lookup. Retrieving the top few chunks cannot answer it. These questions need a different architecture.

Is RAG Still Relevant?

This question is asked increasingly often as context windows have grown, and it deserves a real answer rather than a defensive one.

The argument against RAG: if a model can accept a very large context, why not skip retrieval and supply the whole corpus?

Where that argument holds: for small, stable corpora that comfortably fit in context, retrieval infrastructure is genuine overhead. A few hundred pages of documentation that rarely change may not justify a vector database.

Where it does not hold:

Cost. You pay per token on every request. Sending an entire corpus for each query is dramatically more expensive than sending the handful of relevant passages, and that difference scales with traffic.

Latency. Larger contexts take longer to process.

Scale. Corpora frequently exceed even large context windows. Enterprise document stores are not measured in hundreds of pages.

Attribution. Retrieval tells you which documents informed the answer. That matters for verification and for regulated environments.

Access control. Filtering at retrieval enforces per-user permissions. Sending everything and asking the model to respect permissions is not a security model.

The practical position is that RAG remains the default for corpora of meaningful size, while long context has made it unnecessary for a class of smaller applications that previously reached for it by default. Both being true at once is why the argument persists.

Conclusion

RAG retrieves relevant material and puts it in the prompt so the model answers from current, verifiable sources rather than from training data alone. It addresses the fundamental limitation that a model's knowledge is frozen at training time and cannot include anything private.

The pattern is not difficult to implement badly. A basic pipeline can be assembled quickly, which is why so many exist and why so many underperform. The difficulty is in retrieval quality — chunking, embedding choice, hybrid search, reranking — and in measuring retrieval separately from generation so you know which half is failing.

For teams starting out, the highest-value early investment is an evaluation set: a list of representative questions with their correct source documents. It converts "the answers seem worse" into a measurable retrieval number, and it is what makes every subsequent decision empirical rather than speculative.

Frequently Asked Questions

What is a RAG example?

A support assistant answering from internal documentation is the canonical case. A user asks about a refund window; the system embeds that question, retrieves the most relevant passages from indexed policy documents, places them in the prompt with the question, and the model answers citing those sources. Other common examples include querying legal or compliance archives, answering from product manuals, and searching internal engineering wikis — anywhere the source material is private, current, or too large to fit in a prompt.

How do I build a RAG system?

At minimum: extract text from your documents, split it into chunks, embed each chunk with an embedding model, and store the vectors. At query time, embed the question, retrieve the nearest chunks, put them in the prompt with instructions to answer only from the provided context, and generate. Frameworks such as LangChain and LlamaIndex provide these components, and pgvector lets you use an existing PostgreSQL database rather than adding a dedicated vector store. Build the evaluation set before optimizing anything.

What is the difference between RAG and fine-tuning?

RAG supplies knowledge at query time by putting documents in the prompt; fine-tuning changes model weights by training on examples. Use RAG for facts that change, need citation, or must be removable. Use fine-tuning to teach a consistent format, tone, or specialized task the model handles poorly. They are complementary rather than competing — a fine-tuned model can serve a RAG pipeline. As a rule of thumb, RAG teaches the model what it should know, fine-tuning teaches it how to behave.

Is RAG still relevant with long context windows?

For corpora of meaningful size, yes. Long context removes the need for retrieval when a small, stable corpus fits comfortably in the window, and in those cases the infrastructure is genuine overhead. It does not address cost per token on every request, latency on large contexts, corpora larger than any window, attribution of which source produced an answer, or per-user access control enforced at retrieval. RAG has become less automatic than it was, not obsolete.

How to explain RAG in an interview?

State the problem first: a model's knowledge is frozen at training time and excludes anything private, so it cannot answer from your data. Then the mechanism: retrieve relevant documents and put them in the prompt, so the model answers from supplied material. Then demonstrate depth by naming a real trade-off — that most RAG failures are retrieval failures rather than generation failures, so retrieval quality should be measured separately. That last point signals practical experience more than reciting the pipeline stages.

What is a vector database, and do I need one?

A vector database stores embeddings and searches them efficiently by similarity, using indexes such as HNSW to avoid comparing against every stored vector. You need one when the corpus is large enough that brute-force comparison becomes slow. For smaller collections, pgvector on an existing PostgreSQL instance, or even in-memory similarity search, is often sufficient — and adding a separate database to a stack has its own operational cost worth weighing.