What is retrieval-augmented generation (RAG)?

TL;DR: Retrieval-augmented generation (RAG) grounds an LLM's answers in your data by retrieving relevant documents at query time and putting them in the prompt, so the model answers from real sources instead of its frozen training memory. It's the default architecture for factual AI apps — and it fails in two distinct places, retrieval and generation, each of which you evaluate separately.

An LLM on its own only knows what was in its training data, up to a cutoff, and can't cite where anything came from. RAG fixes both by fetching relevant information and handing it to the model at answer time. This is the foundational explainer: what RAG is, why it's everywhere, and where it breaks.

The problem RAG solves

A raw LLM has three limits for factual apps:

• Frozen knowledge — it doesn't know your documents, your product, or anything after its training cutoff.
• Hallucination — asked something it doesn't know, it confidently makes something up.
• No provenance — it can't tell you where an answer came from.

RAG addresses all three by grounding answers in retrieved sources.

How RAG works

1. Index — your documents are split into chunks and embedded into vectors that capture meaning, stored in a vector index.
2. Retrieve — at query time, the user's question is embedded and used to find the most relevant chunks.
3. Augment — those chunks are inserted into the prompt as context.
4. Generate — the model answers using that context, ideally citing it.

The model's job shifts from "recall this fact" to "answer using these sources," which is far more reliable and lets you point at where an answer came from. (See knowledge-base RAG for AI chat.)

The two places RAG breaks

RAG has two failure points, and lumping them together makes debugging impossible:

• Retrieval failure — the wrong chunks (or no relevant chunks) get fetched. No matter how good the model is, it can't answer from context it never received.
• Generation failure — the right context was retrieved, but the model ignored it, misread it, or hallucinated anyway.

Which one broke determines the fix — reranking and chunking for retrieval, prompt and model for generation — so you must evaluate them separately. That's exactly what RAG evaluation metrics do: contextual relevancy scores retrieval, faithfulness and answer relevancy score generation. (For the performance side, see debugging a slow RAG pipeline.)

RAG vs. fine-tuning

They solve different problems. RAG gives the model knowledge at query time — best for facts that change or are too large to train in. Fine-tuning changes the model's behavior and style — best for tone, format, and task specialization. Many apps use both. (See evaluating a fine-tuned LLM.)

How Currai fits

Because RAG fails in two places, you have to see both — and Currai traces the full RAG flow: the query, which chunks were retrieved, and the answer that used them. That lets you score retrieval and generation separately (contextual relevancy vs. faithfulness), find whether a bad answer came from bad retrieval or bad generation, and run those scores on production traffic, not just a test set. See RAG evaluation metrics and run LLM evals on production traces, or start with Currai free.

Frequently asked questions

What is retrieval-augmented generation?

An architecture that grounds an LLM's answers in your data by retrieving relevant documents at query time and adding them to the prompt, so the model answers from real sources instead of its frozen training memory — reducing hallucination and enabling citations.

Why use RAG instead of just a bigger prompt or a better model?

RAG lets the model answer from data that changes, is private, or is too large to fit in a prompt or training run — with provenance. A bigger model still doesn't know your documents; RAG hands them to it at answer time.

Where does RAG usually fail?

In two places: retrieval (the wrong or no relevant context is fetched) and generation (the right context is retrieved but the model ignores or misreads it). You evaluate them separately because the fixes are different.

Is RAG better than fine-tuning?

They solve different problems. RAG supplies knowledge at query time; fine-tuning changes behavior and style. For factual, changing, or private data, RAG is usually the right tool — and many apps combine both.