RAG evaluation metrics: answer relevancy, faithfulness, and more
The core RAG evaluation metrics explained — answer relevancy, faithfulness, contextual precision, contextual recall, and contextual relevancy — and what each catches.
TL;DR: A RAG system has two halves — retrieval and generation — and each fails differently, so you need metrics for both. Answer relevancy and faithfulness score the generated answer; contextual precision, recall, and relevancy score the retrieved context. Together they tell you not just that a RAG answer was bad but which half broke, which is the difference between guessing and fixing.
Retrieval-augmented generation (RAG) is the dominant pattern for grounding LLMs in your data: retrieve relevant context, then generate an answer from it. When a RAG answer is wrong, the cause is either bad retrieval (the wrong context was fetched) or bad generation (good context, bad answer) — and a single "was it good?" score can't tell them apart. The standard RAG evaluation metrics decompose quality so you can localize the failure.
This guide explains the core RAG metrics, what each one catches, and how to use them together.
The two halves of RAG
| Half | Job | Metrics |
|---|---|---|
| Retrieval | Fetch the right context | Contextual precision, recall, relevancy |
| Generation | Answer well from context | Answer relevancy, faithfulness |
Evaluate both, because a perfect generator can't fix bad retrieval, and perfect retrieval can't save a hallucinating generator.
Generation metrics
These score the answer the model produced, given the context it was handed.
Answer relevancy
Does the answer actually address the question? Answer relevancy measures whether the response is on-topic and directly responsive, penalizing answers that are vague, padded, or drift off the question. A high-faithfulness answer can still score low here if it dodges what was asked.
Catches: off-topic, evasive, or padded answers.
Faithfulness
Is the answer supported by the retrieved context? Faithfulness (also called groundedness) measures whether the claims in the answer are actually backed by the retrieved passages, rather than invented. This is the core anti-hallucination metric for RAG: a faithful answer only asserts what the context supports. (See AI hallucination evaluations: metrics and methods.)
Catches: hallucinations — claims not grounded in the retrieved context.
Retrieval metrics
These score the context the retriever fetched, before generation.
Contextual relevancy
Is the retrieved context relevant to the question? It measures what proportion of the retrieved context is actually pertinent, penalizing retrieval that returns mostly noise. Irrelevant context wastes the model's attention and invites distraction.
Catches: noisy retrieval that returns off-topic passages.
Contextual precision
Are the relevant chunks ranked above the irrelevant ones? Contextual precision measures whether the most relevant retrieved chunks appear first, since models weight earlier context more heavily. Good chunks buried below junk can be missed.
Catches: poor ranking — right chunks retrieved but ordered badly.
Contextual recall
Was all the necessary context retrieved? Contextual recall measures whether the retrieval fetched everything needed to answer fully, comparing against the information an ideal answer requires. Low recall means the answer is missing pieces the retriever failed to fetch.
Catches: incomplete retrieval — necessary context left behind.
Reading the metrics together
The power is in the combination — the pattern localizes the failure:
- Low faithfulness, good retrieval metrics → the generator is hallucinating; fix the prompt/model, not retrieval.
- Good generation metrics, low contextual recall → the retriever isn't fetching enough; fix chunking, top-k, or the query.
- Low contextual precision → retrieval finds the right chunks but ranks them badly; add reranking.
- Low answer relevancy, high faithfulness → the answer is grounded but dodges the question; fix the prompt.
A single quality score would hide all of these distinctions. (See debug a slow RAG pipeline.)
How to run RAG evaluations
- Build a dataset of questions with (ideally) reference answers and the context needed to answer them — drawn from real usage where possible.
- Capture the full RAG trace per question: query, retrieved chunks, and the generated answer.
- Score both halves — retrieval metrics on the chunks, generation metrics on the answer.
- Evaluate on production traffic, not just offline, since real questions surface retrieval gaps your dataset won't. (See run LLM evals on production traces.)
- Fix the half the metrics implicate, then re-measure.
How Currai fits
Currai traces each RAG request end to end — the query, the retrieved chunks, and the generated answer — and scores both halves against these metrics on real production traffic. When an answer is wrong, the metric pattern tells you whether retrieval or generation broke, and the trace shows you exactly which chunk or claim caused it. You stop guessing which half to fix. See debug a slow RAG pipeline and traces and evals in one place, or start with Currai free.
Frequently asked questions
What are the main RAG evaluation metrics?
Generation metrics — answer relevancy (does the answer address the question?) and faithfulness (is it supported by the context?) — and retrieval metrics — contextual relevancy, contextual precision (ranking), and contextual recall (completeness).
What is faithfulness in RAG evaluation?
Faithfulness (groundedness) measures whether the claims in the generated answer are actually supported by the retrieved context rather than invented. It's the core anti-hallucination metric for RAG.
What's the difference between contextual precision and recall?
Precision measures whether the relevant retrieved chunks are ranked above irrelevant ones (ordering); recall measures whether all the context needed to answer was retrieved at all (completeness). Low precision is a ranking problem; low recall is a fetching problem.
How do RAG metrics help me debug?
The pattern localizes the failure: low faithfulness with good retrieval means the generator hallucinates; good generation with low recall means the retriever misses context; low precision means ranking needs reranking. A single score hides all of this.