How to evaluate an LLM summarization task, step by step
Summarization looks easy to grade and isn't — a fluent summary can be unfaithful, incomplete, or subtly wrong. Here's how to score the two things that actually matter: faithfulness to the source and coverage of what mattered.
TL;DR: A good summary is faithful (says nothing the source doesn't) and complete (covers what mattered), and those two pull against each other — a shorter summary is more faithful and less complete. Evaluate both explicitly with an LLM-as-a-judge against the source, because fluency and length are easy to measure and tell you nothing about whether the summary is actually right.
Summarization is deceptively hard to evaluate. The output is always fluent, so it looks good, and the classic overlap metrics (does it share words with a reference?) reward paraphrase over correctness. This tutorial scores the things that actually matter.
The two dimensions that matter
- Faithfulness (precision) — does every claim in the summary follow from the source? A summary that invents, exaggerates, or misattributes is worse than useless, because readers trust it. This is the same groundedness metric that governs RAG. (See RAG evaluation metrics.)
- Coverage (recall) — does the summary include the information that actually mattered from the source? A perfectly faithful summary that drops the key point failed its job.
These trade off: cut length and you gain faithfulness but lose coverage. Good summarization evaluation scores both so you can see the trade-off instead of optimizing one blindly.
Why reference-overlap metrics fall short
Metrics that compare the summary to a human "reference summary" by word overlap (the ROUGE family) miss the point three ways: a faithful summary phrased differently scores low, an unfaithful summary that reuses source words scores high, and you need expensive reference summaries in the first place. They measure surface similarity, not correctness.
Step-by-step
- Build a dataset of source documents — ideally real ones from your app's traffic, spanning the lengths and domains you actually summarize.
- Score faithfulness with a judge — give the LLM-as-a-judge the source and the summary, and have it flag any claim not supported by the source. The metric is the fraction of claims that are grounded.
- Score coverage with a judge — identify the key points in the source, then check how many the summary captures. This needs the source too; you're scoring recall of what mattered.
- Watch length as context, not a goal — track it so you can read the faithfulness/coverage trade-off, but don't optimize length directly.
- Run on production, not just samples — score real summaries continuously and fold failures back in. (See run LLM evals on production traces.)
Metric summary
| Dimension | Question | How |
|---|---|---|
| Faithfulness | Is every claim supported by the source? | Judge with source provided |
| Coverage | Are the key points included? | Judge against extracted key points |
| Length | What's the trade-off point? | Track as context, don't optimize |
How Currai fits
Currai scores summarization on the dimensions that matter — faithfulness and coverage — with an LLM-as-a-judge that sees the source alongside the summary, so "grounded" and "complete" are checkable rather than assumed. It runs offline and on production traffic with one rubric, ties scores to the prompt and model version that produced them, and turns unfaithful summaries from real traffic into test cases. See the LLM evaluation playbook and traces and evals in one place, or start with Currai free.
Frequently asked questions
What makes a good LLM summary?
Faithfulness — every claim follows from the source — and coverage — the key points are included. The two trade off against length, so a good evaluation scores both rather than optimizing one.
Why not use ROUGE or reference-overlap metrics?
They measure word overlap with a reference summary, not correctness. A faithful summary phrased differently scores low, an unfaithful one reusing source words scores high, and you need costly reference summaries to run them at all.
How do I score summarization without a reference summary?
Use an LLM-as-a-judge with the source provided: score faithfulness by flagging claims not supported by the source, and coverage by checking how many of the source's key points the summary captures. No reference summary required.
Is faithfulness the same as the RAG groundedness metric?
Effectively yes — both ask whether the output's claims are supported by a given source. Summarization faithfulness and RAG groundedness are the same idea applied to different tasks.