LLM-as-a-judge: the complete guide to running evals at scale
How to use a model to grade model outputs — writing rubrics that hold up, avoiding the biases that quietly skew scores, and validating the judge against humans so you can trust it at scale.
TL;DR: LLM-as-a-judge uses a strong model to score outputs against a rubric, which is the only way to evaluate open-ended text at scale without a golden answer for every case. It works when the rubric is specific, the judge is calibrated against human labels, and you control for known biases — and it quietly lies to you when it isn't.
Most valuable LLM outputs — a summary, a support reply, an agent's plan — have no single correct answer to diff against. That's what makes evaluation hard, and it's why LLM-as-a-judge became the default method: a capable model can read an output and a rubric and produce a score, at a scale and cost no human panel can match. This guide is how to do it so the scores mean something.
Why LLM-as-a-judge, and when
Reach for a judge model when:
- Outputs are open-ended (no exact golden answer).
- You need to score thousands or millions of cases — every production trace, not a sample. (See run LLM evals on production traces.)
- The quality you care about is a judgment (helpfulness, faithfulness, tone), not a lookup.
Don't reach for it when a cheaper check works: exact match, schema validation, a
regex, or a deterministic assertion. A judge for something a === can decide is
just expensive and noisier.
Writing a rubric that holds up
The rubric is the whole game. Vague rubrics produce confident, meaningless scores. Good ones:
- Score one dimension at a time — faithfulness or helpfulness or tone, not a blended "quality" number that hides what failed.
- Define the scale concretely — say what a 1 looks like and what a 5 looks like, with examples. "Rate 1–5" without anchors invites drift.
- Ask for a reason, then a score — requiring the judge to justify its call before scoring reduces arbitrary numbers and gives you something to audit.
- Reference the source — for faithfulness, hand the judge the retrieved context so "grounded" is checkable, not guessed.
The biases that skew judge scores
A judge model has systematic tendencies you must control for, or your leaderboard is measuring the bias, not the output:
- Position bias — in pairwise comparisons, it favors whichever answer came first. Mitigate by swapping order and averaging.
- Verbosity bias — it rates longer answers higher regardless of quality.
- Self-preference — it favors outputs from its own model family.
- Leniency drift — it clusters scores at the top of the scale unless the rubric forces discrimination.
Pairwise ("which is better, A or B?") is often more reliable than absolute scoring because relative judgments are more stable — which is the idea behind arena-style, LLM-as-a-judge ranking.
Validate the judge — don't just trust it
A judge you haven't checked is a random number generator with good grammar. Before you rely on it:
- Human-label a sample — have people score a few hundred cases.
- Measure agreement — check how often the judge matches the humans. Low agreement means the rubric or judge model is wrong for the job.
- Iterate the rubric — refine wording where judge and humans diverge, then re-measure.
This is where human-in-the-loop review earns its keep: humans calibrate the judge once, then the judge scales to all your traffic while humans spot-check the edges.
Running it at scale
| Step | Practice |
|---|---|
| Choose the check | Deterministic if possible; judge only for real judgments |
| Write the rubric | One dimension, anchored scale, reason-before-score |
| Pick the mode | Pairwise for ranking, absolute for tracking a metric |
| Debias | Swap positions, watch for verbosity/self-preference |
| Calibrate | Validate against human labels, measure agreement |
| Deploy | Score every production trace, not a sample |
How Currai fits
Currai runs LLM-as-a-judge over your traces — offline test sets and live production traffic alike — with rubrics you control and every judgment linked back to the trace it scored, so a score is always auditable rather than a black box. It supports pairwise and absolute scoring, keeps judge results next to the human labels you use to calibrate them, and aggregates scores across prompt versions and variants so you can see which change actually won. See why A/B test LLM prompts and traces and evals in one place, or start with Currai free.
Frequently asked questions
What is LLM-as-a-judge?
Using a capable language model to score other models' outputs against a rubric. It's the standard way to evaluate open-ended text — summaries, replies, plans — at a scale where human grading is impossible.
Is LLM-as-a-judge reliable?
Only when calibrated. Judges have position, verbosity, and self-preference biases, and vague rubrics produce meaningless scores. Validate the judge against human labels and control for the biases, and it becomes reliable enough to run on all your traffic.
Pairwise or absolute scoring?
Pairwise ("A vs B") tends to be more stable and is better for ranking versions or picking a winner. Absolute scoring (1–5) is better for tracking a single metric over time. Many teams use both.
How do I validate an LLM judge?
Human-label a sample, measure how often the judge agrees with the humans, and refine the rubric where they diverge. If agreement stays low, the rubric or the judge model is wrong for the task.