Reusable LLM evaluator templates: build once, run everywhere
Design reusable LLM evaluators with stable rubrics, typed inputs, calibration examples, and versioning that work in development and production.
Reusable LLM evaluator templates turn a one-off scoring prompt into a quality contract that multiple teams can run against experiments and production traces. The template defines what data it expects, what the score means, how ambiguous cases are handled, and which version produced each result. Reuse is valuable only when those semantics remain stable.
The alternative is evaluator sprawl. One team calls a score correctness,
another calls a different rubric accuracy, and a third silently changes a
prompt without recalculating historical results. The dashboard looks precise,
but its numbers cannot be compared.
What belongs in an evaluator template?
A production-ready evaluator needs more than a judge prompt.
| Component | Purpose |
|---|---|
| Input contract | Names the trace fields or dataset columns the evaluator reads |
| Rubric | Defines each criterion in observable language |
| Scale | Explains every allowed score and its direction |
| Examples | Anchors good, bad, and borderline judgments |
| Failure policy | Says what happens when inputs are missing or malformed |
| Metadata | Records evaluator name, version, model, and prompt version |
Start with one narrow question. “Did the answer cite only retrieved evidence?” is testable. “Was the answer good?” mixes correctness, style, safety, and relevance into a score nobody can diagnose.
A reusable judge template
The evaluator should accept normalized fields rather than application-specific objects. A useful contract for groundedness is:
The judge rubric can then define 0 as materially unsupported, 1 as mixed or
unclear, and 2 as fully supported. Asking for evidence in unsupportedClaims
makes the result useful during debugging instead of producing a number alone.
Keep judge temperature low, require structured output, and treat parse failure as an evaluator error rather than a failing application example. Otherwise model or schema failures quietly depress the product-quality metric.
Calibrate before reuse
Reusable does not mean universally valid. Build a calibration set containing clear passes, clear failures, and disagreements that domain experts actually care about. Have humans label it before the automated evaluator sees the answers. Then measure agreement and inspect confusion by score—not just one aggregate percentage.
The LangSmith evaluation concepts recommend starting with manually curated examples of good behavior for each critical component. The same principle applies regardless of platform: examples define the standard more reliably than an adjective-filled prompt.
If the judge disagrees with humans systematically, fix the rubric or examples. Do not tune until it agrees with a tiny set and declare victory; keep a held-out calibration slice to detect overfitting. Our guide to human-in-the-loop agent evaluation shows how to route uncertain cases back to reviewers.
Version evaluators like application code
Every material rubric, model, or output-schema change creates a new evaluator version. Store the version beside every result. When you upgrade, rerun both versions on a shared dataset and compare:
- Human agreement.
- Score distribution.
- Failure explanations.
- Cost and latency per evaluation.
- Sensitivity to prompt injection inside the evaluated content.
Never overwrite historical scores with new semantics. A chart that joins two rubrics under one metric name creates false quality drift.
Run the same contract offline and online
Offline evals compare candidate prompts, models, or agent designs before release. Online evals sample real production traces after release. The evaluator contract should be identical across both paths; only the source and sampling policy change.
In Currai, keep the trace input, output, retrieved context, tool results, model, and prompt version together. That record lets the same evaluator consume a dataset example during development and a production trace after deployment. See the AI agent evaluation stack for the full trace-to-dataset loop.
Reuse without losing context
Templates work best for criteria shared across products: groundedness, policy compliance, tool-call validity, output format, and refusal behavior. Product- specific task success usually needs a configured variant or separate evaluator.
The goal is not the fewest evaluator files. It is a small library of clearly named, calibrated quality contracts whose results remain comparable. Build the contract once, version it deliberately, and connect every score back to the trace that explains it.