Your AI agent passes evals. That's the problem
A passing eval score is a false sense of security when the eval set is narrow and static. Here's why agents that ace offline evals still fail in production — and the fix.
TL;DR: A green eval score tells you the agent handled the cases you thought to test — not the cases production will throw at it. Static, hand-crafted eval sets drift out of sync with real traffic, miss the long tail, and reward overfitting. The fix isn't more offline cases; it's evaluating on production traces, so your eval set is a moving reflection of reality instead of a snapshot of your assumptions.
Here's an uncomfortable pattern: a team builds an eval set, tunes their agent until it passes, ships with confidence — and then production is full of failures the evals never caught. The passing score wasn't a lie, but it answered a narrower question than the team thought. It said "the agent handles these cases," and quietly implied "...and therefore production," which doesn't follow.
This is the trap of treating a passing eval as proof of quality. Let's look at why it happens and what to do instead.
What a passing eval actually proves
A green eval score proves exactly one thing: the agent handled the specific inputs in your eval set, judged by the specific rubric you wrote. That's genuinely useful — it catches regressions and gives you a baseline. But it does not prove the agent handles production, for three reasons.
Reason 1: your eval set is a snapshot of your assumptions
You wrote the eval cases based on what you imagined users would do. Real users are more creative, more careless, and more varied than any team's imagination. The cases you didn't think to write are exactly the ones that fail in production — and a passing score on the cases you did think of tells you nothing about them.
Reason 2: static eval sets drift
Even a good eval set decays. Your product changes, your users change, the world changes — and a frozen eval set keeps testing yesterday's reality. Six months later you're acing a benchmark that no longer represents the traffic you actually serve. The score stays green while relevance quietly drains away.
Reason 3: tuning to the eval set is overfitting
When you tune an agent until it passes a fixed set, you risk optimizing for that set specifically rather than for general quality — the classic overfitting failure. The agent gets very good at your test cases and no better (sometimes worse) at everything else. A high score on the set you optimized against is the least trustworthy score of all.
The compounding problem with agents
Agents make this worse. An agent is a multi-step loop, and offline eval cases tend to test single, clean scenarios. But production failures often come from compounding — a slightly-off decision at step two that snowballs by step eight, or a rare tool error that only surfaces on real data. Clean offline cases rarely reproduce the messy, stateful conditions where agents actually break. (See three ways AI systems fail even when evals pass.)
The fix: evaluate on production traces
The problem isn't evaluation — it's evaluating against a static snapshot instead of living reality. The fix is to move evaluation onto production traffic:
- Trace production. Capture real agent runs — inputs, steps, tool calls, outputs.
- Evaluate those traces. Run your rubric on real traffic, not just the offline set, so you score what actually happened. (See run LLM evals on production traces.)
- Mine failures. The low-scoring production traces are the cases your offline set was missing — the real long tail.
- Feed them back. Turn those real failures into new eval cases, so your offline set keeps catching up to reality. (See turn production traces into better AI.)
Now your eval set is a moving reflection of production, not a frozen snapshot of your assumptions — and a passing score means something closer to what you wanted it to mean.
How to read your own green score
When your agent passes evals, ask:
- How old is the eval set? If it's static and months old, distrust the score.
- Where did the cases come from? Imagination or real traffic?
- Are you evaluating production too, or only the offline set?
- Did you tune against this exact set? If so, the score is optimistic.
A passing score is a starting point for these questions, not the end of the conversation.
How Currai fits
Currai closes the gap between "passes evals" and "works in production" by putting evaluation on real traffic. It traces every agent run, scores production traces against your rubric, surfaces the low-scoring long tail your offline set missed, and turns those failures into test cases. Your eval set stops being a snapshot of your assumptions and becomes a living reflection of what users actually do. See traces and evals in one place and the definitive AI agent evaluation guide, or start with Currai free.
Frequently asked questions
If my agent passes evals, why does it still fail in production?
Because a passing score only proves the agent handled your specific eval cases — which reflect your assumptions, not the full variety of real traffic. The failures that hurt are usually cases you never thought to test.
Does this mean offline evals are useless?
No. Offline evals catch regressions and give a baseline — that's valuable. The mistake is treating a passing offline score as proof of production quality. Complement offline evals with evaluation on production traffic.
What does it mean to evaluate on production traces?
Capturing real production runs and running your evaluation rubric on that actual traffic, rather than only on a static offline set. This scores what really happened and surfaces the long-tail failures your offline cases missed.
How do I stop overfitting to my eval set?
Don't rely on a single static set you tune against. Continuously add real failures from production, evaluate on live traffic, and treat a high score on the exact set you optimized against as optimistic until confirmed on fresh data.