Why AI agents pass evals but disappoint users in production

An AI agent can pass every pre-release eval and still disappoint users in production because the test suite measures known behavior under controlled conditions. Real sessions introduce unfamiliar intents, incomplete context, changing account state, unreliable tools, long interaction chains, and outcomes that cannot be judged from the final response alone.

The green score is not necessarily wrong. It is incomplete.

Teams get into trouble when they interpret "the agent passed our examples" as "the product works for our users." Closing that gap requires two quality loops: repeatable AI Evals before release and AI Agent Monitoring on real sessions after release.

What a passing eval score actually tells you

An eval is a controlled question with a defined scoring rule. It can establish that a prompt change still produces valid JSON, an agent selects the expected tool, a support answer follows policy, or a model refuses a prohibited request.

Those are valuable release gates. They prevent known failures from returning.

But every eval result is conditional on four choices:

1. The examples: which user requests entered the dataset.
2. The context: which documents, account state, memory, and tool responses were provided.
3. The judge: which dimensions and thresholds counted as success.
4. The boundary: whether the test scored one answer, one tool call, or the complete user journey.

A passing result applies inside that boundary. Production expands all four.

The four gaps between AI Evals and Production AI

1. The intent coverage gap

Eval datasets are strongest on known jobs. Production users introduce new goals, combine several requests, omit important details, use unexpected vocabulary, and change direction in the middle of a session.

A billing eval might test "When will my refund arrive?" A real user may say, "I closed the account last week and the card is gone, so where is the money going?" The second request mixes refund timing, account state, payment routing, and anxiety. It is not merely a paraphrase.

Intent signals reveal this gap. Rephrasing, clarification loops, repeated requests, and new clusters of similar questions show where the production distribution has moved beyond the dataset.

2. The context gap

Offline cases usually provide clean, sufficient context. Production context is messy:

• The knowledge base contains two policy versions.
• A customer has permissions in one workspace but not another.
• Memory preserved an old preference after the user changed it.
• Retrieval returned a relevant passage from the wrong product tier.
• A tool returned partial data that still looked plausible.

The model can behave correctly given the wrong context and still produce a harmful result. Response-only scoring often attributes the failure to the model when the root cause was retrieval, identity, memory, or data freshness.

3. The workflow gap

Many agent failures emerge across steps rather than inside one step. A slightly wrong interpretation changes the search query. The weaker retrieval changes the plan. The plan calls a tool with incomplete arguments. A retry repeats the action. The final response may still sound confident.

Component evals can pass while the assembled workflow fails. Agent Reliability therefore requires end-to-end traces containing prompts, retrieval, tool calls, retries, state transitions, errors, and the final result.

Read three ways AI systems fail even when evals pass for more on distribution shift and compounding behavior.

4. The outcome gap

An answer can satisfy the rubric without completing the user's job.

The agent may provide correct setup instructions, but the user cannot find the setting. It may create a support ticket, but omit the diagnostic details the human needs. It may recommend the right plan, but fail to recognize that the user is blocked on a security review.

These are product failures, even when response quality is acceptable. They become visible through Conversation Analytics: what the user did next, whether they corrected the agent, whether the task continued, and whether the same intent returned later.

Production signals a test set cannot manufacture reliably

Production signal	Likely interpretation	What to inspect
User repeats the request with new wording	The agent missed or only partly resolved the intent	Conversation, intent classification, retrieved context
User corrects a fact or constraint	Context, memory, or reasoning was wrong	Trace and source evidence
User asks whether the task is finished	Progress or completion state was unclear	Plan, tool results, final response
User performs the work manually	The agent was slower, less trusted, or incomplete	Workflow length and human repair
User abandons after a plausible answer	The response may not have enabled the next step	Outcome and subsequent session behavior
Same account returns with the same problem	The earlier resolution did not hold	Cross-session intent history
Human reverses an agent action	The tool call was unsafe or incorrect	Authorization, arguments, confirmation, audit log
High-value question becomes generic support	Buying or expansion intent was missed	Intent signals and routing

No single signal proves failure. A user may leave because the answer solved the problem. The useful unit is a pattern across comparable sessions, intents, user segments, and outcomes.

Why adding random eval cases does not solve the problem

When production quality drops, the instinct is to make the offline suite bigger. Volume alone does not create representativeness.

A thousand synthetic cases can repeat the same assumptions. A hundred variations of one clean support question do not test an expired session, a stale policy, a slow dependency, or a user who changes goals after the first tool call.

Add a case when it protects a named behavior or reproduces a real failure. Keep a small, deliberate release suite that engineers can understand, then complement it with sampled evaluation on production traces.

A three-dataset evaluation strategy

Release-gate dataset

This is the stable set run on every important prompt, model, retrieval, tool, or harness change. It covers required behavior, high-severity risks, and previously fixed regressions.

The set should be deterministic enough to block a bad release and small enough to diagnose when it fails.

Fresh production sample

Sample recent sessions by intent, customer segment, language, model version, and workflow. Score end-to-end outcomes as well as individual response dimensions.

This dataset shows whether quality holds on current traffic. Keep the sample fresh so yesterday's user distribution does not become another frozen benchmark.

Incident and edge-case set

Promote recurring or high-severity production failures into a dedicated set. Keep the original context, tool behavior, and expected recovery path whenever possible.

This is how agent improvement accumulates: production discovers the case, the trace explains it, the fix changes the system, and the incident becomes a durable guardrail.

Metrics for AI Agent Monitoring after launch

Keep eval pass rate, but pair it with metrics tied to real behavior.

Metric	Definition	Why it matters
Intent resolution rate	Share of sessions where the user's inferred goal was completed	Measures useful outcomes rather than fluent answers
Correction rate	Share of sessions where users correct facts, scope, or actions	Exposes misunderstood context and low trust
Repetition rate	Share of sessions where users restate the same intent	Reveals partial or failed resolution
Correct escalation rate	Share of risky or unsupported cases routed appropriately	Measures whether human fallback protects users
Tool success rate	Valid, authorized actions completed with the intended result	Separates workflow quality from response quality
Human repair rate	Agent outputs or actions that a person must redo or reverse	Captures hidden operational cost
Repeat unresolved intent	Same user or account returns with the same unresolved goal	Tests whether a resolution held over time
Cost per resolved intent	Total model and tool cost divided by useful resolutions	Prevents cheap failures or expensive successes from hiding

Segment the metrics. An overall average can stay flat while a new customer cohort, language, workflow, or high-value intent degrades.

How to diagnose a green-eval, red-production incident

Start from the failed user outcome and move inward.

1. Reconstruct the user goal. Read the full conversation and identify what the person was trying to accomplish, including later corrections.
2. Locate the divergence. Find the first point where the agent's interpretation, context, plan, or action moved away from that goal.
3. Classify the root cause. Separate prompt behavior from retrieval, memory, tool, permission, UI, or product-policy failures.
4. Measure the pattern. Search for similar intents and traces to determine frequency, affected segments, and severity.
5. Fix the correct layer. Do not add prompt text to compensate for a broken API or stale knowledge source.
6. Create a regression case. Preserve enough context to reproduce the failure.
7. Verify in fresh production traffic. Confirm the correction and repetition rates fall for the affected intent after release.

The last step matters. Passing the new regression test proves the fix works on the captured case. Production confirms whether it improved the user experience.

The operating loop for self-improving AI agents

A self-improving agent is not one that edits itself without review. It is a product system that turns production evidence into controlled, testable changes.

The loop is:

1. Capture sessions and agent events.
2. Group conversations by user intent.
3. Detect friction, violations, errors, and unresolved outcomes.
4. Rank patterns by frequency, severity, user value, and repair cost.
5. Trace the root cause to the prompt, knowledge, tool, harness, or product.
6. Propose a focused change and add a representative eval.
7. Review, ship, and monitor the affected production cohort.

This loop connects AI agent analytics with engineering work. A dashboard is only useful when it changes what the team fixes next and proves whether that fix helped.

How Currai fits

Currai understands what a custom AI application is supposed to do, then automatically surfaces production failures the team did not know to look for. Applications send sessions and agent events through native capture endpoints or OpenTelemetry, allowing Currai to connect user intent with retrieval, model calls, tools, errors, latency, cost, and the final outcome.

Teams can discover unknown failure patterns, inspect the complete agent path, turn representative production behavior into evaluation coverage, and create reviewable improvement proposals. Currai is not another observability dashboard; it shortens the path from an unexpected production failure to a verified product change.

See run LLM evals on production traces, prompting from production conversations, and the AI agent evaluation guide.

Frequently asked questions

Why does an AI agent fail in production after passing evals?

Because evals cover selected examples, context, judges, and workflow boundaries. Production introduces unfamiliar intents, changing state, unreliable tools, multi-step interactions, and user outcomes that may not be visible in the final response.

Are offline AI Evals still useful?

Yes. Offline evals are essential for release gates and regression protection. They become misleading only when a team treats them as a complete measure of production quality.

What should AI Agent Monitoring measure?

Measure intent resolution, corrections, repeated requests, correct escalation, tool success, human repair, repeat unresolved intents, latency, and cost per resolved intent. Segment results by intent and user cohort.

How do production failures improve an eval suite?

Use the conversation to reconstruct the user goal, use the trace to locate the root cause, fix the correct layer, and preserve the representative failure as a regression case. Then verify the affected outcome on fresh production sessions.

What are self-improving AI agents?

In practical production systems, self-improving AI agents use monitored user sessions to identify recurring failures, propose focused changes, add eval coverage, and verify improvements after human review and release.

Close the production feedback loop with Currai

Currai understands intended agent behavior and automatically finds failures teams did not know to test for. It connects those failures to the conversation and agent execution so teams can create a reviewable fix and verify the result after release. Use Currai's integration skill or start with Currai free.