Stop guessing at prompts. Learn from production conversations
Use real production conversations to find agent failures, make focused prompt changes, build better evals, and verify that user outcomes improve.
Prompt reviews often begin with a collection of reasonable opinions.
Product wants the agent to be clearer. Support wants fewer apologies. Legal wants stronger boundaries. The founder wants every answer to feel unmistakably on-brand.
The prompt grows. The meeting ends. Nobody checks what users were actually struggling with.
That is not a learning loop. It is group editing with a model at the end.
Prompting from production conversations means finding repeated behavior in real user sessions, turning that evidence into a focused prompt change, testing the change against representative failures, and measuring the result after it ships.
It replaces general advice with a specific reason to edit the prompt.
A prompt review gets complicated quickly when the team has opinions but no production evidence.
Why prompt changes drift away from users
Prompt Engineering is easy to start and surprisingly difficult to maintain.
Early instructions are usually simple. As new problems appear, teams append more rules: be brief, ask clarifying questions, never assume, use this tool first, avoid that phrase, include these caveats, follow this tone.
Months later, the prompt contains layers of decisions without the evidence that produced them. Nobody knows which instruction still helps, which one conflicts with another, or which behavior it was meant to correct.
Common requests such as these make the problem worse:
- Make the answer more useful.
- Ask smarter follow-up questions.
- Do not sound like a bot.
- Avoid hallucinations.
- Be concise but comprehensive.
Each request points toward a desirable outcome. None identifies the production failure that needs to change.
Without evidence, a prompt edit is difficult to evaluate because success was never defined.
What production conversations reveal
Production Conversations provide something a workshop cannot: users attempting real jobs with real constraints.
Their behavior exposes intent signals that surveys and thumbs-down buttons often miss.
| Production behavior | What to investigate |
|---|---|
| The user repeats an ask with different wording | The prompt lacks examples for that intent |
| The agent completes only part of a compound request | Multi-intent planning is weak |
| The user corrects the agent's interpretation | Clarification happens too late or not at all |
| A risky request proceeds without confirmation | The escalation threshold is underspecified |
| The agent repeats reference material after a failed attempt | Recovery instructions are missing |
| The user leaves after a generic answer | The prompt did not produce a useful next action |
Now the team can define a narrow change.
Instead of "make billing answers better," try this:
When a customer asks when a charge will settle, retrieve the account status before explaining policy or linking documentation.
The second version identifies a situation, a required action, and an observable result. It can become both a prompt instruction and an eval case.
Separate prompt failures from system failures
Not every poor conversation needs a new sentence in the system prompt.
The agent may have received stale retrieval context. A tool may have returned an empty result. Session memory may have dropped an important constraint. The user interface may have hidden the action the agent expected the user to take.
Before editing the prompt, inspect the full trace and ask:
- Did the agent receive the information it needed?
- Did it select the right tool?
- Did the tool return a usable result?
- Did the prompt explain how to handle that result?
- Did the response move the user's task forward?
This prevents the system prompt from becoming a patch file for failures caused elsewhere in the product.
Conversation Analytics is most useful when it stays connected to traces, tools, retrieval, memory, and outcomes. The conversation shows the experience. The trace helps explain the cause.
A production-driven prompt workflow
Use this operating loop for AI Agent Improvement:
- Cluster recent sessions by intent and outcome.
- Find a repeated failure with meaningful user impact.
- Read the conversations and their traces together.
- Decide whether the prompt is the correct intervention point.
- Make the smallest instruction or example change that addresses the pattern.
- Replay the failed sessions against the new version.
- Promote representative cases into Agent Evals.
- Release the prompt behind a version or experiment.
- Check whether the production behavior improves.
The final step turns prompt editing into agent improvement.
An offline score can tell you the new prompt handles known examples. Production behavior tells you whether users rephrase less, abandon less, complete more tasks, or trust the agent with more complex work.
The board becomes useful once every clue links back to a real session.
Give every prompt change a source and a prediction
A meaningful prompt diff should carry its own explanation.
Record four things with the change:
| Field | Example |
|---|---|
| Observed pattern | Customers say "undo this charge," but the agent returns policy text |
| Production evidence | The pattern appeared across 14 sessions and led to five abandoned tasks |
| Prompt change | Add reversal language examples and require an account lookup before guidance |
| Expected result | More correct tool calls and fewer policy-only responses |
This makes future review easier. The next person editing the prompt can see why the instruction exists and replay the evidence that justified it.
It also helps teams build self-improving AI agents without pretending the agent should rewrite its own behavior unchecked. Automation can identify patterns and propose a change. People can review the evidence, assess risk, and decide what ships.
Keep changes small enough to learn from
Large prompt rewrites can improve a headline metric while hiding which edit caused the movement. They can also fix one intent and regress several others.
Prefer a sequence of narrow changes:
- Add examples for one missed intent.
- Clarify one tool-selection rule.
- Define one escalation boundary.
- Improve one recovery behavior.
- Remove one instruction that conflicts with a higher-priority rule.
Small changes create cleaner comparisons. Cleaner comparisons produce better Agent Evals. Better evals make the next release safer.
That is the practical path from AI agent analytics to durable prompt quality.
TL;DR
Do not let prompt decisions depend on opinions alone.
Use Production Conversations to find repeated failures. Connect those failures to traces and intent signals. Make a focused change, replay the real examples, add important cases to Agent Evals, and verify the outcome after release.
The objective is not a more impressive prompt document.
The objective is an agent that works better for users.
Close the prompt improvement loop with Currai
Currai turns production sessions and agent events into user stories, intents, violations, alerts, errors, traces, evaluation evidence, and Auto Improve proposals.
That gives product and engineering teams one path from a repeated conversation pattern to the evidence behind it, the prompt or product change it suggests, and the production behavior that proves whether the fix worked.
Connect an application through native HTTP or an existing OpenTelemetry pipeline. No first-party Currai SDK is required. Use the Currai integration skill to instrument the real agent path, or start with Currai free.
FAQ
Should product leaders still review prompts?
Yes. Product judgment and taste still matter. Production evidence gives that judgment a concrete user problem to solve and a result to measure.
How large should a prompt change be?
Keep it small enough to isolate its effect. If one release changes tone, tool selection, escalation, and output structure together, it becomes difficult to explain why a metric moved.
Should every failed conversation become an eval?
No. Prioritize failures that repeat, affect valuable workflows, create material risk, or represent behavior the team never wants to regress.
Is Conversation Analytics enough on its own?
No. Conversations reveal the user's experience, while traces reveal the system behavior behind it. The strongest diagnosis connects both with task outcomes.