How to evaluate multi-turn customer support conversations in Currai

Single-turn evals are useful, but customer support rarely fits into one message. Users clarify, push back, change details, ask follow-up questions, and expect the assistant to remember what already happened.

That is why multi-turn support conversations need their own eval strategy.

Evaluate the conversation, not just the last answer

A final response can look correct in isolation and still fail the conversation. Maybe the assistant asked for information the user already provided. Maybe it changed its answer between turns. Maybe it answered a refund question correctly after three vague replies that should have been avoided.

Currai traces can preserve the full session context: user messages, assistant responses, prompt versions, retrieval steps, tool calls, latency, cost, and metadata. That lets the eval judge the behavior users actually experienced.

Use support-specific rubrics

Good customer support evals should map to the policies and workflows your team cares about. Common rubrics include:

• Policy accuracy: did the assistant explain the correct policy?
• Context retention: did it use information from earlier turns?
• Next step clarity: did the user know what to do next?
• Escalation: did the assistant hand off when required?
• Consistency: did the assistant avoid contradicting itself?
• Tone: was the response clear, calm, and appropriate?

For a refund workflow, the rubric might require the refund window, eligibility, required order details, and review timeline. If any of those are missing, the conversation should fail the eval even if the answer sounds polite.

Inspect traces before automating

Do not start by writing a giant judge prompt. Start by reading conversations.

Open a sample of support traces in Currai and write down repeated failures. Maybe the assistant misses cancellation details. Maybe it gives generic refund answers. Maybe it fails to escalate billing disputes. Those patterns become the evals.

This avoids a common mistake: scoring what is easy to score instead of what actually hurts users.

Compare prompt versions on real support traffic

Once the eval exists, use it to compare prompt versions or experiment arms. A new prompt might improve policy accuracy but increase verbosity. A model change might reduce latency but miss more escalations. A retrieval update might improve groundedness while increasing cost.

Currai keeps quality scores next to trace context, so the team can inspect the tradeoff rather than treating the eval score as a black box.

Keep humans in the loop

Support evals benefit from domain experts. A support lead often knows the edge cases better than the engineering team. They can review failing traces, adjust rubrics, and decide which failures matter most.

The goal is not to automate support judgment away. The goal is to make that judgment repeatable across real production conversations.

Related: Sessions and users, LLM evals, and PMs should own the AI eval loop.