LLM chatbot evaluation: metrics and conversation testing
Chatbots fail across a conversation, not in a single reply. Here are the metrics that actually catch chatbot failures — and why turn-level scoring alone always misses them.
TL;DR: Evaluating a chatbot one reply at a time misses how chatbots actually break — they lose track of context, contradict earlier turns, forget constraints, and fail to complete the task the whole conversation was for. Real chatbot evaluation scores both individual turns and the conversation as a unit, and does it on production dialogues, not just scripted ones.
A chatbot is a multi-turn system, but most evaluation treats it like a single-turn one: send a question, grade the answer. That catches obvious wrong replies and nothing else. The failures that make users abandon a chatbot live between turns. This guide covers the metrics that catch them.
Turn-level metrics (necessary, not sufficient)
For each individual reply, the durable ones are:
- Relevance — does the reply address what was just asked?
- Faithfulness / groundedness — is it supported by the knowledge base rather than made up? (See RAG evaluation metrics.)
- Correctness — is the information actually right?
- Tone / safety — appropriate, non-harmful, on-brand.
These matter, but a chatbot can pass all of them on every turn and still fail the conversation.
Conversation-level metrics (where chatbots actually fail)
- Context retention — does turn 8 remember what the user said in turn 2?
- Consistency — does it contradict something it said earlier?
- Constraint adherence — does it hold a limit stated once ("I'm vegetarian," "budget is $500") across the whole dialogue?
- Task / goal completion — did the conversation, as a whole, accomplish what the user came for?
- Conversational efficiency — did it get there without needless loops or re-asking for information already given?
Scoring these requires evaluating the dialogue as a unit, which is a different job from grading a reply. (See multi-turn LLM evaluation and evaluating multi-turn support conversations.)
Metric map
| Level | Metric | Catches |
|---|---|---|
| Turn | Relevance, faithfulness, correctness | Wrong or ungrounded single replies |
| Conversation | Context retention, consistency | Forgetting and self-contradiction |
| Conversation | Constraint adherence | Dropping a stated limit mid-dialogue |
| Conversation | Task completion | Pleasant chats that never solve anything |
Test scripted dialogues, then real ones
Scripted multi-turn test cases catch regressions before you ship — but they only cover conversations you thought to write. The failure modes that hurt come from real users going off-script: unexpected topic switches, ambiguous references, frustration. So the evaluation loop has two halves:
- Offline — a suite of multi-turn scenarios scored on every prompt/model change.
- Production — score real conversations continuously, and feed the ones that went wrong back into the offline suite. (See run LLM evals on production traces.)
Because open-ended dialogue rarely has a golden answer, most of this scoring is done with LLM-as-a-judge against a rubric.
How Currai fits
Currai traces each chatbot conversation as a connected session — every turn, retrieval, and tool call in order — so you can score both the individual replies and the dialogue as a whole. Run conversation-level rubrics (retention, consistency, task completion) against production chats, watch them across prompt and model changes, and promote any failed real conversation into a test case. See the AI agent evaluation guide and traces and evals in one place, or start with Currai free.
Frequently asked questions
How is chatbot evaluation different from evaluating a single LLM call?
A single call is graded on one output. A chatbot must also be graded across the conversation — context retention, consistency, constraint adherence, and whether the whole dialogue completed the user's task. Turn-level metrics alone miss all of that.
What are the most important chatbot evaluation metrics?
Turn-level: relevance, faithfulness, correctness. Conversation-level: context retention, consistency, constraint adherence, and task completion. The conversation-level ones catch the failures users actually complain about.
Can I evaluate chatbots without golden answers?
Yes — open-ended dialogue usually has no single correct reply, so scoring is done with an LLM-as-a-judge against a rubric that defines what a good turn and a good conversation look like.
Should I test scripted or real conversations?
Both. Scripted multi-turn scenarios catch regressions before release; real production conversations catch the off-script failures your scripts never imagined. Feed the real failures back into your scripted suite.