Evaluating LLM tool-calling and task completion
An agent can call the right tool and still fail the task — or complete the task despite a clumsy path. Here's how to score tool-calling and task completion as separate metrics, so a failure points to a fix.
TL;DR: Agent evaluation splits into two questions that people conflate: did the agent use its tools correctly (selection, arguments, execution) and did it complete the task? These are separate metrics — an agent can nail every tool call and miss the goal, or reach the goal despite a wrong turn. Scoring them apart is what turns "the agent failed" into "the agent picked the wrong tool on step three."
When an agent fails, "it didn't work" is useless. The value is in knowing where it broke — and that requires scoring tool use and task outcome as distinct things. This post breaks agent evaluation into the layers that actually localize a failure. (For the broader picture, see the AI agent evaluation guide.)
The two questions, kept separate
- Tool-calling correctness — a process metric: did the agent select the right tool, with the right arguments, and handle the result?
- Task completion — an outcome metric: did the agent, end to end, actually accomplish what the user asked?
They diverge constantly. An agent can execute a flawless sequence of tool calls that adds up to the wrong answer, or stumble through a messy path and still land the goal. Score only completion and you can't debug; score only tool use and you miss whether any of it mattered.
Breaking down tool-calling
Tool-calling itself has layers, each a separate failure point (this is the same decomposition as MCP evaluation):
- Selection — did it pick the right tool for the step? Measure correct-tool rate, false-trigger rate (called a tool when none was needed), and miss rate (skipped a tool it needed).
- Arguments — given the right tool, were the arguments well-formed and correct? Watch for missing fields, wrong types, and hallucinated values.
- Execution & result handling — did the call succeed, and did the agent handle errors and empty results sensibly instead of plowing ahead on garbage?
Poor selection usually traces to unclear tool names or descriptions — a definition fix, not a model fix.
Scoring task completion
Completion needs an end-to-end judgment: given the user's goal and the agent's final state, was the task done correctly? Open-ended goals have no golden answer, so this is typically scored with an LLM-as-a-judge against a rubric that defines "done." For multi-step goals, also watch efficiency — did it complete in a reasonable number of steps, or loop and backtrack? (Steps cost tokens and latency — see testing agent cost-efficiency.)
You can't score this without traces
Every metric above lives inside the run: the sequence of tool calls, their arguments, their results, and the reasoning between them. A log of just the final answer hides all of it. You need the full trace of the agent's path to attribute a failure to a specific step. (See observability for AI agents and tool calls.)
The metric map
| Layer | Type | Metric |
|---|---|---|
| Selection | Process | Correct-tool / false-trigger / miss rate |
| Arguments | Process | Argument correctness / schema validity |
| Execution | Process | Execution success, error-recovery rate |
| Completion | Outcome | Task-completion rate |
| Efficiency | Outcome | Steps to completion, loop/backtrack rate |
Read them together: process metrics tell you where it broke, the outcome metric tells you whether it mattered.
How Currai fits
Currai traces each agent run as the full sequence — every tool selected, its arguments, its result, and the outcome — so you can score tool-calling and task completion as separate metrics and pin a failure to the exact step. Run those scores on production traffic, watch them across prompt and model changes, and turn failed runs into test cases. See the AI agent evaluation guide and traces and evals in one place, or start with Currai free.
Frequently asked questions
What's the difference between tool-calling and task-completion metrics?
Tool-calling is a process metric — did the agent select the right tool, with the right arguments, and handle the result? Task completion is an outcome metric — did the agent actually accomplish the user's goal end to end? An agent can pass one and fail the other.
How do I measure tool selection?
With correct-tool rate (did it pick the expected tool), false-trigger rate (called a tool when none was needed), and miss rate (skipped a tool it needed). Poor selection usually means unclear tool names or descriptions.
How do I score task completion without a golden answer?
With an LLM-as-a-judge against a rubric that defines "done" for the task, since open-ended goals rarely have a single correct final state. Pair it with an efficiency metric for multi-step tasks.
Why do I need traces to evaluate agents?
Because every agent metric lives inside the run — the tool calls, arguments, results, and reasoning between steps. A log of just the final answer can't tell you which step failed, so you can't attribute or fix the failure.