Tool retrieval quality is the hidden ceiling on agent performance
Agents cannot use tools they fail to discover. Evaluate retrieval, ranking, schema understanding, and execution separately to find the real limit on tool-calling quality.
An agent with five tools can receive the entire registry in its prompt. An agent with five hundred tools needs retrieval. At that point, tool-calling quality is bounded before the model chooses an action: the right tool must first enter the context.
If retrieval misses, better reasoning cannot recover what the agent never saw.
Split the tool pipeline into stages
Evaluate four distinct decisions:
- Retrieval: did the candidate set contain the necessary tool?
- Ranking: did the right tool appear high enough to receive attention?
- Selection: did the model choose it instead of a plausible alternative?
- Execution: were the arguments valid, authorized, and effective?
A single task-completion score confounds these stages. When a run fails, teams often rewrite the prompt or change the model even though the correct tool was absent from the retrieved set.
Build a tool-discovery dataset
Create tasks labeled with required, optional, interchangeable, and forbidden tools. Include natural user language rather than tool names. Add difficult cases:
- several tools with overlapping descriptions;
- one tool whose name is misleading but description is correct;
- composite tasks requiring tools from different namespaces;
- tools available only under certain permissions or account state;
- a correct answer that requires no tool;
- adversarial tool metadata containing instructions.
Version the registry snapshot with each case. Tool catalogs change frequently, and an evaluation is not reproducible without the exact names, descriptions, schemas, and retrieval index used.
Measure retrieval before end-to-end success
Track recall at k for required tools, forbidden-tool exposure, mean rank, and
candidate-set size. Then measure downstream selection and completion conditioned
on retrieval success.
That conditional view is powerful:
| Pattern | Likely bottleneck |
|---|---|
| Low retrieval recall | Index, query construction, or metadata |
| High recall, low selection | Ambiguous descriptions or model routing |
| Correct selection, invalid arguments | Schema design or context |
| Valid call, wrong outcome | Tool implementation or stale state |
Use traces to retain the retrieval query, candidate scores, presented tool definitions, model decision, arguments, response, and final outcome.
Improve metadata with evidence
Tool descriptions should state purpose, preconditions, side effects, required permissions, and the distinction from nearby tools. Avoid stuffing descriptions with every keyword; that can raise retrieval frequency while making ranking and selection worse.
Run paired tests for each metadata change. Monitor both recall and unnecessary tool exposure. A registry that always retrieves ten vaguely relevant tools may look healthy on recall while increasing confusion, tokens, and risk.
Evaluate dynamic availability
Production catalogs depend on the user, workspace, authentication state, feature flags, and service health. Add cases where the ideal tool is unavailable. The agent should choose a safe fallback, explain the limitation, or request the minimum action needed to continue.
Never retrieve tools the user is not authorized to invoke merely to let the model decide. Authorization belongs in the retrieval and execution layers, not only in the prompt.
Our tool-calling evaluation guide covers argument and task-completion scoring. Pair it with retrieval metrics to evaluate the complete path from user intent to verified outcome.
Watch the production tail
Offline cases cover known intents. Production traces reveal new vocabulary, new tool collisions, and long-tail combinations. Sample low-confidence retrievals, no-tool outcomes, fallback use, and cases where users immediately retry.
Promote representative failures into the discovery dataset. Over time, the tool registry becomes an evaluated interface rather than a pile of descriptions.