How to evaluate web-search agents in production
Compare search agents by answer quality, evidence, freshness, search depth, latency, cost, and recovery on your own research workload.
Adding web search solves one problem—access to current information—and creates several others. The agent must write useful queries, choose sources, decide when to search again, reconcile disagreement, and cite evidence that actually supports its answer.
A single accuracy score cannot explain where that pipeline failed.
Evaluate the complete search configuration
Search quality depends on a combination of decisions:
- The model that plans queries and synthesizes results
- The search engine or native provider search
- Whether results are retrieved before the model call or through a tool
- The number of searches or total search budget
- Whether the agent can fetch full pages
- Stopping and retry behavior
Change one variable at a time when diagnosing a regression. When comparing complete product configurations, record every variable so the result can be reproduced.
Build a workload that resembles user research
Include several task shapes:
| Task | Example behavior to test |
|---|---|
| Fact lookup | Find one current, verifiable answer |
| Multi-hop research | Combine facts from several sources |
| Broad collection | Fill a table without missing entities |
| Comparison | Apply consistent criteria across options |
| News synthesis | Separate event time from publication time |
| Ambiguous query | Ask for clarification or search alternatives |
Freeze expected answers only when the fact is stable. For changing questions, evaluate evidence quality, source freshness, and reasoning against a timestamped reference set.
Score evidence separately from prose
A fluent answer can cite irrelevant pages. Evaluate each important claim:
- Is there a citation?
- Does the linked source support the claim?
- Is the source authoritative enough for the decision?
- Is the information current?
- Did the answer represent uncertainty and disagreement honestly?
Measure unsupported-claim rate and citation precision alongside answer accuracy. For high-stakes research, require primary sources where available.
Search depth is a product decision
More search turns can improve difficult research while increasing latency and cost. They can also make a weak planner wander through redundant queries.
Compare one, several, and larger search budgets on the same cases. Plot quality against total cost and wall-clock time. Select a budget by task class rather than giving every request the maximum allowance.
Track worst-case cost as well as the mean. Repeated failed searches often drive the expensive tail.
Test failure and recovery
Production search agents encounter rate limits, blocked pages, duplicated results, stale snippets, contradictory sources, and prompt injection inside web content. Include these conditions deliberately.
The desired behavior may be to change the query, switch sources, reduce the claim, or tell the user that reliable evidence was not found. Inventing an answer is never a valid recovery strategy.
Monitor search behavior after launch
Offline benchmarks cannot anticipate every emerging query. In production, watch repeated searches, abandoned tasks, low-quality domains, missing citations, user corrections, cost spikes, and long-running sessions.
Open the trace behind each outlier. Search queries and result sets often explain the failure faster than the final answer does.
Evaluate search agents with Currai
Currai connects user intent, search calls, sources, model synthesis, citations, cost, latency, and outcome. Teams can group recurring research failures and turn them into evaluation cases grounded in real usage.
Instrument with the Currai integration skill.