Why model routing is an evaluation problem
A router is only as good as the workload and evaluators used to train, tune, and verify its quality-cost decisions.
Model routing is often presented as infrastructure: classify a request, choose a model, and send the call. But the routing rule contains a product judgment. It decides how much quality is sufficient for each piece of work.
That makes routing an evaluation problem before it becomes an optimization problem.
You cannot optimize an undefined outcome
A router can minimize token price and still make the product worse. It can maximize a generic benchmark score while overspending on routine work. The objective needs application-specific measures such as task completion, groundedness, policy adherence, tool correctness, latency, and user repair.
Only after quality is measurable can the team ask which model mix achieves the required result at the lowest total cost.
Build a routing matrix from the workload
Segment representative cases by meaningful task properties:
- Intent or workflow
- Difficulty
- Required capabilities
- Risk and reversibility
- Language or modality
- Context length
- Expected tool trajectory
Run every eligible model against every case under the same harness. The result is a matrix of quality, latency, reliability, and cost—not one leaderboard column.
Find the quality-cost frontier
A configuration is dominated when another option is both better and cheaper. The remaining configurations form a Pareto frontier: improving quality from one point requires paying more, while reducing cost requires accepting a measured tradeoff.
The product team chooses where to operate on that frontier. A low-risk content tag may tolerate a small quality reduction. A refund, compliance decision, or healthcare workflow may not.
Evaluate the routing decision itself
End-to-end scores tell you whether the system worked. Router diagnostics explain why it selected a model. Track:
| Signal | Question |
|---|---|
| Classification accuracy | Did the router recognize the task? |
| Regret | How far was the choice from the best eligible model? |
| Over-routing | Did it pay for capability the case did not need? |
| Under-routing | Did a weak model cause failure or repair? |
| Stability | Does a small wording change produce a wild model change? |
| Coverage | How often does the policy fall back to a default? |
Inspect mistakes by task group. An acceptable average can conceal systematic under-routing for one language or customer segment.
Include the agent harness
Model performance changes with prompts, tools, retrieval, memory, and runtime policy. Compare routing configurations inside the real harness. A model that wins with one tool description may fail after the schema changes.
Version the router, candidate set, prompts, and evaluators together. Rerun the suite when any of them changes.
Learn from production without surrendering control
After launch, collect route decisions and outcomes. Use real failures to improve the evaluation set, but do not optimize blindly on engagement or aggregate success. Preserve safety floors, audit changes, and review segments where the router has limited data.
Shadow evaluation is useful: calculate what a new policy would have selected without changing the model that serves the user. Compare predicted cost and quality before rollout.
Evaluate routing with Currai
Currai connects router decisions to the full agent trace and the user outcome. Teams can measure cost per successful task, compare policies, find under-routed workflows, and turn routing failures into permanent regression cases.
Instrument with the Currai integration skill.