How to detect silent regressions in an agent harness
Context trimming, tool edits, retry policies, and model-specific tuning can degrade agents without breaking tests. Use paired trajectory evals to catch silent harness regressions.
Agent harness changes rarely fail like ordinary code. A shorter context policy may reduce cost while quietly dropping a constraint. A renamed tool may still work but route less accurately. A retry policy may raise completion rate by doubling latency and side effects.
Nothing crashes. The agent simply becomes worse in ways users notice later.
Version the complete harness
Give every run a harness version that resolves to the exact:
- system and task instructions;
- context assembly and summarization policy;
- tool names, descriptions, schemas, and permissions;
- model settings and provider adapter;
- retry, timeout, budget, and stopping rules;
- memory and checkpoint implementation;
- environment and evaluator versions.
A source commit alone is often insufficient when prompts, tools, or feature flags live elsewhere. Store their resolved hashes with the trace.
Use paired evaluation
Run baseline and candidate harnesses on identical task inputs and starting state. Repeat stochastic cases and compare each pair, not just two overall averages.
Measure outcome, trajectory, resources, and control:
| Layer | Example metrics |
|---|---|
| Outcome | Correctness, task completion, policy compliance |
| Trajectory | Tool choice, unnecessary steps, context use |
| Resources | Tokens, cost, wall time, retries |
| Control | Correct stopping, approval, escalation, recovery |
A candidate should not be promoted because one top-line score improved. Define guardrails for dimensions that must not regress, especially safety and tail cost.
Create change-focused probes
The regression suite should include broad representative tasks and small probes for the surface being changed.
For a context-policy update, test late constraints, conflicting instructions, facts needed after many turns, and references recovered through retrieval. For a tool-description update, test collisions with neighboring tools and no-tool cases. For retry changes, inject transient and permanent failures and verify idempotency.
Probes diagnose the mechanism; end-to-end tasks show whether the mechanism matters to users. Keep both.
Inspect regressions through traces
Aggregate metrics tell you where to look. Paired traces explain the movement. Compare the first divergence: different retrieved context, tool candidate set, model choice, argument, error handling, or stopping decision.
Record a failure class rather than patching one example. A missing constraint may indicate summarization loss across many tasks. Add the discovered class to a held-out regression set so the fix does not simply memorize the visible case.
Our agent harness engineering guide describes the broader hill-climbing loop. The key regression practice is making every candidate compete against the current production harness on the same work.
Shadow before promoting
Offline suites cannot represent the full production distribution. Run the candidate in shadow mode on sampled live inputs where privacy and cost policy allow. It should not execute external side effects. Compare proposed trajectories and evaluator results with production.
Watch slices that averages hide: long conversations, uncommon tools, large repositories, low-resource languages, high-latency providers, and accounts with restricted permissions.
Roll out with a reversible identity
Release the harness behind a versioned flag. Monitor outcome and trajectory metrics by version, retain a fast rollback, and keep traces from both cohorts. When a regression appears, you should be able to name the affected version and first divergent decision without reconstructing configuration from memory.