Measure human repair, not just agent success
An agent-generated result is not productive if a human must spend an hour repairing it. Add review, correction, rerun, and cleanup effort to your evaluation scorecard.
An agent finishes a task, the tests pass, and the dashboard records a success. Then an engineer spends forty minutes removing unrelated edits, correcting an edge case, and rewriting the summary before the change can ship.
The agent succeeded according to the benchmark and failed according to the team. Human repair is the missing variable.
Define the full unit of work
The useful outcome is not an agent artifact. It is an accepted artifact. Measure the workflow from delegation to acceptance, including review, corrections, reruns, and cleanup.
For each task, capture:
- agent runtime and cost;
- reviewer time to understand the result;
- time spent correcting or reverting it;
- number of feedback rounds and reruns;
- defects found before and after acceptance;
- whether the human completed the task instead;
- elapsed time until the result was usable.
Do not count normal approval time as a failure. The goal is to separate expected governance from avoidable repair caused by poor agent work.
Classify repair instead of collecting one duration
A single repair-time metric cannot tell you what to fix. Label the intervention:
| Repair class | Typical cause |
|---|---|
| Requirement repair | Missed constraint or wrong interpretation |
| Functional repair | Incorrect behavior or incomplete implementation |
| Scope repair | Unrelated or overly broad changes |
| Quality repair | Maintainability, style, or architecture mismatch |
| Safety repair | Permission, privacy, security, or compliance issue |
| Evidence repair | Missing tests, inaccurate summary, unsupported claim |
| Environment repair | Setup or dependency changes needed to verify work |
Attach each label to the trace segment that created the problem. That turns a vague complaint—"the agent creates work"—into an actionable harness, tool, or instruction issue.
Use quality-adjusted productivity
Report completion rate beside repair burden. One simple operational measure is:
accepted tasks / (agent cost + normalized human repair cost)
The exact formula depends on your organization. More important is keeping the components visible. Never hide a safety intervention inside an average dollar value, and do not assume every minute of expert review is interchangeable.
Useful companion metrics include clean acceptance rate, median repair minutes, p90 repair minutes, reruns per accepted task, and percentage of cases where the human takes over. Slice them by task type, model, tool, repository, and reviewer.
Capture review without creating surveillance
Measure the work product, not keystrokes. Let reviewers select a repair class, record active review time coarsely, and link the accepted diff or artifact. Make the purpose explicit: improving the system, not scoring individual employees.
Calibrate labels with a small shared set so "minor" means roughly the same thing across reviewers. Periodically double-review samples to estimate disagreement.
Build repair into offline evals
Production repair data can seed realistic regression cases. Select examples with high repair cost, remove sensitive data, and preserve the original requirement, agent trajectory, reviewer feedback, and accepted outcome.
Run candidate models and harnesses on the same starting state. Compare not only whether they pass tests, but whether reviewers accept them with less repair. This complements the trajectory metrics in our agent harness engineering guide.
Optimize the joint system
Sometimes the right fix is not a smarter model. Better diffs, stronger tests, clearer citations, or a structured handoff can reduce review time without changing task success. In other cases, high requirement-repair rates reveal poor context assembly, while repeated scope repair points to loose edit policies.
The goal is a productive human-agent system. A lower raw completion rate can be the better release if accepted work arrives faster and with fewer dangerous surprises.