Evaluating open-source asynchronous coding agents
Evaluate asynchronous coding agents across repository setup, planning, execution, tests, review, recovery, security, latency, and cost.
Asynchronous coding agents work in the background, often for minutes or hours, then return a patch or pull request. Evaluation must cover the entire job: repository preparation, planning, tool execution, code changes, tests, review quality, and honest reporting. A patch that looks plausible is not proof the task was completed safely.
Open-source agents make the harness inspectable, but they do not make evaluation automatic. Pin the exact agent commit, model, container, tools, and resource budgets for every experiment.
Build repository-level tasks
Use tasks with a clean starting commit, explicit requirements, a reproducible environment, and tests that distinguish superficial from correct changes. Include bug fixes, small features, refactors, dependency updates, and tasks where the correct action is to ask for clarification.
Keep hidden acceptance tests separate from instructions. Add repository-specific checks for formatting, types, migrations, security, and generated files. Prevent network access or external side effects unless the task explicitly requires them.
Trace the asynchronous job
Create one trace per job, even when execution crosses workers. Propagate the trace context through queues and checkpoints. Represent planning, file reads, searches, shell calls, edits, tests, and review as nested observations.
Record agent and repository versions, model, prompt, environment image, timeout, step budget, token usage, cost, and final commit. A trace makes a three-hour job inspectable without replaying a terminal transcript line by line.
Evaluate more than test passage
| Dimension | What to check |
|---|---|
| Correctness | Visible and hidden tests, requested behavior |
| Scope | No unrelated or destructive changes |
| Maintainability | Fits project conventions and APIs |
| Security | No secrets, unsafe commands, or weakened controls |
| Communication | Accurate summary, limitations, test report |
| Efficiency | Time, steps, tokens, and cost per accepted change |
Use deterministic checks for artifacts and tests. Use static analysis and project linters for code quality. A calibrated judge can assess requirement coverage or summary accuracy, but human review remains appropriate for architecture and security-sensitive changes.
Test interruption and recovery
Terminate jobs during tool execution, restart workers, expire credentials, and simulate failing tests. The agent should resume from durable state or explain why it cannot. It must not repeat irreversible operations after a retry.
Evaluate cancellation too. A user who cancels should not receive a later pull request from a stale worker. Trace queue events and checkpoints so duplicate work can be diagnosed.
Compare harnesses fairly
Run baseline and candidate on identical commits and tasks. Repeat stochastic cases. Report success together with median cost and wall time; a faster agent that requires more human repair is not more productive.
Analyze failure classes rather than chasing one score. If the candidate improves test debugging but increases unrelated edits, fix its context or edit policy instead of changing the model blindly.
Currai connects long-running trace segments across the job and shows the model, tools, tests, tokens, and cost that produced each patch. Combine it with Terminal-Bench evaluation for external benchmarking and product-specific repository tasks for release decisions.