Evaluating coding agents with Terminal-Bench 2.0

Terminal-Bench 2.0 evaluates coding agents on 89 realistic tasks in command-line environments, each with a human-written solution and tests. The benchmark is valuable because it measures long-horizon execution, not just code completion. A leaderboard score, however, is meaningful only when the model, harness, environment, budget, and number of attempts are reported together.

The Terminal-Bench 2.0 paper reports the task design and evaluation methodology. Read it before comparing results produced by different scaffolds.

What Terminal-Bench measures

Tasks require agents to inspect an environment, use shell tools, modify files, and satisfy verifiable tests. This exposes planning, tool use, recovery, and state-management failures that static coding questions miss.

Pass rate is the primary outcome, but it does not explain why an agent failed. Instrument each run so benchmark evaluation becomes engineering evidence.

Pin the full configuration

Record:

• benchmark version and task ID;
• container image and resource limits;
• agent harness and commit;
• model name and provider version;
• system prompt and tool definitions;
• temperature, maximum tokens, steps, and time;
• number of attempts and caching policy.

A model score from one harness is not automatically a property of the model. Changing context management, shell feedback, retries, or timeouts can change the result substantially.

Trace every run

Create one trace per task attempt. Represent planning generations, shell calls, file operations, tests, and retries as child observations. Capture exit status, duration, token use, and cost. Store safe diffs or artifact references when possible.

When a task fails, classify the first consequential error: misunderstood task, missed file, wrong command, bad edit, failure to inspect test output, loop, budget exhaustion, or false completion claim. Later errors are often consequences.

Repeat and report uncertainty

Agents are stochastic. Run multiple attempts under the same configuration and report pass@1 or the benchmark's required aggregate clearly. Do not compare one side's best-of-five with another side's first attempt.

For harness experiments, use paired tasks and inspect changed outcomes. Report quality together with median tokens, wall time, and cost per successful task. A small pass-rate gain may not justify a large resource increase.

Avoid benchmark contamination

Do not expose solutions, hidden tests, or task-specific hints to the agent. Keep development and held-out evaluation tasks separate. Record network policy and external retrieval because access can change difficulty and reproducibility.

Benchmark gaming creates a high score that says little about production coding. Review suspicious shortcuts and publish the harness configuration needed to reproduce claims.

Add product-specific evaluations

Terminal tasks are not your users. Complement the benchmark with sanitized tasks from your repositories: framework conventions, review expectations, security rules, test infrastructure, and typical change sizes. Evaluate communication and safe escalation in addition to tests.

Currai can connect each benchmark result to its model generations, shell spans, tokens, latency, and cost. Use the trace to turn failures into harness experiments, then apply the same method to real coding tasks. See debugging deep agents for the trace design and agent cost efficiency for balanced resource evaluation.