Long-horizon agent evals need checkpoints, not just final scores

A short agent run can often be judged by its answer. A long-running agent may plan, edit, test, wait for external systems, recover from failures, and hand work to a human over several hours. Reducing that trajectory to passed or failed throws away the evidence needed to improve it.

Long-horizon evaluation should measure progress at checkpoints.

Define observable milestones

Start with task milestones that mean something outside the agent's narration. For a repository task, they might be:

1. environment verified;
2. requirements and constraints identified;
3. plan formed with affected surfaces;
4. first coherent implementation produced;
5. relevant tests passing;
6. regression suite passing;
7. diff reviewed for scope and safety;
8. artifact and handoff completed.

The agent does not receive credit merely for saying a milestone is complete. Attach deterministic evidence: command output, repository diff, deployed preview, ticket update, or reviewer decision.

Score progress as well as completion

Some failed runs contain valuable partial work; others spend an hour looping near the start. Track:

• time and cost to each milestone;
• percentage of milestones reached;
• regressions to an earlier state;
• repeated actions without new evidence;
• number and quality of replans;
• residual value when the run stops;
• human effort required to finish.

This distinguishes an agent that reaches a correct implementation but times out during a slow suite from one that never understood the task. Both failed the deadline, but they require different fixes.

Test durable execution deliberately

Long runs encounter worker restarts, expired credentials, rate limits, and user interruptions. Inject those events during evaluation. After resumption, verify that the agent:

• restores the right task and environment version;
• does not repeat irreversible actions;
• preserves completed evidence;
• notices stale external state;
• continues from a valid checkpoint or safely replans;
• reports what happened.

LangGraph's durable execution model is one example of a runtime built around persistence and human intervention. The evaluation principle applies regardless of framework: the checkpoint must be inspectable and tied to the trace.

Keep the trace continuous across workers

Use one trace identifier for the logical job and propagate it through queues, workers, scheduled callbacks, and resumptions. Represent each attempt and checkpoint as a nested span. Record model and harness versions on every resumed segment because a long job can cross a deployment boundary.

A continuous trace lets you answer questions that terminal logs cannot:

• Which milestone consumed the budget?
• Did a retry repeat work or build on it?
• Was the final failure caused by reasoning, infrastructure, or cancellation?
• What did the human change before resuming?

See evaluating asynchronous coding agents for a repository-focused application of this trace structure.

Evaluate stopping and handoff

Longer is not automatically better. Agents need to stop when the task is done, when progress has stalled, when a risk boundary requires approval, or when the remaining uncertainty belongs with a human.

Create cases where the correct result is a concise escalation with evidence. Score whether the agent identifies the blocker, preserves useful state, proposes the smallest decision needed, and avoids pretending to have completed the task.

The handoff should include completed milestones, changed artifacts, verification results, unresolved risks, and the exact next action. That output can be evaluated even when the original task is incomplete.

Compare agents with progress curves

Plot verified progress against time and cost, not only final success rate. Two agents may finish the same number of tasks while one reaches useful checkpoints earlier and leaves cleaner partial work when interrupted.

Pair the curve with final correctness, safety, and human repair. This prevents a fast but reckless trajectory from winning on time-to-milestone alone.