The environment is part of the agent
Dependencies, credentials, repository state, tools, and network access can change agent performance as much as the model. Treat the environment as a versioned evaluation input.
An agent is never just a model and a prompt. It operates inside an environment: a checkout, runtime, package cache, shell, credentials, tools, services, and network policy. Change that environment and the same model can move from capable to helpless—or appear to improve without learning anything.
That makes the environment part of the system under evaluation.
Why environment quality changes the result
Consider a coding task that requires running integration tests. One worker has the right database image, seeded fixtures, and package cache. Another has an expired credential and an undocumented setup script. A comparison between their final patches measures environment luck alongside agent capability.
Environment differences commonly affect:
- whether the repository builds before the agent touches it;
- which tools and versions are available;
- whether services, secrets, and test data can be reached;
- how much time is lost to installation or recovery;
- what context leaks through caches, history, or previous runs;
- which actions are blocked by sandbox and network policy.
Cursor reached a similar conclusion after operating cloud agents: environment quality and durable execution are central to autonomous performance in What we've learned building cloud agents.
Give every run an environment identity
Pin the environment as deliberately as the model. Record an immutable image or snapshot identifier, repository commit, dependency lockfile hash, tool versions, network policy, secret names, service revisions, and setup-script commit.
Attach that identity to the trace before the first model call. If a score moves, you can then group results by environment version and determine whether the model, harness, or runtime changed.
Do not reuse a dirty workspace between eval cases. Leftover files and warm caches create hidden dependencies that disappear in production. Start from a known snapshot and explicitly label any shared cache that remains.
Separate setup failure from task failure
A final failed label is too coarse. Use a state model such as:
| State | Meaning |
|---|---|
| Provisioning failed | Worker or image could not start |
| Baseline unhealthy | The untouched project did not pass its health check |
| Setup recovery failed | Agent could not repair an allowed setup problem |
| Task failed | Environment was healthy, but requested behavior was not delivered |
| Task passed | Acceptance criteria passed in a clean verification step |
This distinction prevents infrastructure incidents from looking like model regressions. It also reveals a useful product metric: how often the agent can diagnose and recover from realistic environment faults.
Evaluate environment robustness
Run the same task across controlled variants. Start with a golden environment, then introduce one perturbation at a time:
- cold dependency caches;
- a missing optional tool;
- an expired read-only credential;
- a transient package-registry failure;
- a restarted service;
- a repository with uncommitted user changes;
- reduced CPU, memory, disk, or wall-clock budget.
Score task success together with setup time, recovery success, unnecessary changes, and cost. The goal is not to make every agent repair arbitrary infrastructure. It is to learn which assumptions are essential and whether the agent fails honestly when they are violated.
Trace setup as first-class work
Put provisioning, dependency installation, health checks, migrations, and service startup into spans beneath the agent run. This makes a two-hour trace legible: you can see whether time went to reasoning, package installation, test execution, or repeated recovery.
For long-running workflows, durable runtimes such as LangGraph expose checkpointing and persistence. Regardless of framework, retain checkpoints with the environment identity so a resumed job cannot silently continue on a different runtime.
Currai's agent observability guide explains how to represent the model, tools, and orchestration in one trace.
Make environment changes earn their promotion
Treat an image or setup-script update like a model change. Run paired cases on the old and new environments, inspect failure-class movement, and promote only after the new version improves reliability without leaking answers or weakening isolation.