Debugging and evaluating deep agents with end-to-end tracing

Deep agents are difficult to debug because one user request can become a plan, dozens of model calls, filesystem operations, tool invocations, and delegated subagent runs. End-to-end tracing reconstructs that execution tree. Evaluation then determines whether the outcome and the path were acceptable.

Logging only the initial prompt and final answer discards the evidence engineers need most: where the plan changed, which context a subagent received, and why the agent decided to stop.

Define the trace boundary

Create one root trace for the user-visible task. Under it, represent meaningful units of work consistently:

research-and-update (trace)
├── plan (generation)
├── inspect-repository (span)
│   ├── choose-files (generation)
│   └── read-files (span)
├── delegate-tests (span)
│   └── test-subagent (trace or span subtree)
├── edit (span)
│   └── propose-patch (generation)
└── verify (span)

Store prompts and completions on generations; file, shell, browser, or API work on spans; and the total task outcome on the root trace. The Currai nested-trace guide provides TypeScript and Python instrumentation patterns.

Capture the decisions that change execution

Useful traces record more than timing. Attach the plan version, selected tool, tool arguments, retry number, active subagent, checkpoint ID, model, token use, and error classification. For files or large payloads, store a safe reference or summary rather than duplicating sensitive content.

Use stable span names such as agent.plan, tool.shell, and subagent.run. Place volatile values in metadata. Stable naming lets dashboards compare error and latency distributions across runs.

Debug common deep-agent failures

The plan was plausible but incomplete

Open the planning generation and compare its input with the user's constraints. If a constraint was absent, context assembly failed. If present but ignored, the planning prompt or model decision needs attention. Add an evaluator that checks whether required deliverables appear in the plan before execution.

A subagent solved the wrong task

Inspect the delegation span and the subagent's first generation. Check whether the parent passed the relevant files, success criteria, and authority boundary. Evaluate delegation quality separately from the subagent result; otherwise the downstream agent is blamed for a malformed assignment.

The agent loops

Group repeated tool names, arguments, and errors. A loop often means the tool returned an ambiguous failure or state did not persist between iterations. Add maximum-step and cost budgets, then evaluate whether the agent escalates or summarizes progress when it cannot proceed.

The final answer overstates completion

Compare claimed actions with successful tool spans. A deterministic evaluator can fail any run that says a test passed without a matching successful test result. This is stronger than asking a judge whether the answer sounds honest.

Evaluate outcome and trajectory

Deep-agent quality needs at least four score families:

Use deterministic checks for tests, files, schemas, and budgets. Use calibrated LLM judges for plan quality or semantic task completion. Use human review for novel, high-impact failures. The AI agent evaluation guide explains how to combine these layers.

Turn traces into regression tests

When production debugging finds a real failure, sanitize the trace and promote its input, environment requirements, and expected behavior into the offline dataset. Preserve the failure category so later experiment reports can show whether a candidate fixes that class without harming others.

Then run paired experiments: baseline and candidate on the same tasks, repeated where stochasticity matters. Inspect every regression as a trace instead of accepting an aggregate score at face value.

End-to-end tracing is the record; evaluation is the decision. Together they make deep-agent behavior reproducible enough to improve deliberately, even when the agent's execution path is long and dynamic.