From trace to root cause: debugging agent failures across models, tools, and environments

An agent returns the wrong result. The model is the most visible component, so it gets blamed first. But the model may have received the wrong document, an ambiguous tool, stale state, or a misleading error from the environment.

Root-cause analysis begins by reconstructing the causal chain.

Use a layered failure taxonomy

Classify the first defective layer, not every downstream symptom.

Layer	Example failure
Intent	Requirement or success condition was misunderstood
Context	Relevant information was missing, stale, or conflicting
Retrieval	Wrong document, memory, or tool was selected
Reasoning	Available evidence supported a better decision
Tool interface	Name, description, or schema caused misuse
Tool execution	Correct call returned an error or wrong result
Orchestration	Retry, routing, memory, or stopping policy failed
Environment	Dependency, credential, service, or resource failed
Evaluation	Grader mislabeled the actual outcome

Later layers can fail because an earlier one did. If retrieval supplied an outdated policy, the model's incorrect answer is a symptom. Fixing the prompt may hide the issue without repairing the retrieval system.

Reconstruct a complete trace

Keep the user request, resolved instructions, retrieved context, tool candidates, model inputs and outputs, tool arguments and responses, environment metadata, retries, costs, and final artifact under one trace.

Redact sensitive values without removing structure. A record that says only tool_call_failed cannot distinguish invalid arguments from expired credentials or a provider outage.

Currai's traces, spans, and generations model provides the hierarchy needed to follow one agent run across these layers.

Find the first divergence

Choose a successful comparison run with the same task or failure family. Walk the two traces until behavior first diverges:

1. Did they receive the same environment and instructions?
2. Did they retrieve the same context and tools?
3. Did the model choose the same plan or action?
4. Did the tool return the same state?
5. Did retry and stopping policies respond the same way?

The first meaningful divergence is usually closer to the fix than the final bad answer. Record the evidence and confidence for the root-cause label.

Reproduce the smallest failing segment

Replay only the layer under investigation when possible. Test retrieval against the same index snapshot, execute the tool with recorded arguments in a safe environment, or rerun the model with the resolved prompt and context.

Then change one variable. If a new model appears to fix the case, rerun enough times to separate a robust improvement from stochastic luck. If a tool fix works, verify that the original model succeeds without prompt changes.

Turn incidents into regression cases

The output of root-cause analysis should be more than a ticket. Create a redacted case with starting state, expected behavior, layer-specific checks, and the final system-level acceptance criterion.

Tag it with the failure class and impacted versions. Future candidates should run against both the minimal reproduction and an end-to-end case. This prevents a local fix from shifting the failure elsewhere.

Report uncertainty honestly

Not every failure has one cause. Use primary, contributing, and unknown labels. Track inter-reviewer agreement and revisit high-impact unknowns. Root-cause dashboards that force false precision create confident but ineffective roadmaps.

Aggregate by failure class, tool, environment, and harness version. Prioritize by user impact and recurrence, not by which component has the loudest errors.