The modern AI agent stack: tracing, evals, prompts, and deployment
A practical architecture for production agents spanning models, orchestration, tools, data, prompts, tracing, evaluations, deployment, security, and feedback loops.
The modern AI agent stack separates execution from evidence. Models, orchestration, tools, data, and deployment run the product; versioned prompts, traces, evaluations, security controls, and feedback loops make its behavior measurable and improvable. No framework eliminates the need for those layers.
The stack by responsibility
| Layer | Responsibility |
|---|---|
| Model gateway | Provider access, routing, limits, fallbacks |
| Orchestration | State, loops, handoffs, durable execution |
| Tools and MCP | External actions with typed contracts |
| Knowledge and memory | Retrieval, context, persistent state |
| Prompt management | Versions, labels, variables, experiments |
| Observability | Traces, generations, tools, tokens, latency, cost |
| Evaluation | Offline experiments and online quality measurement |
| Deployment | Runtime, queues, checkpoints, scaling, rollback |
| Security | Identity, permissions, redaction, audit, approvals |
Choose products per boundary. A broad platform may cover several layers; a composable stack may use specialist services. Preserve portable telemetry and explicit interfaces so one decision does not lock every other layer.
Trace across layer boundaries
Create one root trace for the user task and propagate context through model calls, tools, retrieval, subagents, and background jobs. Record prompt, model, tool, and application versions. Without that lineage, a quality change cannot be connected to the deployed configuration.
OpenTelemetry's GenAI conventions provide a portable path. Currai accepts OTLP spans and also offers direct TypeScript and Python SDKs; see instrumenting with OpenTelemetry.
Make prompts deployable artifacts
Store prompts as versioned objects rather than anonymous strings in source code. Link the served version to every generation. Use labels for controlled promotion and weighted experiments for real traffic. Compare quality, latency, and cost before selecting a winner.
The Currai prompt guide documents versioning and A/B traffic splits. Keep model settings and tool schemas in the same experiment metadata.
Connect offline and online evals
Offline evals test candidates on curated datasets. Online evals score sampled production traces. Failed production examples should flow back into the dataset, and fixes should be tested offline before redeployment.
Use deterministic checks for schemas, tools, and policies; calibrated judges for semantic quality; humans for ambiguous or high-impact decisions. The evaluation readiness checklist helps establish the minimum foundation.
Separate observability from deployment risk
Deployment needs durable state, idempotent tools, retries, timeouts, rollbacks, and isolation. Observability needs to survive those boundaries without becoming a side effect that can block the user path. Batch and flush telemetry correctly, propagate IDs through queues, and redact before export.
Design for replacement
Use provider-neutral application interfaces, typed tool contracts, portable traces, and exportable datasets. Pin versions and test upgrades. The stack will change; the evaluation contract should tell you whether a replacement actually improves the product.
Currai occupies the evidence layer: traces, generations, spans, sessions, tokens, cost, prompts, and evaluations. It works with the runtime and deployment stack you already chose, giving teams a stable record while models and frameworks continue to evolve.