What an LLM activity dashboard should actually show

An activity dashboard answers a basic operational question: what is using the AI stack?

That usually begins with requests, tokens, spend, models, providers, and API keys. These views are essential for accounting and capacity planning. They are not enough to explain whether the product is working.

The most useful dashboard connects infrastructure activity to the user journey that produced it.

Start with trustworthy usage accounting

Every request should carry a stable timestamp, model, provider, key or project, input tokens, output tokens, latency, and cost. Aggregate views should support the same dimensions so teams can move from a total bill to the workload behind it.

At minimum, provide:

• Spend and tokens over time
• Request volume and error rate
• Model and provider distribution
• Project, workspace, key, and environment filters
• Exportable raw records
• Near-real-time updates with clearly stated freshness

Avoid mixing estimated and finalized cost without labeling them. A live counter may be provisional while provider reconciliation happens later.

Add the dimensions an agent team needs

An agent request rarely stands alone. It belongs to a session, user, workflow, agent version, prompt version, and often a tool trajectory. Carry those identifiers into the activity model.

Question	Dimension needed
Which workflow drove the cost spike?	Agent, route, intent
Did the new prompt improve outcomes?	Prompt version, deployment
Which users experienced failures?	User and session
Is one provider producing retries?	Provider, status, retry reason
Are expensive calls completing tasks?	Outcome and cost per success

Without this context, a cost increase can be mistaken for waste when it is actually successful growth—or accepted as growth when it is repeated retries.

Design for investigation, not decoration

Every aggregate should lead somewhere. Clicking an error-rate spike should open the affected sessions. Selecting a model should reveal its latency distribution, tool failures, and outcome rate. A chart that cannot be traced back to evidence is a reporting surface, not an observability surface.

Useful dashboard layers are:

1. Overview: volume, spend, success, latency, and active users.
2. Segmentation: model, provider, workflow, intent, version, and environment.
3. Exceptions: errors, policy violations, regressions, and unusual cost.
4. Evidence: the exact conversation and trace behind a point.

Measure cost per outcome

Tokens and dollars are inputs. The economic unit that matters is usually a completed task: a resolved support request, booked appointment, qualified lead, or correct research answer.

Track cost per successful outcome and the distribution around it. Include retries, fallbacks, retrieval, tool execution, speech services, and judge calls when they contribute materially. A cheap model that needs three attempts may be more expensive than a stronger model that succeeds once.

Detect changes before users report them

Baseline key metrics by workflow and version. Alert on meaningful deviations, not every noisy fluctuation. Examples include:

• Tool errors rising after a schema change
• P95 latency increasing for one provider
• Cost per resolved conversation drifting upward
• A new intent causing repeated escalations
• A model version producing more unsupported claims

Alerts should include affected sessions and the comparison window so the owner can begin with evidence.

Build the product view with Currai

Currai adds conversation, intent, violation, error, and trace context to raw AI activity. Teams can see what changed, which users were affected, and whether usage resulted in a correct outcome—not merely how many tokens were consumed.

Connect through native HTTP or OpenTelemetry with the Currai integration skill.

Sources and further reading

• OpenRouter: Activity dashboard
• Currai: track token cost
• Currai: grouped LLM monitoring charts