Agent memory observability: trace and evaluate long-term memory
Trace agent memory reads and writes, then evaluate retrieval, freshness, contradiction handling, privacy, usefulness, latency, and cost.
Agent memory observability records when an agent searches, reads, writes, updates, and deletes memory. Memory evaluation determines whether the right fact was stored, retrieved at the right time, applied correctly, and removed when it became stale or unsafe. Without both, memory failures look like model failures.
Long-term memory changes future behavior, so an incorrect write can affect many later sessions. It deserves the same traceability as a database mutation.
Model memory as explicit operations
Do not hide memory inside one opaque model call. Represent operations as spans:
Capture safe identifiers, query, filters, result count, scores, selected memory, write reason, version, latency, and status. Avoid placing raw secrets or unredacted personal data in observability metadata.
Evaluate memory writes
A memory should be durable only when it is useful beyond the current turn. Evaluate whether a proposed write is:
- supported by the conversation or authoritative source;
- stable enough to remember;
- scoped to the correct user, workspace, or agent;
- non-duplicative;
- compliant with consent and retention policy;
- stored with provenance and a timestamp.
Use deterministic rules for schema, scope, and prohibited fields. Use a calibrated judge for semantic usefulness. Send sensitive or ambiguous memories to human review rather than making persistence automatic.
Evaluate memory retrieval
Construct multi-session examples where earlier turns establish facts and later turns require, conflict with, or should ignore them. Measure recall of required memories, precision of selected memories, freshness, and whether irrelevant memories distract the answer.
Also test the correct non-use of memory. A billing preference should not leak into another user's session; an expired delivery address should not override a new explicit address. Track session and user identifiers consistently, as described in Currai sessions and users.
Test contradiction and update behavior
Memory systems need an update policy, not only append. Create cases where a user corrects a fact, two sources disagree, or a newer timestamp supersedes an older record. Score whether the agent:
- detects the conflict;
- uses the applicable source of truth;
- updates or invalidates the old memory;
- explains uncertainty when precedence is unclear.
Trace the retrieved versions and the eventual write so reviewers can reconstruct why a fact changed.
Measure downstream usefulness
Retrieval metrics alone do not prove memory helps. Run paired experiments with memory enabled and disabled on the same tasks. Compare task success, personalization correctness, repeated questions, tokens, latency, and cost.
Segment by memory type. Profile preferences may help quickly; large conversation summaries may increase tokens without improving outcomes. Evaluate cost per successful session rather than the cost of the memory lookup alone.
Privacy and deletion are evaluation criteria
Test deletion requests end to end: the memory is removed, cannot be retrieved, and does not remain in derived summaries or indexes beyond the documented policy. Test tenant isolation and prompt injection inside stored memory. A memory entry is untrusted input when it returns to the model.
Production monitoring should alert on cross-user retrieval, write spikes, repeated contradiction, and unusual memory size. Sample traces for review and promote confirmed failures into the offline suite.
Currai groups turns by session and user while preserving each memory span inside its run. Combined with stateful agent evaluation, that gives teams a reproducible way to improve memory without treating it as invisible model context.