Evaluating Agent Memory: Does Recall Actually Help?

Compare outcomes with and without recall

A search hit is not evidence that memory helped. Suppose an agent previously learned that invoice exports require decimal amounts without currency symbols. In a fresh session, ask it to implement another export and grade the generated file. Retrieving that preference earns a retrieval score; producing a valid export earns a task score. Record both.

Use three conditions: no recalled memory, the production recall policy, and a diagnostic condition supplied with a human-selected set of relevant memories. Keep the model version, tools, repository snapshot, current prompt and execution budget fixed. The diagnostic condition helps locate failures: if selected memories help but production recall does not, inspect capture, indexing and ranking before changing the agent prompt.

Start each condition in a fresh environment. Disable other persistent memory channels or hold their contents constant, and restore the same memory snapshot before every trial. Otherwise a previous attempt can teach the next one its answer. Anthropic's agent evaluation guidance specifically warns that leftover state can inflate scores. Repeat each task under each condition; compare distributions rather than a single convincing transcript.

Build histories that make remembering consequential

Write each case as earlier sessions, a later request, eligible memory facts and a grading rule. Put the decisive fact in an earlier session, absent from the current prompt. For example, session one records the export convention, session two discusses an unrelated billing bug, and session three requests a new export. The final artifact must follow the convention without the evaluator restating it.

LongMemEval tests information extraction, reasoning across sessions, temporal reasoning, knowledge updates and abstention. Use those categories to diversify your own suite; its conversational benchmark does not establish a pass rate for your coding agents.

• Stable preference: preserve the recorded export format in a later implementation.
• Updated fact: record a new staging database name after the old one; require the later value.
• Combined evidence: combine a deployment restriction from one session with an exception recorded later.
• Irrelevant history: add plausible billing memories that should not affect an authentication fix.
• Missing evidence: ask about an unrecorded customer preference; require clarification instead of invention.
• Scope boundary: place another person's preference in an inaccessible fixture; require that it never enters the caller's context.

Split development and held-out cases by project or history, keeping related sessions together. Freeze held-out histories before tuning recall thresholds. For capture tests, let the system extract memories from those histories, then evaluate later requests without exposing reference answers to extraction.

Score what reaches the agent

Label each candidate memory for relevance to the later request, validity at that time and permission to disclose. Keep a reference set of facts needed for the task. Label the final injected payload as well as search results: a correct search hit can disappear during truncation, while a relevant file can include distracting paragraphs.

Define the counting unit before calculating retrieval metrics. For mixed memory files, annotate individual facts so one useful sentence cannot make a long file look entirely relevant. Deduplicate repeated facts when scoring coverage; count every repeated token when scoring context cost.

• Recall precision: relevant, valid, permitted injected facts divided by all injected facts.
• Evidence coverage: distinct required facts present in the payload divided by all required facts.
• Empty-recall correctness: requests needing no historical context that receive none, divided by all such requests.
• Task success: trials whose final artifact meets the case's acceptance criteria, divided by all trials.

Precision is undefined when nothing is injected; report those cases separately. Cases with no required historical facts likewise belong outside the coverage denominator. ARES distinguishes context relevance, answer faithfulness and answer relevance. Apply that separation here, while recognizing that a memory-supported answer can still repeat an obsolete fact. For retrieval diagnostics beyond memory, use the retrieval evaluation guide.

Expose harmful recalls and delayed answers

Count harmful retrieval and harmful behavior separately. An obsolete staging hostname entering context is a retrieval defect even when the agent checks it and recovers. Connecting to that obsolete host is an outcome defect. Record the memory identifier, its conflicting successor and the action taken, so a successful recovery does not conceal a ranking problem.

Include fixtures containing quoted instructions such as a historical request to disable export validation. The later task should treat that text as recorded data and preserve current requirements. Grade the resulting code or tool action, rather than whether the agent says it ignored the instruction. The memory poisoning guide explains that separate threat surface.

Inject recall timeouts, empty results and unavailable services. Measure median and p95 added latency across every request, including requests returning nothing. Grade whether the agent continues appropriately, asks for missing facts or invents continuity. Choose a timeout budget from the workflow's measured response needs; an interactive prompt and an unattended CI task need separate acceptance limits.

Calculate token cost alongside success

Store one row per completed trial, including task identifier, repetition, condition, success, harmful recall, injected tokens, latency and total monetary cost. Retain failed attempts in the denominator. Include extraction, embedding, retrieval-model and generation costs where incurred; allocate one-time ingestion costs over an explicitly stated reuse horizon. Compare that horizon with the expected lifetime of the memories.

For a table named memory_eval_trials, store success and harmful_recall as non-null integers restricted to zero or one. Store token counts as integers and cost as numeric. This PostgreSQL query uses documented aggregate functions to summarize each condition:

SELECT variant,
       count(*) AS trials,
       avg(success) AS success_rate,
       avg(harmful_recall) AS harmful_trial_rate,
       avg(injected_tokens) AS mean_injected_tokens,
       avg(total_cost_usd) AS mean_task_cost
FROM memory_eval_trials
GROUP BY variant;

Define harmful_recall as whether any invalid or disallowed memory reached the agent during that trial. Track harmful actions in a separate field. Add paired task-level comparisons before interpreting the aggregate: an improvement on export formatting must not hide regressions on changed database names. Report absolute success-rate differences and cost per successful task, including spending on failures. If there are no successes, cost per success is undefined. Also report recall tokens separately from total tokens, since extra context might reduce later searching.

Testing recall with SPYN3

SPYN3's opt-in context layer exposes context_search and context_read to any MCP client. Search uses Postgres full-text search by default; semantic search requires the workspace's own model key. Its Claude Code and Codex plugin recalls relevant memories on UserPromptSubmit, adds nothing if SPYN3 is slow or down, and can be disabled for a session with SPYN3_MEMORY_OFF=1. Use that switch for a plugin-off comparison, ensuring no other recall path remains active. Evaluate search modes separately rather than attributing their differences to memory alone.

Set the release rule before inspecting results

Choose the minimum useful task improvement, latency budget, cost ceiling and unacceptable harmful behaviors before running the held-out suite. These are your operating requirements, not universal memory benchmarks. Treat disclosure across a scope boundary as a release blocker even if the average task score rises.

Report wins and losses by task family and include uncertainty. Repeated attempts on one history are not independent new histories; estimate uncertainty by resampling whole histories with their trials together. Use deterministic artifact checks where possible. For judgments such as whether a remembered preference was useful, hide the condition from reviewers and calibrate automated graders against human labels.

Keep the held-out suite fixed for the release decision, then add observed production failures to a separate regression set. Rerun when extraction, ranking, memory formatting, model version or context budget changes. Preserve the no-recall comparison: memory earns its context budget through better completed work, fewer repeated questions or lower measured cost on the tasks you actually run.