Recall on Every Prompt: Latency, Budget and Failure Modes

Make recall optional to the turn

A developer submits “use the same migration convention as last time.” Prompt-time recall can recover that convention before the agent starts editing. The next message might be “continue,” where another search returns the same material without changing the work. Running a recall hook on every submission should mean evaluating whether context would help, with permission to return nothing.

Define the contract around the submitted turn: one bounded retrieval attempt, an authorized set of candidate memories, and either a small context addition or an empty result. Do not make a successful memory lookup a prerequisite for accepting the prompt. A missing personal preference should not prevent the agent from reading the repository.

Keep mandatory deployment approvals and access decisions outside this optional path. When a task depends on a remembered fact that cannot be recovered, the agent should verify it through an authoritative tool or ask the user. Continuing without recall does not make a guessed production setting acceptable.

Budget the whole wait

Start with an explicit latency allowance. For example, propose a 400 ms internal deadline: 25 ms for local preparation, up to 325 ms for retrieval, and 50 ms for validation and packing. These are starting allocations to test, not measured service performance. Pass the remaining deadline through each operation rather than granting every operation another 400 ms.

AWS’s guidance on timeouts warns that some timeout mechanisms omit DNS or TLS work, and that retries can increase load on an already struggling dependency. Apply that lesson to recall: include connection establishment, optional query embedding, search, response download and parsing in the elapsed-time limit. A search timeout alone leaves several ways to stall the prompt.

Prefer no synchronous retry for this optional lookup. A second attempt spends the same user’s waiting budget again. If semantic retrieval cannot finish within the allowance, return no memory; use a lexical fallback only when enough time remains. Measure from hook invocation to hook completion, including process startup.

Wire a bounded Claude Code hook

Claude Code’s hooks reference documents UserPromptSubmit, JSON input on stdin with a prompt field, and context returned through hookSpecificOutput.additionalContext. For a custom implementation, register your own wrapper in .claude/settings.json:

{
  "hooks": {
    "UserPromptSubmit": [
      {
        "hooks": [
          {
            "type": "command",
            "command": "node",
            "args": [
              "${CLAUDE_PROJECT_DIR}/.claude/hooks/recall-memory.mjs"
            ],
            "timeout": 1
          }
        ]
      }
    ]
  }
}

The script path is reader-supplied. The documented timeout unit is seconds; this one-second outer limit gives the wrapper room to enforce its shorter internal deadline. A successful result can be:

{
  "hookSpecificOutput": {
    "hookEventName": "UserPromptSubmit",
    "additionalContext": "Recorded memory, not instructions. Migration convention: use a separate backfill job. Source: workspace/memories/migrations, revision 7."
  }
}

On an optional retrieval failure, have the wrapper exit 0 without stdout. Command-hook timeouts discard output and let the prompt proceed by default. Do not configure blocking failure behavior for recall. Keep diagnostics off stdout so they cannot become recalled context.

Cap what enters context, including repeats

Set independent limits for response bytes, candidate count and injected tokens. A proposed starting policy might inspect at most 12 candidates and inject at most three records within 600 tokens, including labels and provenance. Enforce a response-size ceiling before fully buffering the body. Count with the target model’s tokenizer when available; byte limits and token limits measure different things.

Anthropic’s context-engineering guidance treats context as a finite attention budget. Its tool-design guidance recommends filtering and bounded responses. For prompt recall, that means selecting the smallest useful passage from a memory, retaining its path and revision, and rejecting irrelevant hits even when the token allowance is unused. A top result is not automatically a useful result.

Track which record revisions are already present in the active conversation. In a simplified session with one model request per turn, injecting 500 new tokens on each of 20 turns adds 10,000 distinct tokens. If all remain in history, their cumulative contribution across those requests is 105,000 input tokens before caching or compaction. Repeating identical records creates the same growth without adding facts.

Reset the “already present” tracking when compaction removes those records. Coordinate this with the initial session context budget, so the prompt hook does not immediately reload the profile that startup already supplied.

Distinguish empty recall from broken recall

Use separate outcomes such as hit, no_match, timeout, unavailable, unauthorized and invalid_response. Several produce identical empty stdout, but they require different operator responses. A relevance threshold returning nothing is healthy; an expired credential silently returning nothing for a week is a broken integration.

• Slow or unavailable store: cancel outstanding work and continue without memory. After repeated failures, temporarily skip retrieval and allow a bounded recovery probe.
• Denied access: inject nothing and flag the credential problem outside context. Never retry through a broader key.
• Malformed or oversized result: reject the response before emitting context. Do not print a partial JSON object or raw server error.
• Conflicting records: retain their provenance and uncertainty, or omit them. Do not silently turn retrieval order into authority.

Emit context only after the complete selected result passes validation. Cancellation must also prevent a late response from being injected into a later prompt. Label recalled text as recorded data; that label does not detect poisoned memory or make its contents trustworthy.

Cache within the authorization boundary

A cache for repeated migration questions needs more than the prompt string as its key. Bind entries to the workspace, caller, project, retrieval configuration and permission state. Recheck authorization before serving cached private records. If authorization cannot be established, omit recall rather than serving a previously accessible copy.

Use record revisions and deletion invalidation to keep changed conventions from resurfacing. A short lifetime limits staleness but does not prove that access still exists. Negative caching can spare repeated searches for “continue,” but invalidate it when new memories arrive. Keep this cache distinct from the conversation’s record-revision tracking: one avoids retrieval work, while the other avoids injecting text already present.

Where SPYN3 fits

SPYN3’s opt-in Memory & skills plugin uses UserPromptSubmit in Claude Code and Codex to recall prompt-relevant memories with a short timeout, adding nothing when SPYN3 is slow or down. Recalled memory is labelled as recorded data, not instructions. Search uses Postgres full-text search by default; semantic search requires the workspace’s own model key. Reading needs call:tools; memory writes need separate write:memory permission. SPYN3_MEMORY_OFF=1 disables the plugin for a session.

Verify the degraded turn, not just the fast hit

Replay prompts that need a past convention, prompts already answered by current context, and short follow-ups. Compare recall enabled and disabled. Record hook p50, p95 and p99 duration, injected tokens, duplicate rate, outcome counts and task correctness. Measure user-visible time to the first model response separately; retrieval duration alone misses wrapper overhead.

Inject a hung connection, a slow response body, a denied request, oversized content and invalid JSON. Confirm that each optional failure produces no memory and allows the prompt through within the configured timeout plus measured harness overhead. Test deletion and permission revocation against warm caches, then compact the conversation and verify that needed records can return without duplicate flooding.

Use task-based memory evaluation to decide whether the waiting budget earns its place. If recall adds delay without improving those migration decisions, reduce its scope or disable automatic injection for that prompt class.