The Read Path: Retrieving the Right Memory at the Right Time

A memory that exists but never resurfaces might as well not exist. The read path - storage → context window - is where memory systems win or die, and it has four stages: trigger, search, scoring, and injection. Let’s walk them in order.

Stage 1: The trigger - when does retrieval happen?

Two philosophies, genuinely different in feel:

Automatic retrieval (memory as a reflex). Before every model call, the system silently searches memory using the latest user message and injects whatever scores well. The agent doesn’t decide; it just finds relevant memories already present, the way you don’t decide to remember your friend’s name when they walk in. Pros: nothing is forgotten by laziness; consistent latency; works with any model. Cons: retrieval fires even when useless (cost, noise), and the query is naive - the user’s literal message may be a poor description of what’s actually needed.

Agent-driven retrieval (memory as an action). The agent has search tools - search_memory("Maya deploy preferences") - and chooses when to look things up, like you deliberately racking your brain. Pros: queries can be smart, iterative (“that turned up nothing; rephrase”), multi-step; retrieval happens exactly when the agent senses a gap. Cons: the agent must realize it should look - and the most dangerous failure is not knowing what you don’t know. The agent that never suspects a relevant preference exists never searches for it.

Production systems increasingly do both: a cheap automatic pass injects obviously-relevant memories (“here’s what I recall that might matter”), and search tools let the agent dig deeper on demand. The automatic pass also serves as a hint: seeing partial memories reminds the agent that a memory store exists and is worth querying.

Stage 2: The search - casting the net

With a trigger fired and a query in hand, cast a wide net over storage. From Blog 7 we have the machinery: embed the query, pull the nearest neighbors from the vector store; run keyword search in parallel (hybrid) so exact names and error codes aren’t missed; apply metadata filters (this user, this project, valid-now); walk graph edges if a graph exists. The net’s job is recall - don’t miss anything plausibly relevant. Grab a generous candidate set, say 50, knowing most won’t make the cut.

One refinement worth knowing: query construction. The user’s raw message is often a poor search key (“it’s broken again 😤” embeds terribly). Better systems build the query from richer signal: the current task description, entities in play, recent turns - or even have a small model write the search query. Garbage query in, garbage memories out; the net is only as good as where you throw it.

Stage 3: Scoring - who deserves the workbench?

Fifty candidates, room for five (the workbench is small and rots - Blog 2). Pure similarity is not enough to pick, because resemblance is not usefulness: a five-year-old fact can resemble the query perfectly and still be poison (superseded), and a critical allergy can resemble nothing while mattering absolutely.

The classic answer - introduced by Stanford’s Generative Agents and echoed everywhere since - scores each candidate on three signals and blends them:

score = relevance + recency + importance

• Relevance: the similarity from stage 2 - how much does this memory resemble the need?
• Recency: how fresh is it? Typically an exponential decay - each day (or each hour) since last access multiplies the weight by a bit less than 1 - so scores fade smoothly rather than falling off a cliff. Fresh context wins ties; stale-but-similar sinks.
• Importance: how much does this memory matter intrinsically? Often an LLM-assigned 1-10 stored at write time (“user mentioned a lethal allergy” ≫ “user likes blue themes”), or a proxy like access frequency. Importance is the counterweight that lets a poorly-matching-but-critical memory force its way in.

The blend weights are product decisions: a therapy-companion app leans recency (this week’s state), a codebase agent leans relevance, a safety-critical assistant leans importance. There is no universal setting - the formula is standard, the weights are your product.

Optional sharpener: reranking. Embedding similarity is fast but shallow. A reranker takes the query and each top candidate together and reads them as a pair - much more accurately judging “does this actually answer that?” - then reorders. Costs a little latency on a handful of items, often dramatically improves the final five. (General retrieval tooling; memory systems inherit it.)

Stage 4: Injection - how memories enter the prompt

Retrieved memories must now be placed in the context, and placement is not cosmetic:

• Labeled, not smuggled. Wrap them in a clearly marked block: “Relevant things you remember about this user: …”. The model should know these are its memories - so it can weigh, mention, or even doubt them - not mysterious facts floating in the prompt.
• With provenance when it matters. “(from conversation, July 12)” lets the model - and the user - trust and verify. Products like Claude’s memory surface when a memory is being used; that transparency starts at injection.
• Frame as fallible. Memories are beliefs, not ground truth. A frame like “you previously noted (may be outdated): …” measurably changes how the model handles contradictions - it asks (“still deploying Tuesdays?”) instead of assuming.
• Budgeted. Injection obeys a token budget (a few thousand tokens is a common ceiling; production stacks report ~7k tokens per query where naive full-context stuffing burns 25k-100k). When over budget, drop the lowest scores or compress several memories into a summary line each.

The two failure modes to design against

Every read-path decision trades between them:

• Silent miss (needed memory not retrieved): the agent re-asks a known question or repeats a solved mistake. Invisible in logs - the answer merely could have been better. Countered by recall-heavy nets, hybrid search, good queries, agent-driven digging.
• Noisy hit (irrelevant/stale memories injected): pays tokens, invites context rot, and can actively mislead. Countered by scoring, reranking, budgets, and the update discipline of Blog 5.

If forced to choose, production stacks bias toward precision (fewer, better memories) - because every injected token costs money and attention on every call, while a miss costs only occasionally. But the real answer is instrumentation: log what was retrieved and whether it helped, and tune (Blog 12).

Quick recap

• Read path = trigger → search → scoring → injection.
• Trigger: automatic (reflex - consistent, naive) vs agent-driven (action - smart, can forget to look); mature systems use both.
• Search wide for recall (hybrid vector+keyword+filters, well-built queries); then score for precision: relevance + recency (decay) + importance - weights are product decisions; rerankers sharpen the final cut.
• Injection: labeled, provenanced, framed as fallible, token-budgeted (~thousands, not tens of thousands).
• Tune between silent misses and noisy hits - and instrument to know which you’re suffering.

Next: the mirror image - the write path: deciding what deserves to be remembered at all, and the housekeeping (consolidation, decay, forgetting) that keeps a memory store healthy for years.