Where Memories Live: Files, Vector Databases, Graphs, and SQL

We know what agents remember. Now: where do those memories physically sit? There are four main substrates, and the choice shapes what your memory system can and cannot do. The second one requires us to learn the most important trick in modern AI infrastructure - embeddings - so we’ll take our time there.

Substrate 1: Plain files

The humblest option: memories as text files. A profile.md, a folder of episode summaries, a CLAUDE.md of rules - possibly just in a directory the agent can list, read, and write with ordinary file tools.

Do not dismiss it. Files have superpowers the fancy options lack: perfectly inspectable (open it, read it, edit it), versionable in git, zero infrastructure, and - crucially - the agent itself can navigate them the way a person would: list the folder, read what looks relevant, grep for a keyword. Claude Code’s memory directory works exactly like this, and Anthropic’s own guidance embraces file-based “structured note-taking” as a first-class memory pattern.

Files fail at scale of lookup: when there are fifty thousand memories, “read what looks relevant” needs real search. Which brings us to the substrate built exactly for that - but first, the trick it depends on.

The trick everything depends on: embeddings

Here is the problem. A memory says “Maya’s team ships new releases on Tuesdays.” Later the agent wonders about “deployment schedule.” No shared word! Keyword search (“deploy”) misses it completely. We need search by meaning, not by spelling.

An embedding is a list of numbers - a point in space - that represents the meaning of a piece of text. A special neural network (an embedding model) reads text and outputs, say, 1,536 numbers. The model is trained with one goal: texts with similar meanings get nearby points; unrelated texts get faraway points.

A 2-D cartoon of the idea (real spaces have hundreds of dimensions):

   "ships releases on Tuesdays"  •
                    "deployment schedule"  •      ← close together!
        "deploy process"  •

                                        "Maya is vegetarian"  •   ← far away

“Ships releases” and “deployment schedule” share no words, but the embedding model - having read half the internet - knows they mean nearly the same thing, so their points land close. Distance between points = difference in meaning. That single property turns “find memories about this” into geometry: embed the query, find the nearest stored points. (In practice “nearness” is measured by cosine similarity - the angle between the number-lists - a detail you can happily treat as “distance”.)

Substrate 2: The vector database

A vector database is a database built around exactly that operation: store each memory with its embedding, and answer “give me the K stored items nearest to this query point” - fast, even across millions of items (using clever approximate-search indexes; “approximate” because exact nearest-neighbor search at scale is too slow, and a 99%-right answer in a millisecond beats a perfect one in a minute).

The write side: each memory is embedded once when stored. The read side: embed the query, fetch top-K nearest, get back the memory texts plus similarity scores. This is the workhorse of nearly every memory product (and of RAG generally): meaning-based recall over unlimited scale, cheap and fast.

Its blind spots, so you design around them: it retrieves by resemblance only - it has no idea what is true, current, or important (a stale fact embeds just as nicely as a fresh one - hence the metadata and scoring of Blogs 5 and 8); exact identifiers (“error TS2345”) are ironically weak for pure meaning-search, so production search is usually hybrid - vector search plus classic keyword search, results merged; and a vector store cannot follow chains between facts. For chains we need substrate 3.

Substrate 3: The knowledge graph

A knowledge graph stores memories as nodes (entities: Maya, payments-service, Stripe) and edges (relations: works_at, owns, written_in) - the triplet shape from Blog 5.

What it buys: multi-hop answers (“who manages the person who owns payments?” - walk two edges), one home per entity (everything known about Maya hangs off one node instead of being scattered across similar-sounding text snippets), and - in temporal graphs like Zep’s - time-aware edges with validity intervals, so contradictions close old edges instead of deleting history (Blog 5’s timeline idea, made physical).

What it costs: every memory must be parsed into entities and relations (an LLM step that can err), and the machinery is the heaviest of the four. The honest rule: graphs shine when your domain is genuinely relational - organizations, multi-user products, systems with many interconnected components. For a single user’s preferences, a graph is a forklift for a grocery bag.

Substrate 4: Plain old SQL / structured records

Some memory is naturally tabular: user settings, task status, order history, access counts. A boring relational table - exact lookups, filters, sorts, counts (“episodes from July with outcome=FAILURE, newest first”) - is unbeatable for this. Any time the query is exact rather than fuzzy, SQL beats semantic search on precision, speed, and simplicity. Metadata filtering in real systems (fetch memories WHERE user_id = maya AND created > June) is this substrate working alongside the others.

Choosing - and why the answer is usually “several”

The one-line summaries:

Production memory stacks are hybrids on purpose, typically: a small always-loaded file/profile (procedural rules + core facts), a vector store for the searchable long tail (with SQL-style metadata filters attached), and a graph if the domain demands relational or point-in-time answers. The architecture blueprint in Blog 13 assembles exactly this.

Quick recap

• Four substrates: files (inspectable, agent-navigable), vector DBs (meaning search at scale), knowledge graphs (relations + time), SQL (exact structured queries).
• Embeddings = meaning as coordinates: similar meaning → nearby points → retrieval becomes “find nearest points.” The one trick powering all semantic recall.
• Vector search retrieves by resemblance, not truth - metadata, hybrid keyword search, and scoring must supply the rest.
• Real systems combine substrates; the question is never “which one” but “which job goes where.”

Next: the read path - deciding when to retrieve, what to retrieve, and how retrieved memories actually enter the prompt.