Semantic Memory: Facts, Profiles, and the Truth Problem

Agent Memory, Made Clear · Part 5 · 7 min read

Semantic memory is what most people mean when they say “the AI remembers me”: clean, timeless facts. Maya uses FastAPI. The team deploys on Fridays. The user is vegetarian. It looks like the easy type - just a list of facts! - but it hides the single hardest problem in agent memory: facts change, and a confidently remembered stale fact is worse than no memory at all. Let’s build up to that.

Step 1: Where facts come from - extraction

Users do not hand over tidy facts. They bury them in conversation:

“Honestly the migration is killing me - ever since we moved everything to FastAPI last month I’ve been living in the docs. Anyway, can you look at this handler?”

Somewhere in that sigh is a durable fact: team migrated to FastAPI (~June 2026). Extraction is the step that finds it: after an exchange, an LLM is asked - “does this contain stable facts, preferences, or decisions worth remembering? Output them as short standalone statements.” It might extract ["User's team migrated to FastAPI (around June 2026)", "User is currently deep in a migration project"], ignoring the small talk.

Notice extraction is itself an LLM judgment call, and its taste defines the product. Extract too eagerly and memory fills with trivia (“user said ‘honestly’”); too conservatively and the FastAPI fact is missed. Good extractors are steered by explicit criteria - identity, preferences, relationships, projects, constraints yes; pleasantries, one-off details no - and by the application’s purpose (a medical assistant and a coding assistant should extract very different things from the same sentence).

Step 2: What shape do facts take? Three options

Shape 1: The profile. One structured document per user - name, role, stack, preferences, current projects - always loaded into every session in full. This is roughly what ChatGPT and Claude memory maintain: a compact, human-readable profile (typically tens of facts), synthesized from conversations, that the user can view and edit. Strengths: tiny, always available, no retrieval step, fully inspectable. Limit: it must stay small; it cannot hold thousands of facts about a codebase or product.

Shape 2: The fact collection. Every fact is an independent little record (“Maya prefers small commits”), stored searchably (Blog 7), retrieved on demand when relevant. Scales to thousands of facts; only pays context tokens for what’s recalled. This is the heart of memory layers like Mem0. Limit: retrieval can miss; and facts about the same thing can drift apart (see the truth problem below).

Shape 3: The knowledge graph. Facts stored as triplets - (subject, relation, object): (Maya, works_at, Stripe), (payments-service, written_in, Python), (Maya, manages, Priya). Individually less natural to write, but they connect: the graph can answer chains - “who manages the person who owns the payments service?” - by walking edges, something a bag of independent sentences struggles with. This is the approach of graph memories like Zep/Graphiti and Mem0’s graph variant. Cost: heavier machinery to build and maintain.

Real products almost always combine shapes: a small always-loaded profile for the core (“who is this user”), a searchable fact collection for the long tail, and - when relationships matter (enterprises, multi-person contexts) - a graph. Rule of thumb: profile for the ten facts that always matter, collection for the ten thousand that sometimes do.

Step 3: The truth problem - facts change

Now the hard part. Maya, in March: “we deploy on Fridays.” Maya, in July: “we switched deploys to Tuesdays.” What must the memory system do?

If it just appends, storage now contains both “deploys on Fridays” and “deploys on Tuesdays”. Retrieval will someday surface the March fact - and the agent will confidently schedule a Friday deploy, having remembered wrong. Users forgive an agent that forgets; they do not forgive one that misremembers. So every serious semantic memory has an update step: a new candidate fact is not blindly stored - it is compared against existing related memories, and a decision is made. The now-standard formulation (used by Mem0 among others) gives the deciding LLM four verbs:

ADD - genuinely new, nothing related exists → store it.
UPDATE - refines something known (“prefers Python” + “specifically Python 3.12” → merge).
DELETE - contradicts something known → the old fact must go (“switched to Tuesdays” kills “Fridays”).
NOOP - already known or not worth storing → do nothing.

This turns memory writing from logging into maintaining a belief system: reconcile, don’t accumulate. (Mechanically: the system retrieves the top-K most similar existing memories, then asks an LLM to choose the verb - full pipeline details in Blog 9.)

The elegant refinement: invalidate, don’t erase. Deleting “deploys on Fridays” loses real history - they did deploy on Fridays until July, and questions like “why did the June incident happen on a Friday?” need that. Temporal-graph systems (Zep is the flagship) therefore stamp every fact with a validity interval: deploys_on(Fridays), valid March→July and deploys_on(Tuesdays), valid July→now. Contradiction doesn’t erase the old fact; it closes its interval. Queries default to “currently valid” facts, but point-in-time questions still have answers. Memory stops being a snapshot and becomes a timeline.

Step 4: Facts carry metadata or they rot

A production fact is never just a sentence. The minimum viable record:

text: "Maya's team deploys on Tuesdays"
created: 2026-07-15        source: conversation #812
confidence: stated directly     (vs inferred from a hint)
valid_from: 2026-07-15     valid_until: (open)
access: retrieved 14 times, last on 2026-08-01

Each field earns its place downstream: timestamps power recency scoring and staleness checks; source powers trust and debugging (“why do you think that?!”); confidence separates “user said it” from “system guessed it” - critical for how assertively the agent uses the fact; access counts power forgetting (Blog 9) - a fact never retrieved in a year is a candidate for archival.

What good semantic memory feels like

Worth stating the product bar, because it is high. Good semantic memory is invisible when right and graceful when wrong: the agent simply knows your stack, your constraints, your team - without re-asking - and when it uses a memory it can say where it came from; when you correct it, the correction sticks instantly and permanently. Every mechanism in this blog - selective extraction, reconciling updates, validity intervals, provenance metadata - exists to hit exactly that bar.

Quick recap

Semantic memory = timeless facts, distilled from messy conversation by LLM extraction - whose “taste” defines product quality.
Three shapes: always-loaded profile (small core), searchable fact collection (long tail), knowledge graph triplets (when relationships and multi-hop questions matter). Real systems combine them.
The truth problem: never blindly append - reconcile via ADD / UPDATE / DELETE / NOOP; better yet, close validity intervals instead of erasing (temporal memory).
Facts need metadata - time, source, confidence, access stats - or the whole system rots.
The bar: never re-ask, never misremember, corrections stick.

Next: procedural memory - how agents remember how to act, from self-edited instructions to libraries of learned skills.