Working Memory: The Context Window (and Why Bigger Isn’t the Answer)

Before building any external memory, we must master the memory the model already has: the context window. Every clever memory system ultimately delivers its results into this window, so its properties shape everything downstream.

What the context window is

The context window is simply the maximum amount of text the model can take as input in one call, measured in tokens (a token ≈ 3/4 of a word).

Everything the model will consider must fit inside it: the system instructions, the conversation so far, tool outputs, retrieved documents, retrieved memories - everything. If it is not in the window, then for this call, it does not exist.

In human terms, this is working memory: what you are holding in mind right now. Not what you know - what you are currently thinking about. Modern windows are large (hundreds of thousands of tokens - entire books), which sounds like it might make external memory unnecessary. Let’s see why it doesn’t.

Problem 1: The window still ends - and then everything is gone

However big the window, it is finite, and an agent’s life is not. A customer relationship spans years. A codebase project spans months. Tool-using agents burn tokens astonishingly fast - one web page can be thousands of tokens, one test run can dump tens of thousands of tokens of logs. Long-running agents hit the ceiling not in months but in hours.

And the deeper issue from Blog 1 remains: the window is per-call input, not storage. When the session ends, the window’s contents evaporate. A context window cannot remember last Tuesday, no matter its size.

Problem 2: You pay for the whole window, every single call

Here is the economics most people miss. The model re-reads the entire input on every call. If your agent carries 200,000 tokens of history and the user sends a 10-word message, you pay for processing ~200,000 tokens to answer those 10 words. And again for the next message. A long conversation re-billed on every turn grows quadratically in total cost.

(One real mitigation exists: prompt caching - providers can remember the processed form of an input prefix that repeats exactly, making re-sends much cheaper and faster. It is a vital cost tool, and we return to it in Blog 11. But it only discounts re-reading; it does not create persistence, and cache entries themselves expire.)

Latency scales the same way: the fatter the input, the slower the first token of the response. Stuffing the window is paying more money for a slower, and - as we will now see - often worse answer.

Problem 3: Context rot - more text makes the model dumber

The most counterintuitive problem. It is tempting to assume a model uses token #400,000 as reliably as token #400. It does not.

As the input grows, the model’s ability to find and use any particular fact inside it degrades. Findings that repeat across studies of long-context models:

• Lost in the middle. Facts near the beginning or end of the input are recalled better than facts buried in the middle.
• Distraction. Irrelevant text does not just occupy space; it actively pulls the model’s attention. Two hundred thousand tokens of miscellaneous history can make the model worse at a task than a clean two-thousand-token briefing.
• Degradation with length. Even on simple “find the fact” tests, accuracy drifts down as the input gets longer.

The field calls this context rot. The practical law it gives us:

The context window is not a warehouse. It is a workbench. Work goes well when the workbench holds exactly the tools for the current job - not everything you own.

So “just use a bigger context” fails three ways

Let’s put the three problems together, because this is the argument for this entire series. Suppose we skip building memory and simply dump all history into a giant window:

1. It ends. The agent’s life outlives any window; sessions still forget each other.
2. It’s expensive and slow. Full history re-processed per call - the costliest possible design.
3. It rots. Recall of what matters gets worse as the pile grows.

Compare the alternative: store history outside; on each call, select a small, sharp set of relevant memories into the window. Persistent (survives sessions), cheap (thousands of tokens instead of hundreds of thousands - production memory stacks routinely report ~90% token savings vs full-context), and more accurate (a clean workbench beats a heaped one).

Bigger windows and external memory are not competitors, they are partners: the window is where thinking happens; memory decides what deserves to be there. Even the most memory-rich agent ultimately lives or dies by what is on the workbench at the moment of the call.

Managing the workbench

Since the window is precious, agents actively manage it. A preview of the toolbox (Blog 11 goes deep):

• Budgeting: treat the window as a budget split across fixed instructions, recent conversation, retrieved memories, and tool outputs - each slice capped.
• Compaction: when a long session nears the limit, summarize the transcript so far, then continue with the summary in place of the raw history. (This is a lossy save - details die in summaries - which is exactly why important things should be written to real storage before they get compacted away.)
• Offloading: instead of holding a 50,000-token file dump in the window, write it to a file and keep only the path and a two-line description. Fetch again on demand.
• Isolation: give messy exploratory work to a sub-agent with its own private window; only its short summary returns to the main agent.

Notice what all four have in common: they move content between the window and outside storage. Working memory management and long-term memory are the same discipline at two timescales.

Quick recap

• The context window = per-call input = the agent’s working memory. If it’s not in the window, it doesn’t exist for this call.
• It is not persistence: contents evaporate after the session, however large the window.
• You re-pay for the whole window every call (prompt caching softens the price of exact repeats, but that’s a discount, not memory).
• Context rot: long, cluttered inputs degrade recall - the window is a workbench, not a warehouse.
• Therefore: store outside, select a small relevant set in. Bigger windows complement memory; they don’t replace it.

Next: the map of everything we might store outside - the four types of memory.