Context Engineering and the Optimization Playbook

Agent Memory, Made Clear · Part 11 · 7 min read

Memory decides what an agent can know; context engineering decides what it is actually thinking about on this call - the art of composing the context window, every call, from all the competing claimants: instructions, conversation, tool outputs, retrieved memories. This blog is the optimization playbook: the techniques, then the money math that justifies them.

The mindset: every token must earn its seat

From Blog 2: the window is a workbench that you re-pay for on every call, and cluttering it makes the model worse (context rot), not just poorer. So the governing question of context engineering is brutal and simple: for this call, does this token earn its seat? Anthropic’s guidance for its own agents says exactly this - find the smallest set of high-signal tokens that maximize the likelihood of the desired outcome. Everything below is a technique for enforcing it.

Technique 1: Token budgets - the window as a portfolio

Treat the window as a budget with allocations, not a bucket:

system + standing rules (procedural)   ~1-2k tokens   (fixed)
memory: profile (always-on core)       ~0.5-1k        (fixed)
memory: retrieved (episodic/semantic)  ~2-7k          (capped)
recent conversation                    ~5-20k         (rolling)
current tool outputs / files           the rest       (managed!)

Caps force the right behavior upstream: the retrieval budget makes Blog 8’s scoring select instead of dump; the rules budget keeps CLAUDE.md sharp (Blog 6); the tool-output allocation - usually the biggest and least controlled - gets the next two techniques. Production stacks that run disciplined budgets report retrieval costs around ~7k tokens per query where full-history stuffing burns 25k-100k+, at equal or better answer quality.

Technique 2: Compaction - summarize and continue

A long-running session eventually fills any window. Compaction: when near the limit, summarize the transcript so far - decisions made, current state, open items - and start a fresh window seeded with that summary plus the last few raw turns. (Claude Code’s auto-compact is this, productized.)

Compaction is a lossy memory write, so the craft is in what survives: keep decisions and their reasons, current task state, unresolved threads, hard-won lessons; drop raw tool dumps, dead ends, pleasantries. Two operational rules: compact before you’re forced to (a panic-summary at 99% full is written badly), and write real memories first - anything that deserves long-term storage (Blog 9) should be extracted before the raw text is compacted away, because compaction’s summary is not a substitute for the memory store.

Technique 3: Structured note-taking - the agent’s external scratchpad

Instead of holding everything in-window, the agent writes notes to files as it works: a running notes.md, a to-do list, a findings log - then re-reads them when needed. The window holds a pointer (“full analysis in notes.md”), not the payload. This is Anthropic’s “structured note-taking” pattern, and it is stunningly effective for long tasks: after a compaction or even a fresh session, the agent re-reads its notes and resumes - the notes are working memory made durable (and they double as raw material for Blog 9’s background consolidation).

The same move applies to bulky tool results: don’t carry a 50k-token log dump in-window; save to file, keep the path plus a two-line synopsis, re-open on demand.

Technique 4: Sub-agent isolation - spend tokens in a side room

Some work is necessarily messy: searching a big codebase, reading twenty documents. Do it in the main window and the mess lives there forever (you re-pay for the exploration on every subsequent call). Instead, hand it to a sub-agent with its own private window; it burns tens of thousands of tokens exploring and returns a distilled 1-2k-token answer. The main agent’s workbench receives the conclusion, not the journey. This is Blog 2’s isolation, matured into architecture: deep multi-agent research systems work precisely because each sub-agent’s context is disposable while only distilled summaries flow up.

Technique 5: Prompt caching - pay full price once

Providers can cache the processed form of a prompt prefix that repeats exactly, cutting cost (often ~10x cheaper for cached tokens) and latency on every subsequent call that reuses it. The catch - exactly - has a design consequence: put the stable stuff first (system prompt, rules, profile), volatile stuff last, and don’t churn the stable region (every edit to your always-on blocks invalidates the cache behind it). Caching also reframes a classic trade-off: a big always-loaded profile is cheaper than it looks if it never changes mid-day - but it still occupies attention (context rot doesn’t care about your discount), so caching complements selection; it never replaces it.

The money math: what memory is worth

Let’s make the business case concrete, because “optimization” ultimately means dollars. Take an assistant doing 40-turn conversations averaging 100k tokens of accumulated history per turn if naively stuffed:

Naive full-history: ~100k input tokens × 40 turns = ~4M input tokens per conversation.
Memory + budgets: ~1.5k fixed + ~5k retrieved + ~10k rolling recent ≈ ~17k per turn ≈ 0.7M tokens - roughly 6x cheaper, before prompt caching discounts the fixed slice further.
And per Blogs 2 and 8, the cheap version typically answers better (clean workbench beats rotten warehouse) and streams faster (less prefill).

That triple win - cost, latency, quality - is why memory layers can honestly market “90% token savings with higher accuracy,” and it is the pitch slide for any memory product you build: memory is not a cost center; it is the optimization.

The optimization checklist

For a production agent, in priority order:

Budget the window - fixed slices for rules/profile, hard caps for retrieval and tools.
Retrieve few, good tokens - Blog 8’s scoring + reranking under the cap; measure tokens-per-query.
Compact proactively, preserving decisions/state/lessons; extract real memories first.
Offload bulk to files/notes; carry pointers, not payloads.
Isolate messy work in sub-agents; return summaries.
Order for caching - stable first, volatile last; don’t churn the stable region.
Instrument everything - tokens per call, cache hit rate, retrieval precision; you tune what you measure (and Blog 12 makes measuring rigorous).

Quick recap

Context engineering = composing the window each call; the test is “does this token earn its seat?”
Five techniques: budgets (portfolio, not bucket), compaction (lossy - save real memories first), note-taking (durable scratchpad), sub-agent isolation (spend tokens in a side room), prompt caching (stable-first ordering).
The money math: memory-with-budgets beats naive stuffing ~6-10x on cost and wins on latency and quality - the actual business case for memory products.

Next: proving it works - benchmarks and metrics for memory, the failure modes, and the security threats (memory poisoning) every launchable product must answer for.