Deep-Dives / Memory & Context
Memory & Context
Context windows, memory stores, compaction, retrieval-augmented memory — keeping the right things on the prompt.
- Engineering the Context WindowTreat the finite window as a budgeted resource: per-category token budgets, position-aware ordering, and utilization metrics.
- Short-Term vs Long-Term MemoryThe in-prompt working set vs the external store: what earns a slot, when to write, when to recall, and the promotion/demotion cycle.
- Memory Types: Episodic, Semantic, ProceduralThree durable memory kinds plus the scratchpad, each written and retrieved differently; reflection promotes episodes to semantics.
- Retrieval-Augmented MemoryRecall as retrieval: state-derived cues, relevance+recency+salience scoring, threshold-before-truncate, and provenance-tagged rendering.
- Context Compaction & Hierarchical MemoryThe compaction ladder, task-structured summarization, MemGPT-style tiering, pressure-triggered hysteresis, and verifying lossy compaction.
- Memory Stores: Vector, KV, Graph & EvictionMatch backend to memory kind, a unified interface, why unbounded stores rot retrieval, and decay/eviction policies.
- Evaluating Memory QualityMemory-specific metrics (recall@k, staleness, constraint survival, write precision) and the pitfalls they catch: poisoning, staleness, drift, compaction amnesia.
- Memory Write-Path ArchitecturesRAG-only is dead for stateful agents — the write path (what earns a slot, when to write, when to update) is the 2026 focus, and the four memory kinds (episodic, semantic, procedural, relational) each want a different policy.
- Memory Poisoning DefensesAgentPoison at 80% ASR with <0.1% poison; MemoryGraft, SpAIware, Morris-II — lifecycle defenses at ingestion, storage, retrieval, and monitoring.
- Long Context: Effective vs AdvertisedWhy RULER, NoLiMa, MRCR v2 diverge from advertised token ceilings by 30-60 points past 200K — and how to budget accordingly.
- Learned Retrievers & MemRLMemRL treats store / retrieve / update / summarize / discard as tools optimized via RL — rank by learned utility rather than semantic similarity alone.
- Forgetting & SupersessionForgetting is five structurally different mutations — supersession, decay, amnesia, purge, drift — and four need intent that only exists at write time, so no recency weight or reranker recovers them. A small model RL-trained to answer from a fact's current value doubled its accuracy and reached 16.7%; rankings also invert with tenure, a curated map falling 96% to 72% by nine weeks while a provenance-typed graph rises to 90%. Put a mutation control plane in front of the store, resolve entities exactly, and make current-ness a query invariant.
- KV Cache Budgeting for Agent LoopsThe weight file says whether the model loads; the KV cache says whether the agent runs, and on a long loop it is routinely larger: a 70B-class model at bf16 cache precision costs about 0.31 MB per token, so 128K context is 40 GB per concurrent session against 42 GB of 4-bit weights. The formula, a worked 80 GB budget, the four levers that change the per-token cost and what each charges, how to reconstruct a vendor's 'fits in 24 GB' claim by division, and why the context cap you enforce — not the average session — is the capacity control.
- Memory Scopes & TenancyEvery memory layer takes the scope as an argument the caller declares — Mem0's user_id, Graphiti's group_id, Cognee's dataset — and a declared scope is not an authorisation decision; Graphiti's own docs say its reference servers let callers read any group_id. The leak is silent because similarity carries no provenance and cross-tenant neighbours are the most similar. Close the scope over the authenticated session, post-filter so a store-side bug fails closed, default automatic writes to the narrow scope, and ship the thirty-line cross-tenant recall test.
- Files as the Context WindowOffloading a large observation to disk is not compression — it converts a mandatory carry into an optional read, and the entire saving lives in two numbers nobody logs: how often the file is re-read and how much of it is read each time. A 48,000-token result at step 3 of a 20-step run costs 816,000 input tokens inline, 816,510 if the agent re-reads the whole file every turn (plus 17 round trips, so strictly worse), and 12,510 if it greps and reads a 2,000-token slice six times. Covers why a path is not a pointer, why files may be read into the tail of the context and never written into its head, and the three things — replay, concurrency, taint — that stop being true once working memory lives on a disk.