Deep-Dives / Memory & Context

Memory & Context

Context windows, memory stores, compaction, retrieval-augmented memory — keeping the right things on the prompt.

  1. Engineering the Context Window
    Treat the finite window as a budgeted resource: per-category token budgets, position-aware ordering, and utilization metrics.
  2. Short-Term vs Long-Term Memory
    The in-prompt working set vs the external store: what earns a slot, when to write, when to recall, and the promotion/demotion cycle.
  3. Memory Types: Episodic, Semantic, Procedural
    Three durable memory kinds plus the scratchpad, each written and retrieved differently; reflection promotes episodes to semantics.
  4. Retrieval-Augmented Memory
    Recall as retrieval: state-derived cues, relevance+recency+salience scoring, threshold-before-truncate, and provenance-tagged rendering.
  5. Context Compaction & Hierarchical Memory
    The compaction ladder, task-structured summarization, MemGPT-style tiering, pressure-triggered hysteresis, and verifying lossy compaction.
  6. Memory Stores: Vector, KV, Graph & Eviction
    Match backend to memory kind, a unified interface, why unbounded stores rot retrieval, and decay/eviction policies.
  7. Evaluating Memory Quality
    Memory-specific metrics (recall@k, staleness, constraint survival, write precision) and the pitfalls they catch: poisoning, staleness, drift, compaction amnesia.
  8. Memory Write-Path Architectures
    RAG-only is dead for stateful agents — the write path (what earns a slot, when to write, when to update) is the 2026 focus, and the four memory kinds (episodic, semantic, procedural, relational) each want a different policy.
  9. Memory Poisoning Defenses
    AgentPoison at 80% ASR with <0.1% poison; MemoryGraft, SpAIware, Morris-II — lifecycle defenses at ingestion, storage, retrieval, and monitoring.
  10. Long Context: Effective vs Advertised
    Why RULER, NoLiMa, MRCR v2 diverge from advertised token ceilings by 30-60 points past 200K — and how to budget accordingly.
  11. Learned Retrievers & MemRL
    MemRL treats store / retrieve / update / summarize / discard as tools optimized via RL — rank by learned utility rather than semantic similarity alone.
  12. Forgetting & Supersession
    Forgetting is five structurally different mutations — supersession, decay, amnesia, purge, drift — and four need intent that only exists at write time, so no recency weight or reranker recovers them. A small model RL-trained to answer from a fact's current value doubled its accuracy and reached 16.7%; rankings also invert with tenure, a curated map falling 96% to 72% by nine weeks while a provenance-typed graph rises to 90%. Put a mutation control plane in front of the store, resolve entities exactly, and make current-ness a query invariant.
  13. KV Cache Budgeting for Agent Loops
    The weight file says whether the model loads; the KV cache says whether the agent runs, and on a long loop it is routinely larger: a 70B-class model at bf16 cache precision costs about 0.31 MB per token, so 128K context is 40 GB per concurrent session against 42 GB of 4-bit weights. The formula, a worked 80 GB budget, the four levers that change the per-token cost and what each charges, how to reconstruct a vendor's 'fits in 24 GB' claim by division, and why the context cap you enforce — not the average session — is the capacity control.
  14. Memory Scopes & Tenancy
    Every memory layer takes the scope as an argument the caller declares — Mem0's user_id, Graphiti's group_id, Cognee's dataset — and a declared scope is not an authorisation decision; Graphiti's own docs say its reference servers let callers read any group_id. The leak is silent because similarity carries no provenance and cross-tenant neighbours are the most similar. Close the scope over the authenticated session, post-filter so a store-side bug fails closed, default automatic writes to the narrow scope, and ship the thirty-line cross-tenant recall test.
  15. Files as the Context Window
    Offloading a large observation to disk is not compression — it converts a mandatory carry into an optional read, and the entire saving lives in two numbers nobody logs: how often the file is re-read and how much of it is read each time. A 48,000-token result at step 3 of a 20-step run costs 816,000 input tokens inline, 816,510 if the agent re-reads the whole file every turn (plus 17 round trips, so strictly worse), and 12,510 if it greps and reads a 2,000-token slice six times. Covers why a path is not a pointer, why files may be read into the tail of the context and never written into its head, and the three things — replay, concurrency, taint — that stop being true once working memory lives on a disk.