Mem0 has been reported at 49.0% and at 94.4% on LongMemEval — same benchmark name, same product, different harness and different year. That spread is wider than the gap between any two of these products, which means the published scores cannot decide anything for you. What can decide it is the question the scores never ask: who writes to memory, who invalidates it, and whether you can get a person back out of it.
At a glance
Four products, four different answers to "what is memory, structurally" — and the structural answer is what you inherit.
| Project | Structural shape | Who decides what is remembered | Deployment |
|---|---|---|---|
| Mem0 | Memory layer over a vector store, scoped by conversation / session / user / organisation | An extraction pass promotes facts between scopes | Open-source library or hosted platform |
| Zep | Temporal knowledge graph (Graphiti) — entities as nodes, facts as edges with validity intervals | An ingestion pipeline resolves entities and dates the edges | Open-source graph engine or hosted service |
| Letta | Agent runtime where memory blocks are labelled, size-capped regions of the context window | The agent itself, via tools, plus background "sleep-time" agents | Self-hosted Letta Server or Letta Cloud |
| LangMem | SDK of memory primitives over the LangGraph store | You do — it hands you extraction and search, not a policy | Library inside your LangGraph app |
Note what the third column is doing. Mem0 and Zep decide for you with a pipeline you configure. Letta hands the decision to the model. LangMem hands it to you. That is the actual axis of choice, and no benchmark measures it.
Why the scores cannot arbitrate
Every product in this category publishes numbers, usually on LongMemEval, LOCOMO or DMR. Read a year of them together and the picture is not a ranking, it is a demonstration of harness sensitivity.
A widely-cited comparison favouring Zep put Mem0 at 49.0% against Zep's 63.8% on LongMemEval, and Zep's paper reports 94.8% on DMR against 93.4% for MemGPT — the research system Letta grew out of. Mem0's own 2026 write-up of its token-efficient algorithm reports around 94.4% on LongMemEval and 92.5% on LOCOMO, at under 7,000 tokens per retrieval call against 25,000-plus for a full-context baseline. A third-party roundup this year measured LangMem at 58.1% on LOCOMO with a p95 search latency of 59.8 seconds against Mem0's 0.2 seconds — a three-hundred-fold gap that tells you about two integrations, not two products.
None of those numbers is dishonest. They differ because a memory score is a measurement of an entire pipeline — which model did the extraction, how the corpus was chunked, what the retriever was allowed to see, whether the harness let the system re-query. This is the same problem as the agent harness, and it is worse here, because in memory the vendor is the harness. Treat a vendor-run comparison of a competitor the way reading benchmarks critically says to: as evidence about what the vendor optimised, not about the ordering.
Four write paths
Mem0 — extraction and promotion between scopes
Conversations pass through an extraction step that distils durable facts, and those facts are held at a scope: this conversation, this session, this user, this organisation. Promotion between scopes is the interesting mechanism — a preference observed once is not the same object as one confirmed across sessions. It is the easiest of the four to add to an agent that already exists, and the broadest in framework support, which is most of why it is the most widely adopted. The cost of that convenience is that the extraction policy is largely someone else's, and extraction is exactly where a wrong write becomes a permanent belief.
Zep — a temporal graph that dates its facts
Zep's Graphiti engine resolves the conversation into entities and relationships, and every edge carries a validity interval: this was true from then until now. That single design choice answers the question every other system fudges — what happens when a fact changes. The old edge becomes history rather than a contradiction, and the agent can reason about when. It is the strongest option when your domain is full of facts that expire (account states, roles, addresses, prices), and the reported cost is a retrieval path in the region of 50–150 ms for graph traversal against 10–50 ms for a vector-only lookup. It is also the most opinionated ingestion pipeline of the four, and entity resolution is a step that can quietly merge two customers.
Letta — the agent edits its own memory
Letta is not a memory library you drop into an agent; it is an agent runtime in which memory is the primitive. Memory blocks are labelled, character-capped regions of the context window that the agent edits through tools, and blocks can be shared between agents. Its distinctive feature is sleep-time compute: background agents that share a primary agent's memory and reorganise it while nothing is being asked of it, so consolidation happens off the critical path instead of inside a user's turn. This is the right shape for long-running autonomous agents that must accumulate a working self-model, and it is the wrong shape if you already have an agent framework you like — adopting Letta is adopting a runtime.
LangMem — primitives, and the policy is yours
LangMem is LangChain's SDK for adding long-term memory to LangGraph agents over the LangGraph store, with extraction for semantic, episodic and procedural memory. It gives you the pieces and expects you to decide the policy: when to write, what to promote, what to expire. If you are already on LangGraph and you have opinions about your domain's memory semantics, that is the cheapest possible integration and the one you can debug. If you do not have those opinions yet, an SDK of primitives is a way to build a bespoke memory system by accident.
The two axes that survive adoption
Invalidation and deletion are where these systems genuinely diverge, and they are the axes nobody benchmarks — partly because they are hard to score and mostly because they only hurt in year two.
Invalidation. Zep is alone in treating a changed fact as a first-class event with a date attached; the others handle it as an overwrite, a contradiction the retriever must resolve, or a policy you wrote. This matters more than accuracy on a benchmark, because a stale fact retrieved with confidence is the single most common way a memory-enabled agent embarrasses you — and unlike a retrieval miss, it does not look like a failure in the trace. The general treatment is in memory write-path architectures.
Deletion. All four derive state — an embedding, an extracted fact, a summary, a graph edge — and none of those artefacts carries the person's name by the time it is written. Zep's greatest strength inverts here: a graph that marks facts historical rather than removing them is doing the opposite of what an erasure request demands, and edges asserted by other people's conversations still describe your subject. Letta's shared memory blocks raise the same question with a different shape: a block contributed to by three agents and edited by a sleep-time agent has no clean per-subject boundary. Mem0's scope promotion means one user's fact may already have been elevated to an organisational one. LangMem leaves the whole problem with you, which is at least honest. We wrote the operational version of this up separately in erasure requests against agent memory; the buying-decision version is: ask each vendor to delete a synthetic subject and then search for them semantically, before you sign.
Portability. A vector store you own is portable; a hosted graph with vendor-assigned entity IDs is less so; a runtime that holds your agent's identity is the least. Rank the four on how much of your memory you could carry to a different product in a week, and the ordering is close to the reverse of how much each one does for you — which is the ordinary shape of this trade, not a criticism of anyone.
When to pick which
| Situation | Pick | Because | Watch out for |
|---|---|---|---|
| Existing agent, want personalisation this month | Mem0 | Lowest integration cost, broadest framework support, scoped memory out of the box | You inherit someone else's extraction policy |
| Facts that expire — accounts, roles, prices, entitlements | Zep | Validity intervals on edges make "when was this true" answerable | Entity resolution can merge two people; graph reads are slower |
| Long-running autonomous agent that must accumulate a self-model | Letta | Memory is a runtime primitive; sleep-time agents consolidate off the critical path | Adopting it means adopting a runtime, not a library |
| Already on LangGraph with clear memory semantics | LangMem | Primitives you compose, in a store you already run and can debug | No policy is supplied, so you will write one whether you meant to or not |
Run the same evaluation on all four, on your own transcripts, before reading anyone's number: fifty real conversations, twenty questions whose answers changed mid-corpus, and one synthetic person you then ask each system to forget. That takes about a week and it measures the three things that decide the outcome — recall on your domain, behaviour when a fact changes, and whether deletion works. In this category the vendor is the harness, so the only comparable benchmark is one you ran yourself.
FAQ
Are the vendor benchmark numbers dishonest?
Mostly no — they are answers to questions each vendor chose. A memory score measures an entire pipeline, including the extraction model, the chunking, and how much the harness let the system re-query. Two honest teams measuring the same product a year apart with different scaffolds will legitimately produce numbers tens of points apart. That is a reason to distrust the ordering, not the people.
Do I need a memory product at all?
Often not. If your agent's useful context fits in a window and your sessions are short, a well-managed conversation buffer plus retrieval over your own documents beats a memory layer, and skips a whole class of stale-fact and deletion problems. Memory earns its keep when facts must survive across sessions and change over time — see agent memory for where that line falls.
Can I use two of them together?
You can, and the common pairing is a general memory layer for user preferences plus a graph for the domain entities that change. It is a real architecture rather than indecision, but it doubles the write paths, and every write path is a place a wrong fact can enter — which is the attack surface described in memory poisoning defenses.
How much does memory cost in tokens?
Less than the full-context alternative, which is the whole point: Mem0 reports its retrievals at under 7,000 tokens against 25,000-plus for stuffing the history in. But the write side is not free — extraction, summarisation and consolidation are model calls that happen per turn or per session, and they are easy to leave out of a cost model that only counts the read path.
What should I benchmark instead?
Three things, in this order: recall of facts your users actually referred back to; correct behaviour when a fact changed mid-conversation; and end-to-end deletion of one subject verified by semantic search, not by key lookup. The first is what the public benchmarks approximate, and the second and third are what decide whether you are still happy in a year. Evaluating memory covers how to build that set.
Further reading
On this wiki:
- Agent memory — what the word covers, and when you need it.
- Memory types — episodic, semantic, procedural, relational.
- Memory write-path architectures — the axis these four differ on.
- Evaluating memory — building the eval this post says to run.
- Erasure requests against agent memory — the delete path, operationally.