Multi-Tenancy for Agents

The isolation you bought is around your account, not around your tenant.

Providers do isolate their prompt caches. Anthropic states that cache entries are isolated between organizations and, on several platforms, between workspaces within an organization; OpenAI states that prompt caches are not shared between organizations. Read those sentences from the position you are actually in: you are one organization, and all of your customers live inside it. The vendor's boundary separates you from other vendors' customers. It does nothing between your tenant A and your tenant B, because from the provider's side there is only you.

What that does and does not mean is worth being precise about, because both the panic and the shrug are wrong:

• A prefix cache cannot hand over content. A hit requires a byte-identical prefix. Tenant B's request only hits an entry created by tenant A if tenant B already sent exactly those bytes — in which case B had the content to begin with. This is the one cache in the list that is structurally incapable of leaking data.
• It can still leak an inference. A cache hit is faster and cheaper than a miss, and both facts are visible to the caller in latency and in the usage fields on the response. A tenant who can guess a prefix can learn whether someone else has sent it. For most products this is noise; if your tenants are competitors, or the prefix could encode a customer name, it is a side channel and you should keep tenant-identifying content out of the cached prefix entirely.
• Cache economics are shared, which is a billing problem. Cache writes cost a premium and reads a fraction; whoever's request happened to write the entry paid for the others. If you bill by usage this quietly cross-subsidises between tenants — see cost attribution for why the invoice will not explain itself.

The practical rule is simple and it survives every provider's implementation detail: the cached prefix carries your system prompt, your tool schemas and your shared instructions; the tenant's data goes after the cache breakpoint. That is also what prompt caching wants for hit-rate reasons, which is a rare case of the secure design and the fast design being the same design.

The semantic cache is the only one that can be wrong, and it is the one teams share.

Every other cache in your stack can only ever be slow or stale. A semantic cache decides a hit by embedding distance, so "what is our renewal date" from tenant B can match the stored answer to "when does our contract renew" from tenant A — and return it, confidently, with no error anywhere. There is no threshold that makes this safe, because the whole mechanism is approximate matching, and the queries that are most similar across tenants are exactly the high-value questions everyone asks in the same words.

• Tenant ID belongs in the cache key, not in the filter. The lookup must be scoped before the nearest-neighbour search runs, not applied to its results. A post-filter still performed a cross-tenant search, which means an implementation change or an indexing bug reopens the hole silently.
• A shared cache is legitimate for tenant-independent content only. Product documentation, public policy text, "how do I export a CSV" — content where the answer does not depend on who asked. Keep that in a separate, explicitly-shared cache with its own key namespace, so the decision to share is written down rather than inherited.
• Instrument the false-hit rate as a safety metric, not a performance one. Sample hits, re-answer without the cache, and compare. A semantic cache is a retrieval system with a false-positive budget; if you do not know the number, you have not set the budget. See semantic caching.
• Consider not having one. For most agent workloads the hit rate is low — agent prompts carry long, unique conversation state — and the prefix cache already captures the bulk of the savings without the failure mode.

In the vector index, a namespace and a metadata filter are not the same control.

Every vector database offers both a way to partition data and a way to filter it, and the documentation makes them sound interchangeable. They are not, and the difference is visible in exactly the situation you care least about it: under load, at scale, in an approximate index.

• Pre-filtered or partitioned search restricts the candidate set before the approximate search runs. This is a boundary. Correctness does not depend on the search returning enough neighbours.
• Post-filtered search retrieves top-k globally and then discards the rows that fail the filter. This is not a leak of content, but it is a recall cliff: for a tenant whose documents are a small fraction of the corpus, top-50 across all tenants may contain two of theirs, and the agent answers "I found nothing" on a corpus that has the answer. Teams debug this for weeks as a model problem.
• Separate indexes per tenant are the strongest and the most operationally expensive option. They also make deletion honest — a tenant offboarding is dropping an index rather than a filtered delete you then have to prove. If you are subject to per-customer deletion commitments, price this in before choosing the shared-index design; retrofitting it means a full re-index. See choosing a vector database and data governance.
• Reranking and hybrid search re-open the question. A reranker fed candidates from a shared pool, or a keyword index that was never partitioned alongside the vector one, will happily undo a careful vector-side boundary. The rule is that every retrieval path carries the tenant scope, including the one someone added last quarter for a demo.

Memory is where tenants bleed, because the write path summarises.

Retrieval reads what you put in. Memory writes something new — a distilled fact, a preference, a lesson — and the distillation step is a model call that will happily merge whatever is in its context. The failure is not a query returning the wrong row; it is a summary that was authored from two tenants' material and is now durably stored as one tenant's truth. No filter downstream can undo that, because the offending record is legitimately owned by the tenant it was written for.

• One tenant per summarisation call, always. Batching memory consolidation across tenants for efficiency is the single highest-risk optimisation in this whole page. If a job processes many tenants, it processes them in many calls.
• Scope the memory key by tenant and by user, and decide which one owns the memory. "Our company prefers metric units" and "Priya prefers terse answers" have different owners and different deletion semantics, and conflating them means a departing employee's preferences persist as company policy.
• Resist cross-tenant learning, or make it an explicit product with consent. "The agent gets better for everyone as it learns" is a compelling roadmap line and a contractual problem; anything derived from tenant A's data and applied to tenant B is a use of their data that your DPA probably did not contemplate. See memory write-path architectures.
• Deletion has to reach the derived layer. A tenant deletes a document; the memory record distilled from it two months ago is still there, and so is the vector. Enumerate every store that can hold a derivative before you sign a deletion SLA. Memory poisoning defenses covers the adversarial version of the same write path.

Noisy neighbours are a capacity problem with a latency signature.

Provider rate limits are enforced per account, so your tenants share one bucket whether or not you modelled it. One tenant running a bulk import can consume your organisation's tokens-per-minute and the symptom every other tenant sees is a slow agent, not an error you can attribute. On self-hosted inference the same dynamic is sharper: a single 200k-token context occupies KV cache that would have held several ordinary sessions, so one tenant's long document reduces everyone's concurrency — the arithmetic is in self-hosted inference for agents.

• Give every tenant a token budget per window, enforced by you, below the provider ceiling. Your queue, your fairness policy, your error message. Discovering fairness through the provider's 429 means the loudest tenant wins.
• Separate the interactive lane from the batch lane. Bulk work is where the volume is; a user waiting on a chat turn should never be behind it. Route long jobs to the batch tier where the economics are better anyway.
• Alert on per-tenant p95, not fleet p95. A fleet average stays healthy while one tenant is comprehensively broken — and that tenant is the one who will email you.
• Decide who eats a runaway loop. An agent that loops burns a tenant's budget in minutes. Per-tenant spend caps with a hard stop, plus the kill switch scoped to one tenant rather than the fleet, is the difference between one bad afternoon and an outage for everyone.

Prove it, because none of these failures raise an error.

Every leak on this page is silent by construction: the wrong answer is well-formed, the degraded recall looks like a knowledge gap, the noisy neighbour looks like a slow model. Nothing in your error budget moves. So the only honest posture is a test that tries to cross the boundary on purpose, running continuously.

• A two-tenant canary suite in CI. Two synthetic tenants with deliberately similar corpora and one unique, distinctive secret each. After every change, ask tenant B every question whose answer is tenant A's secret, through every path: retrieval, cache, memory, and the agent end to end. This suite is cheap, it never expires, and it is the only artifact that will let you answer a security questionnaire honestly.
• Tag every span with the tenant. Traces without a tenant ID cannot answer "did this run read anything it should not have", which is the first question asked in an incident. Agent observability is the substrate; the tenant tag is what makes it a control.
• Test the deletion path, not the deletion API. Insert a distinctive document for a tenant, let the agent run long enough that it lands in memory and in every index, then delete it and re-ask. Most teams discover their derived stores this way.
• Rehearse the disclosure. If a cross-tenant leak did happen, could you enumerate which tenants and which records from your traces? If the answer is no, that is a gap in audit trails, and it is the difference between a scoped notification and a blanket one.

Related: per-customer economics for the billing side of the same partition, data residency and sovereignty for when the boundary must also be geographic, and RAG security for the adversarial reading of step 3.