Staging Environments for Agents

The service playbook does not transfer, because the environment is not yours.

Staging a service is a solved problem because the dependency graph is tractable: your code, a database you can seed, a message bus, and a handful of internal services you can also deploy. Stand up a copy, load fixtures, run the suite. An agent's dependency graph does not have that shape. Written out honestly, it contains:

• A model you cannot deploy a copy of. It is somebody else's service, it is updated on their schedule, and its behaviour is a distribution rather than a function.
• Third-party SaaS you cannot clone. The CRM, the ticketing system, the payment processor. Some offer sandboxes; the sandboxes have different data volumes, different rate limits and, frequently, a different API version.
• A tool catalogue whose text belongs to someone else. An MCP server's descriptions are part of your prompt and change without a commit on your side — the whole subject of third-party tool drift.
• State that is the point. The agent's behaviour is a function of what it finds when it looks: how many records, how messy, how contradictory. A seeded tenant with twelve rows tests a different program than the one you shipped.
• A human. Approvals, interruptions, half-specified requests, and the follow-up two hours later.

So there is no single environment that is "staging". There is a ladder of fidelity, each rung cheap in one dimension and dishonest in another, and the operational question is not "do we have a staging environment" but which rung is allowed to gate which release step.

Pin the model, keep the tools real.

The instinct people bring from service testing is to replace slow, flaky, external things with fast, reliable fakes. Applied to an agent, that deletes the subject matter:

• Mocked tools return instantly, and latency is most of a trajectory. A twenty-step run against real APIs spends the overwhelming majority of its wall-clock inside tool calls. Mock them and your timeout policy, your parallelism, your retry budget and your context-growth curve are all untested — the same argument that governs load-testing an agent system.
• Mocked tools return the errors you thought of. Real ones return HTTP 429 with a Retry-After you did not parse, a 200 with a partially written record, a field that is now a string where it used to be an integer, and an OAuth token that expired mid-run. Error handling is the majority of agent code and mocks test the half you already understood.
• Mocked tools never drift. The single highest-value thing a staging environment can do for an agent is notice that a dependency changed. A mock is a snapshot of your beliefs, so it agrees with you forever.

The model is the opposite case. Pinning it is nearly free and buys real comparability: a specific model version rather than a floating alias, a fixed system prompt and tool-catalogue hash recorded on every run, a snapshot of the retrieval index, and low temperature where the task allows it. Do not overclaim what that buys you — pinned is not deterministic, for the reasons in reproducibility and determinism, so a green staging run is a sample and not a proof. Run each gating scenario several times and read the pass rate; a scenario that passes four times in five is a flaky feature, not a flaky test.

Four rungs, and the rule about what each may gate.

• Tier 0 — recorded and replayed traffic. Real responses captured once and served back from disk. Seconds to run, deterministic on the tool side, free. Catches prompt regressions, parsing changes, control-flow bugs. Gates every commit. Its failure mode is silent staleness, so treat a recording as perishable: re-record on a schedule, fail the build when a cassette is older than your threshold, and — the part teams skip — hash the live response schema in a nightly job so you learn that the recording is wrong from CI rather than from a customer.
• Tier 1 — vendor sandboxes and test modes. Real API surface, real auth, real error taxonomy, seeded tenants. Minutes to run. Catches integration and schema problems that Tier 0 has frozen out. Gates merge. Be explicit about where sandboxes lie: they are usually smaller, sometimes a version behind, and their rate limits rarely match production's.
• Tier 2 — production dependencies, side effects neutralised at the last hop. The agent talks to the real systems with real data volumes and real latency; only the writes are intercepted. Catches everything that is a function of scale and real-world mess. Gates release.
• Tier 3 — real side effects, bounded blast radius. One internal tenant, an employee cohort, a category of low-value transactions. This is where the human half of the environment finally shows up. Gates general availability, alongside the staged rollout in rollout and versioning.

The rule that makes the ladder useful: a tier may only gate a release step whose failures it could have caught. Letting Tier 0 gate a release is how a team ships an agent that has never once seen a rate limit. Write the mapping down, because the pressure to promote on a green Tier 0 run arrives on the day you are late.

The write-blocker is the artefact. Build it at egress, not in the tool wrapper.

Tier 2 is the rung that pays for itself, and it exists or does not exist depending on one piece of infrastructure: something that lets reads through to production and refuses writes, from outside the agent's own code.

• Put it at the network boundary. A forward proxy that every outbound call traverses, classifying each request as read or write by method and route, allowlisting destinations and failing closed on anything it does not recognise. In the tool wrapper it is a suggestion — a model that writes its own HTTP call, or a framework that adds a tool you did not wrap, walks around it. This is the same control described in egress control for agents, pointed at a different threat.
• A blocked write must return something the agent believes. This is the detail that decides whether Tier 2 tests anything. If the proxy returns a 403, every run becomes a test of your error path and you learn nothing about the happy path. Return a realistic success — a recorded response body with a synthetic id — so the agent proceeds to step nine, which is where the bugs are.
• Prefer the vendor's own dry-run when it exists — validate-only flags, test-mode keys, sink domains for outbound mail — because a vendor's dry run is more honest than your emulation of one. Reserve the proxy for everything without one.
• Log every intercepted write as a proposed side effect and diff it against the previous release. The list of writes an agent wanted to make is the single most reviewable artefact this whole apparatus produces, and it is how you notice that a prompt change quietly doubled the number of records touched.

Seed for shape, and never with a copy of production.

The state is part of the program. An agent that reads a tenant with twelve tidy records will retrieve well, terminate early and look excellent; the same agent against four hundred thousand records with duplicates, three naming conventions and a decade of abandoned custom fields is a different system. Seed data is therefore a test asset with a specification, not a fixture someone generated once.

• Seed for volume, mess and permissions — near-duplicates, contradictory records, empty and enormous fields, mixed languages and encodings, records the agent's credential must not be able to see. That last category is the only way to test that permission-aware retrieval actually holds.
• Include the adversarial rows. A document containing instructions is a normal thing to find in a real corpus. If your seed set has no injection attempts in it, your staging environment has never tested the control you care most about.
• Do not clone production data into a sandbox. It is the fastest route from "we improved our testing" to a reportable incident, because sandboxes have weaker access controls, longer-lived credentials and more people in them. Subset and mask, or generate — and keep the seed set under the same retention rules as anything else, per data governance.
• Version the seed set alongside the code. A quality change you cannot attribute to code or model is usually a data change nobody logged.

What no rung can tell you, and the number that tunes the ladder.

Four things are structurally out of reach, and pretending otherwise is how teams over-invest in staging and under-invest in production safety:

• The real request distribution. Your scenarios are the tasks you thought of; users bring the long tail, and the long tail is where agents fail.
• Real adversarial input. You can seed known attacks. You cannot seed the one someone invents next week.
• Provider-side behaviour under load. Capacity, queueing and degradation on someone else's service are not reproducible on your schedule.
• The humans. Approval latency, ambiguous instructions, and what people do when the agent asks a question at 4:55pm.

So a fixed share of your confidence has to be bought in production — canaries, shadow runs, and the deliberate designs in online experiments for agents — and the ladder's job is to make that share small enough to be affordable, not zero.

Tune it with one number: tier escape rate, the count of production incidents traced back to a failure that a lower rung should have caught, attributed to the rung that missed it. Escapes past Tier 0 mean your recordings are stale or your scenario set has a hole; escapes past Tier 1 mean the sandbox diverges from production somewhere you have not written down; escapes past Tier 2 mean the write-blocker is not covering a path. Every escape ends in a specific addition to a specific rung. Track it beside cassette age and the staging-to-production behavioural delta — the difference in tool-call counts, step counts and refusal rates between the same scenarios in Tier 2 and production — and you have a staging environment that gets more honest over time instead of quietly less.

Related: eval-driven agent development for what runs on each rung, repairing what the agent already did for the cost of the writes you fail to intercept, and the agent harness for why the environment is half of what you are actually shipping.