The headline is that agents wrote 436 AWS service emulators and no engineer read the diffs. The part worth your afternoon is the sentence buried in the test suite's README — behaviour that no fidelity test covers is not verified — because it says exactly what made this safe to do, and it is not the agents. AWS relocated correctness out of code review and into an oracle it already owned: a machine-readable spec for every API, and a live reference implementation to diff against. That is the expensive prerequisite, and almost nobody else has it.
At a glance
AWS Emulator launched on 11 November 2026 as a local emulator of the AWS APIs — no AWS account, no service fee, point your SDK at it and run your tests. Two repositories shipped, and the second one is the interesting one.
| Repository | What it is | Language / licence | Who maintains it |
|---|---|---|---|
aws/aws-emulator |
The emulator itself; one crate per service | Rust, CC0-1.0 | Agents, per AWS |
aws/aws-emulator-evaluation-suite |
The fidelity tests, plus a runner | Python, CC0-1.0 | Agents; read-only, no issues or PRs |
What actually shipped, counted rather than quoted
The workspace manifest is public, so the scope is checkable instead of reported. On main as of 13 November 2026, Cargo.toml lists 436 service members, one crate per AWS service, named in the service-com-amazonaws-<namespace>-<service> pattern. Each of those crates contains a smithy-model/ directory holding the service's Smithy JSON — described in the repository's own layout notes as "the Smithy model that drives the wire protocol" — and a build.rs that generates the service declaration from that model.
That structure is the whole story in one directory tree. The wire shape of every API is not something an agent had to infer from documentation; it was generated from a machine-readable artefact AWS has published for years. The agents were writing behaviour against a spec that already pinned down shape.
Some other facts from the README that colour how you should use it: it serves HTTPS on 127.0.0.1:9297 with a self-signed certificate; there is an embedded mode that compiles the emulator into your Rust test binary, one isolated instance per test; building from source wants at least 100 GB of free disk for target/, because a debug build compiles every service crate; telemetry is on by default and turns off with --telemetry off or EMULATOR_TELEMETRY=off. And there is a first-class agent path — aws agent-toolkit add-skill --skill-name aws-emulator, requiring AWS CLI 2.35.0 or later — which tells you who the intended user is.
The README is also unusually blunt about the boundary: "AWS Emulator is for building and testing. Don't use it for production workloads or to test security." Take that literally. An emulator that skips authorization checks will happily accept calls a real account would reject, which makes it useless as a security surface and dangerous as a confidence signal about one.
The sentence that is the whole story
The evaluation suite is where the engineering argument lives. A fidelity test is a short Python program, check_<name>.py, that uses boto3; its run(ctx) function drives a workflow and asserts how AWS behaved — the resulting state of the resources and, where applicable, the error code and the error message. Then comes the line that should be printed on a poster in every team that is thinking about this:
Every fidelity test was verified against at least two sandboxed AWS accounts, so it asserts real AWS behavior. Behavior that no fidelity test covers is not verified.
Read both halves. The first is a provenance claim: these assertions are not somebody's reading of the documentation, they are recorded observations of the real system, cross-checked across accounts so that account-specific quirks do not get baked in as truth. The second is a scope disclaimer, and it is the more valuable sentence, because it converts an unbounded question — "is this emulator correct?" — into a bounded one: "is the behaviour I depend on covered by a test?" That question has an answer you can look up, per service, per release.
This is what it means to have an oracle: a source of truth about correct behaviour that exists outside the code under review. With one, review is mechanical — run the tests, read the score. Without one, review is a human reading a diff and asking themselves whether it looks right, which is the activity AWS removed.
The merge gate is a spec problem wearing an agent costume
AWS's own framing, in the launch posts, is that engineers decided what should be built and created the specifications, tests and merge gates that determine correctness, while agents wrote and checked the code — with every change passing a merge gate that includes security agents doing adversarial review. Launch coverage attributes a sharper version to Matt Goldberg, a Senior Principal Engineer on the AWS Developer Tools team: that human engineers wrote zero lines of implementation code and reviewed zero pull requests. Treat the specific numbers as AWS's account of its own process; the structure is what matters and the structure is visible in the repositories.
The swap is favourable precisely because of the task. For an API emulator, "idiomatic and maintainable" is close to worthless and "byte-identical behaviour to the reference" is everything — and those are the two columns the chart separates. A human reading a diff of a DynamoDB UpdateItem implementation cannot tell you whether set-list element order is preserved the way real DynamoDB preserves it. A test that ran against real DynamoDB can. The gate is not better than a human at reviewing code; it is better at the only question this codebase needs answered.
Which also tells you where it would fail. Invert the task — a product with taste-dependent behaviour, no published spec, and no reference implementation — and the gate has nothing to check against, so it degenerates into style linting plus whatever the reviewer agent's priors happen to be. The pattern here is the one argued in gold runs and the unsolvable task: the ceiling on automated review is the quality of the oracle, not the capability of the reviewer.
What does not transfer
If you are about to propose "we should do what AWS did," here is the list of things they had that you need to check for first.
- A machine-readable spec. Smithy models, one per service, already public and already the source of truth for AWS's own SDKs. Your equivalent is an OpenAPI document that is actually generated from the service rather than hand-maintained beside it. If your spec drifts from your service, it is not an oracle, it is a second thing to be wrong.
- A live reference implementation. AWS could run a candidate behaviour against real AWS and record the answer. This is the piece most teams simply do not have: when you are building the only implementation, there is nothing to diff against, and your tests encode your beliefs rather than observed truth.
- A task with crisp, external success criteria. "Does this API return the same error code" is checkable. "Is this the right abstraction for our billing domain" is not, and no merge gate will make it so.
- Tolerance for a known, published gap. AWS shipped with per-service fidelity scores on every release and a directory of tracked failures —
issues/things-that-do-not-work/, with fixed items moving toissues/things-that-now-work/. That is a product decision as much as an engineering one, and it is the part most organisations will refuse.
That last point deserves emphasis because it inverts the usual instinct. The impressive thing about this release is not a claim that the agents got it right. It is that the gap is enumerated, versioned and published, so a user can answer "will my project work" by reading a list rather than by trusting a vendor. An agent-built artefact with no published gap list is strictly worse than this, however good its agents were.
The blind spots, which are also published
Read the runner's isolation section and you get an honest map of what this process does not defend. The runner accepts only a loopback endpoint, signs with the keys the emulator wrote, and strips your AWS credentials, profiles and proxy settings from the environment before running a test — all good. Then: "It does not restrict the network access of tools and containers a test starts." Fidelity tests are arbitrary Python that may deploy applications and shell out to command-line tools, and that caveat is where the trust boundary actually sits.
So the practical posture for running the suite yourself is the same one you would take for any third-party test corpus that executes code: run it somewhere disposable. The tests are CC0 and agent-authored in a repository that accepts no pull requests, which is a sensible way to keep a corpus coherent and also means the usual community-review signal is absent by design. Combine that with "the repository is read-only" and you have an artefact you should consume as data, not as code you have vetted — the reasoning in vetting a third-party skill applies without modification, and the official aws-emulator skill is exactly such an artefact.
One more that is easy to miss: telemetry is on by default. The README says records never include request or response contents, code, credentials or personal data, and documents two ways to turn it off. If you are putting this in CI across an organisation, decide that once, centrally, rather than per developer.
What to do with this
Two separable things happened here, and they have different half-lives.
The tool. A free, local, no-account emulator of 436 AWS services with per-release fidelity scores is immediately useful for the thing agents are worst at: running a test loop without touching shared infrastructure. It is a better default than mocking, because mocking a cloud API deletes the error shapes and latency that cause your real failures — the argument in staging environments for agents. Check the fidelity score for the specific services you depend on before you believe a green suite, and keep the real-AWS integration run; the emulator moves where you spend it, it does not remove it.
The process. Before copying it, write down your oracle. Not your test suite — your oracle: the thing that tells you the answer is right, independent of the code that produced it. If you can name it, and it is generated rather than hand-written, and you can run a candidate against it, then an agent-written implementation with an automated merge gate is a defensible design. If you cannot name it, what you would be shipping is unreviewed code with a green check mark, and the published-gap-list part — the part that makes AWS's version honest — is the part you would be least likely to copy.
FAQ
Did humans really review none of the code?
That is AWS's account of its own process, stated in its launch materials and reported in launch coverage, with the "zero implementation lines, zero pull requests reviewed" formulation attributed to a Senior Principal Engineer on the team. What is independently checkable is the shape: the code is public, the commit history is public, the fidelity tests are published separately, and both repositories decline pull requests.
Is it a LocalStack replacement?
For API-surface breadth and published fidelity, it is a serious new option; for ecosystem, tooling and years of accumulated workarounds, LocalStack is far ahead. The choice turns on something more specific than service counts, which we work through in the four-way comparison.
Can I trust an emulator written by agents?
Trust the fidelity scores, not the authorship. The authorship tells you nothing either way; the per-service, per-release score plus the tracked-gap directory tells you precisely what has been verified and what has not. That is more than most hand-written emulators give you.
Does this mean agents can now write production services unsupervised?
It means they can when correctness is externally specified and cheaply checkable at scale. An API emulator is close to the best case ever constructed for that: a published wire spec, a live reference implementation, and behaviour that is either identical or not. Most services are not that.
Should I point my coding agent at it today?
Yes, for local and CI test loops, with two caveats: check the fidelity score for your services, and do not read a green emulator run as evidence about authorization or security, which AWS explicitly excludes.
Further reading
On this wiki:
- Gold runs and the unsolvable task — why the oracle, not the reviewer, sets the ceiling.
- Staging environments for agents — why mocking a tool deletes the failures you needed to see.
- Vetting a third-party skill — how to consume an agent-authored artefact you did not review.
- Patch generation and tests — the same gate, at the scale of one change.
- Eval-driven agent development — building the measurement before the thing.
Sources:
- aws/aws-emulator — the emulator; README, workspace manifest and repository layout.
- aws/aws-emulator-evaluation-suite — the fidelity tests, the runner and its isolation notes.
- AWS Emulator User Guide — documentation, including the security and telemetry pages.