Bot verification & agent web access.
The web spent thirty years guessing who was knocking and has started checking signatures instead — and the practical consequence for anyone building an agent is that your cryptographic identity is now a retrieval-quality setting. An unverified agent rarely gets a clean refusal. It gets a stale page, a challenge, a payment demand, or a version of the document with the prices removed, and every one of those parses fine and grounds your answer in something wrong.
Three regimes, and only one of them has teeth.
Three mechanisms sit on top of each other and are constantly confused for one another. They are not peers.
robots.txt— a request, honoured voluntarily, with no enforcement anywhere in the stack. It has been the polite convention since 1994 and it tells you what a site would prefer. It stops nothing.llms.txt— a proposal to hand models a curated, clean version of a site. Adoption sits around 10%, and as of early 2026 no major model provider consumes it in production; the effort has effectively flatlined. Publishing one costs you an afternoon and buys close to nothing today.- Edge enforcement — the CDN or WAF in front of the origin, deciding per request whether to serve, challenge, charge or drop. This one has teeth, because it is the layer that actually holds the connection.
The third has been moving fast. Cloudflare now blocks AI crawlers by default on new domains, folded per-crawler verification, robots management and pay-per-crawl into one console across all plan tiers in April 2026, and disclosed that its network returns more than a billion 402 Payment Required responses to crawlers per day. From 15 September 2026 its defaults also block "mixed-use" crawlers from pages carrying ads. Whatever you believe about the politics of that, it is the layer your agent's request will meet.
If you are debating what to put in robots.txt or whether to publish an llms.txt, you are discussing the two files with no enforcement behind them. The decision that changes outcomes is made at the edge, by a rule you did not write, before your request reaches an application.
Verification replaces inference.
The old way of identifying a client was inference: a User-Agent string, an IP range, a behavioural fingerprint, a CAPTCHA. All of it guesswork, all of it spoofable, and the errors run in both directions — impersonators get through, and legitimate agents get blocked for looking wrong.
Web Bot Auth replaces the guess with a check. It is an IETF draft led by Cloudflare, built on the already-standardised RFC 9421 HTTP Message Signatures: the agent signs its requests with a private key, publishes the corresponding public key, and any server or intermediary verifies the signature before deciding what to serve. A working group was chartered in 2026, and implementations exist at Cloudflare, Akamai, AWS WAF, Vercel and Shopify. It has also been adopted as the authentication foundation under agent-payment work including Visa's and Mastercard's, which is what makes it likely to stick — payments do not tolerate ambiguous identity.
Be precise about what a signature settles. It proves the request came, unmodified, from the holder of a particular key. It says nothing about what that agent will do with the response, whose behalf it acts on, or whether its operator is trustworthy. It is an accountability primitive, not an authorization one: its value is that misbehaviour can now be attributed and a specific key revoked, instead of a whole IP range being blocked and half the internet's agents with it. Authorization is still a separate decision — the one identity and attestation is trying to make answerable.
Unverified traffic degrades; it rarely fails cleanly.
This is the part that catches builders, because a hard 403 would be a gift. What an unverified agent actually meets is a spectrum, and most of it looks like success:
- A challenge page returned with
200 OK. Your parser reads it happily and finds no product listings, so the model concludes there are none. - A
402, which is a price rather than a refusal — and one nobody on your team has decided whether to pay. - Stale cached HTML, served deliberately to unverified clients. Correct-looking, structurally intact, and out of date by a week.
- A thinner document: the article without the tables, the listing without the prices, the page without the JavaScript-rendered half.
- Rate limiting that only bites on the eleventh call, which is exactly where a multi-step research task lives and a smoke test does not.
Every one of those returns a body that parses. That makes this a grounding failure rather than an availability failure, and grounding failures are the ones your evals miss, because the run completes, the answer is fluent, and nothing raised an exception. An agent that quietly reads the stale version of a pricing page will state last week's price with complete confidence — the failure mode grounding exists to prevent, arriving through the network layer instead of the model.
What this changes in your build.
- Sign, and use one key per purpose. If your crawler, your user-facing browsing agent and your checkout agent share one identity, one complaint takes down all three. Separate keys mean a ban is scoped to the behaviour that earned it.
- Make blocked-and-degraded its own failure class. Do not fold
402,403, challenge interstitials and robot notices into a generic fetch error alongside timeouts. They mean something different, they are fixable by a different team, and they need to be visible as a rate on a dashboard. - Assert on content, not on status. The check that catches a challenge page is "did this page contain a price field", not "did this request return 200". Content invariants per source are cheap and they are the only thing that detects silent degradation.
- Decide the payment question before you meet it. A
402in an agent loop is a policy question wearing a status code. Answer it deliberately — see agent commerce — rather than letting a retry loop decide. - If you run a site, the default has already changed. Blocking is increasingly what happens if you configure nothing, which means agent traffic to your product is now a choice you are making passively. Decide whether you want to be legible to agents, and if you do, verify them rather than trying to identify them.
Do two things this week. Register a signing key and sign your agent's outbound requests, because that is a one-afternoon change with a permanent payoff and the ecosystem is standardising around it. Then add one content-invariant assertion per external source your agent depends on. The first buys you access; the second tells you the day you quietly stop having it — and without the second, degraded access is indistinguishable from working software.
Related: browser agents for the surface where this hits hardest, agent identity & permissions for the layer above the signature, and egress control for the mirror image — deciding what your own agents are allowed to reach.