Three of the four ways to make your site agent-readable ask you to publish something no consumer has committed to reading, and for one of them the evidence is now measured and negative: a server-log analysis of roughly 137,000 domains, published in June 2026, found 97% of llms.txt files received no fetches at all. The useful question is not which of these is a standard. It is whether the agent needs to read your content, ask your data a question, or act on a user's behalf — because only the last requires a caller identity, and only one of the four gives you one.
At a glance
Four surfaces, in the order they tend to get proposed in a planning meeting. Standing and dates are as of October 2026.
| Surface | What you publish | Standing | What reads it |
|---|---|---|---|
| llms.txt | A markdown map of your site at /llms.txt | Informal proposal, September 2024, no standards body | Measured as almost nothing |
| NLWeb | An /ask endpoint over your own content, plus an MCP server per instance | Microsoft open-source project, MIT, launched at Build 2025 | Assistants, where a publisher has wired one up |
| WebMCP | Tools registered by your page's JavaScript | W3C community-group draft; the spec says it is not a W3C Standard | Chrome, behind a flag |
| Remote MCP server | An authenticated endpoint exposing typed tools | MCP, governed under the Linux Foundation's Agentic AI Foundation since December 2025 | Every major agent client |
llms.txt — a hint, and the one surface with evidence against it
What it is
Jeremy Howard of Answer.AI proposed llms.txt in September 2024, with a companion spec at llmstxt.org: a markdown file at the root of your site holding an H1 for the project name, a blockquote summary, and H2 sections of annotated links to the pages a model should treat as canonical. A companion /llms-full.txt inlines the text of those pages, and the proposal also suggests publishing a markdown copy of each page at the same URL with .md appended.
What actually happens
This is the only one of the four where the question "does anything read it?" has been answered empirically, and the answer is unkind. The most-cited study is an analysis of server logs across about 137,000 domains, published in June 2026, which reported that 97% of published llms.txt files got zero fetches in the month measured; among the few that were fetched, GPTBot accounted for 4.51% of requests, ClaudeBot 0.80%, DeepseekBot 0.02%. Smaller single-site log reviews point the same way, several reporting no AI-crawler requests for the file at all over windows of one to two weeks. Adoption estimates conflict — an analysis of around 300,000 domains puts it near 10%, while a tracker of 219 notable hosts found 57.6% serving one — but the panels that report high numbers skew heavily toward technology companies, so the two figures are not in tension so much as measuring different populations.
Google's position is worth reading in full because it is split by product. Search Central's guidance on AI features says site owners do not need to create machine-readable AI text files to appear in Search's generative features, because Search does not use them; John Mueller has publicly compared the file to the keywords meta tag. Meanwhile Chrome's Lighthouse 13.3, released in May 2026, promoted an "Agentic Browsing" audit category out of experimental and into the default configuration, and that category includes an llms.txt check. The detail that settles the matter: the audit returns Not Applicable when the file is absent, passes when it exists and is well-formed, and fails only when the server errors fetching it. Even the tool that looks for it declines to penalise you for not having one.
What to do with it
Publishing one costs an afternoon, so the cost side of this decision is not the issue — the issue is that an unread file goes stale invisibly, and a stale map of your site is worse than no map when something eventually does read it. The part of the proposal that has held up is the other part: serving clean markdown of each page. That is fetched, because it is the page, and it helps every consumer from a crawler to a browser agent. If you do one thing from this section, do the .md copies and treat the index file as optional.
NLWeb — the right shape, waiting for a consumer
What it is
Microsoft launched NLWeb at Build 2025 as an MIT-licensed project: you point it at content you already publish — schema.org markup, RSS, a product catalogue — and it stands up a natural-language /ask endpoint over it, with each instance also exposed as an MCP server so an agent can call it as a tool. The framing at launch was HTML for the agentic web: one endpoint per site, answering questions about that site's own data.
What actually happens
The early adopter list — Shopify, TripAdvisor, Eventbrite, O'Reilly, Hearst properties — has barely moved since launch, and coverage through mid-2026 describes adoption as flat. The repository still characterises its implementation code as proof-of-concept demonstrations rather than a production answer, and hosted deployment on the major clouds was still listed as coming soon; the .NET implementation is labelled experimental, leaving Python as the only reference implementation anyone describes as deployment-ready. Cloudflare added NLWeb support through its AI Search product in early 2026, which gives the protocol a second infrastructure backer beyond Azure. Analyst commentary has been patient rather than bullish, with Constellation Research's Michael Ni putting substantial adoption two to three years out.
The reason is a chicken-and-egg problem rather than a technical flaw, and it is worth naming precisely: without an assistant querying /ask at scale, a publisher has no reason to run one, and without publishers running them, no assistant has a reason to commit. That is the same deadlock the M×N interop problem describes, and MCP broke it only because the consumer side — the coding agents — moved first and in volume.
What to do with it
NLWeb is the correct shape for "let an agent ask questions about my catalogue", and if you already have the schema.org data it is genuinely cheap to stand up. Treat it as a bet on the consumer side arriving, size the work accordingly, and note that the valuable artifact is the retrieval index underneath — which serves your own site search and your own MCP server too, so the work is not wasted if the protocol stalls. Index freshness and invalidation is the part that will actually cost you over time.
WebMCP — the only one that acts, and the only one with no caller
What it is
WebMCP lets a page register tools with the browser from its own JavaScript, so an agent operating in the tab can call a typed function instead of driving the DOM. It is a Draft Community Group Report of the W3C Web Machine Learning Community Group, edited by people from Microsoft and Google; the spec page, revised 20 October 2026, states plainly that it is not a W3C Standard and is not on the W3C Standards Track. Chrome shipped an early preview behind a chrome://flags entry, reported as landing in version 146 in February 2026; Edge support is described by its own co-authors as expected rather than shipped, and a review in mid-2026 found no reliable evidence of a generally available WebMCP calling path in agent clients.
What actually happens
It is early, and that is the smaller half of the story. The structural issue is that a registered tool executes as your page's own JavaScript, inside the session the signed-in user already established — so your server receives a request it cannot distinguish from a click. The caller has no identity of its own, which means you cannot authorise it, rate-limit it separately, or revoke it without revoking the user. That is the argument in WebMCP makes your page an API, and it is why WebMCP does not slot neatly beside the other three: it is a better driving surface than the DOM and it is not an authorisation boundary at all.
What to do with it
Build it when you have a UI an agent genuinely needs to operate — a multi-step form, a configurator, something where the state lives in the page — and treat every registered tool as a public endpoint reachable by whatever is in the tab. That means server-side validation on the actual mutation, not in the tool handler, and no tool that performs an irreversible action without the same confirmation a human click would get. Tool schemas and contracts applies unchanged; rendering agent output safely covers the return path.
A remote MCP server — the boring option that is winning
What it is
An authenticated endpoint exposing typed tools, which any MCP client can connect to. MCP's governance moved to the Linux Foundation's Agentic AI Foundation in December 2025, which matters less for the protocol's design than for the fact that it is no longer one vendor's project. The platform vendors have followed: Google Cloud announced fully-managed remote MCP servers for Google and Google Cloud services on 10 December 2025 — Maps Grounding Lite, BigQuery, Compute Engine and GKE first, with Cloud Run, Cloud Storage, AlloyDB, Cloud SQL, Spanner, Looker, SecOps and others named for the following months — with access managed through Cloud IAM, audit logging, and Model Armor positioned against indirect prompt injection. New third-party servers ship continuously; one example from the same week as this post is Notified opening an MCP beta over its PR and IR platforms on 21 October 2026.
What actually happens
This is the only surface on the list where the consumer side is not hypothetical, and the gap is not close. It is also the only one that answers the question the other three dodge: who is calling? A token identifies the client, so you can scope what it may do, meter it, log it, and switch it off — which is the difference between exposing a capability and hoping nothing abuses it. MCP auth covers the current authorisation story and MCP ops in production the operational reality, which is less pleasant than the protocol docs suggest.
The cost is honest work rather than a weekend. You are running an API: versioning, deprecation windows, rate limits, auth, and a tool surface that an agent will use in combinations you did not test. The specific traps are well mapped by now — oversized results blowing a context window (large results and pagination), too many tools diluting selection (MCP tool design), and descriptions that are themselves an injection surface (tool poisoning).
What to do with it
If agents are meant to do anything with your product, this is the one to build, and the others are at best complements. Start with the three to five tools that cover your most-requested integrations, not a mechanical wrapper over your whole REST surface — a hundred thin tools is a worse product than five well-shaped ones.
The axis that actually separates them is authority
Comparing these four on standards status produces a ranking that predicts nothing, because they are not substitutes. Sort them by what the agent is trying to do and the overlap disappears. Reading your content is a public operation with no caller and no liability: the thing that serves it is clean markdown at a stable URL, and llms.txt is a hint about that rather than a way of doing it. Asking a question over your data makes you liable for the answer, which is why it needs an index you keep fresh and a scope you define — NLWeb's shape, or your own search endpoint behind an MCP tool. Acting for a user requires knowing who asked and on whose behalf, and that is a token question. WebMCP sits outside this frame rather than inside the third column: it acts, but the authority it acts with belongs to the human whose session it borrowed.
Two consequences follow for anyone planning this work. First, an agent-readiness project that produces only published files has produced nothing testable; the deliverable that can be verified is a call that succeeds from a real client. Second, if you want to know which agents are reaching you at all, the surface to instrument is your logs and your robots.txt and bot-verification posture, not your llms.txt — bot verification and agent access covers the identification side, which is a prerequisite for deciding anything else.
When to pick which
| Situation | Build this | Skip |
|---|---|---|
| Docs or a knowledge site; you want assistants to cite you accurately | Markdown copies of each page at stable URLs | The llms.txt index, unless a named consumer asks for it |
| A catalogue or dataset people ask questions about | A retrieval index, exposed as an MCP tool; NLWeb if the schema.org data already exists | Waiting for a web standard to settle |
| A product agents should take actions in | A remote MCP server with scoped auth | WebMCP as the primary path |
| A rich in-page UI an agent must drive | WebMCP tools, with server-side validation behind them | Treating a registered tool as trusted |
| Enterprise buyers asking for an audit trail | A remote MCP server, where identity and logging exist | Anything whose caller you cannot name |
And one framing to keep: the asymmetry here is not about quality. llms.txt, NLWeb and WebMCP are all reasonable designs by people thinking clearly about a real problem. The asymmetry is that MCP's consumer side arrived first, in volume, and that is the only thing that has ever made a web-facing format worth publishing. Watch the consumer side, not the specification.
FAQ
Should I publish an llms.txt file?
Only if a consumer you care about has asked for it. The measured evidence is that almost nothing fetches it — 97% of published files saw zero fetches in a 137,000-domain server-log analysis published in June 2026 — and Google's Search guidance says Search does not use these files. Serving clean markdown copies of your pages is the part of the proposal that pays off.
Does llms.txt help with AI search visibility?
No published study has found a meaningful correlation between having the file and being cited in AI answers, and Google says Search does not consume it. Chrome's Lighthouse does audit for it in its Agentic Browsing category, but that audit returns Not Applicable when the file is missing rather than failing you.
Is WebMCP a W3C standard?
No. It is a Draft Community Group Report of the W3C Web Machine Learning Community Group, and the spec page states it is neither a W3C Standard nor on the W3C Standards Track. Chrome shipped an early preview behind a flag, reported as landing in version 146 in February 2026.
Is NLWeb dead?
No, but it is waiting. The code is MIT-licensed and maintained, Cloudflare added support through its AI Search product in early 2026, and the early-adopter list has not grown much since Build 2025. The repository still describes its implementation as proof-of-concept, so treat it as an early bet rather than infrastructure.
Can a remote MCP server replace all three?
For asking and acting, largely yes — it is the surface with real clients and a caller identity. It does not replace serving readable content, since a crawler is not going to open an MCP session to read your documentation, and it does not replace WebMCP where the state an agent needs lives in the rendered page.
What is the single highest-value thing to build first?
If agents should act in your product, a small remote MCP server with three to five well-shaped tools and scoped auth. If they should only read it, markdown at stable URLs. Everything else on this list is a complement to one of those two.
Further reading
On this wiki:
- agents.json and OpenAPI for agents — the adjacent proposal this piece leaves out, and why.
- WebMCP makes your page an API — the authority problem in detail.
- MCP auth — how a caller gets an identity.
- MCP tool design — why five tools beat a hundred.
- MCP ops in production — what running one actually costs.
- Bot verification and agent access — knowing which agents reach you.
- The M×N interop problem — why a consumer has to move first.