llms.txt, measured: the data says it is not doing what you think

The follow-on measurement is the one that settles it: as of May 2026, having an llms.txt file does not measurably improve the odds of being cited by ChatGPT, Claude, Gemini or Perplexity in their search and answer surfaces.

A file that 408 requests out of 500 million asked for is not influencing what an answer engine decides to cite. It cannot be — almost nothing is reading it.

This should not be surprising. Answer engines resolve a query by searching, fetching the results and reading them. That path starts at a page, not at a domain root. There is no step in it where anyone would go looking for a site-level index file, and nothing in any of these products is documented as doing so.

The proposal borrowed its credibility from robots.txt, which every crawler does read. But robots.txt is read for a reason that does not transfer: it is a permission check, and a crawler must consult it before acting. llms.txt is an optimisation, and nothing must consult it at all.

Optional things do not get adopted by being good ideas. They get adopted by a large consumer deciding to require them, and no large consumer has.

There is also a straightforward incentive problem that the proposal never addressed. A file in which a site describes its own most important pages is a file with an obvious motive to exaggerate. Any consumer that started trusting it at scale would immediately need to stop.

Here is the part the sceptical takes get wrong. There is a population of software that reads llms.txt, and it is not the answer engines.

It is coding agents. Cursor, Cline, Continue, Aider and their relatives increasingly look for llms.txt when you point them at a documentation site, because their problem is exactly the one the file solves: a model with a limited context window trying to work out which twelve of your four hundred pages are the ones it needs.

That is a smaller audience than "everyone using ChatGPT" and a much better one if you sell developer tools. A developer whose agent read your docs properly is a developer whose first integration attempt worked.

The honest case for llms.txt is not SEO. It is developer experience for people whose editor reads your docs before they do.

It costs an hour. The real objection to it was never cost; it was that people were being sold it as a ranking factor, and when it does not move rankings they conclude the whole idea is noise.

If citation is the goal, the evidence points at a duller and more familiar list.

1. 01Be in the search index. Every one of these products resolves queries by searching first. Traditional SEO is the precondition, not an alternative to it.
2. 02Answer the question in the first paragraph. Extractive systems lift passages. A page that buries its answer under 600 words of preamble is a page nothing can quote.
3. 03Structure the page so a passage can stand alone. Real headings, short paragraphs, a table where a table belongs. The same properties that make a page chunk well make it quotable — which is not a coincidence, because they are the same operation.
4. 04Publish something only you can. Original measurements, real constants, numbers from a running system. Answer engines cite the source of a fact, and the tenth rewrite of a fact is not its source.
5. 05Serve markdown to clients that ask for it. Returning a markdown representation to a request carrying Accept: text/markdown is cheaper for both sides, and there is early evidence the citation uplift from exposing markdown variants is consistent enough to treat as a standards play rather than an experiment.

Number 5 is the interesting one, and it is llms.txt's idea done at the right level. The insight was correct — models want clean markdown, not your HTML — but a site-level index file is the wrong place to deliver it. Per-page content negotiation is a mechanism that already exists, that every HTTP client understands, and that requires no new standard to adopt.

Keep it small. The failure mode of these files is the sitemap disease — dumping every URL, which reproduces exactly the problem the file was meant to solve.

# Example Corp

> Crawling and retrieval infrastructure. This file lists the pages
> worth reading first; the full index is at /sitemap.xml.

## Docs

- [Quickstart](https://example.com/docs): a working call in five minutes
- [API reference](https://example.com/docs/api): every route, with parameters
- [Authentication](https://example.com/docs/auth): API keys, scopes, rotation

## Concepts

- [How crawling works](https://example.com/docs/crawling): depth, limits, exclusions
- [Rate limits](https://example.com/docs/limits): the actual numbers

## Optional

- [Changelog](https://example.com/changelog)

Twenty links, not two thousand. Descriptions that say what is in the page rather than what it is called. An ## Optional section, which the spec defines for exactly this: material to skip when context is tight.

Then treat it as documentation rather than as marketing, and do not expect it to show up in your analytics — because on current numbers, it will not.

The related question of who is allowed to fetch your pages at all changed considerably this year: Cloudflare's AI-crawler wall. And if you are on the other side of this — building the thing that reads other people's docs — start with web scraping for AI in 2026.