What llms.txt is
llms.txt is a plain markdown file at https://example.com/llms.txt. It names the site, gives a one-paragraph summary, and then lists the pages worth reading first, grouped under headings, each with a short description of what is in the page.
The idea borrows from robots.txt, but inverts it. robots.txt says stay out of here; llms.txt says start here. HTML is a poor container for model input and a site's own navigation is the best summary of what it contains, so handing that summary over directly saves every reader from reconstructing it.
The format
The structure is an H1 with the site name, an optional blockquote summary, then H2 sections of links. A section called ## Optional marks material a reader can skip when its context window is tight.
Twenty links, not two thousand. A file that lists every URL has turned back into a sitemap and recreated the problem it was meant to solve.
Who actually reads it
The popular claim is that publishing llms.txt raises your chance of being cited by ChatGPT, Claude, Gemini or Perplexity. The traffic data does not support it. In one 90-day window of roughly 500 million AI bot visits, 408 requested llms.txt, and as of May 2026 having the file did not measurably improve citation odds. Answer engines start from a search result, which is a page, not a domain root, so there is no step where they would look for a site-level index.
The audience that does read it is coding agents. Tools such as Cursor, Cline, Continue and Aider increasingly check for llms.txt when pointed at a documentation site, because their problem is exactly the one the file solves: a limited context window and four hundred pages, of which twelve matter.
The honest case for llms.txt is not SEO. It is developer experience for people whose editor reads your docs before they do.
The full measurement is in llms.txt, measured.
Common mistakes
- Dumping the sitemap into it. The value is curation. If every page is listed, none is prioritised.
- Descriptions that repeat the title. API reference: the API reference tells a model nothing. Say what a reader will find.
- Expecting it in analytics. On current numbers almost nothing requests it, so do not judge it by traffic.
- Treating it as a substitute for good pages. Agents still fetch the linked pages. Those pages need clear headings and an answer near the top, and ideally a markdown version for clients that ask for one.
How SarvCrawl handles it
SarvCrawl does not depend on llms.txt. A crawl follows a site's links from a start URL, bounded by depth, a page limit and path exclusions, and converts every page it fetches into clean markdown, which is the per-page content an llms.txt file only points at. Those pages are then chunked, embedded and indexed into a knowledge base.
For one-off reading, the Chrome extension copies any open page as markdown to paste into ChatGPT or Claude, and the MCP server gives coding agents in Claude Code, Cursor and VS Code search over the sites you have crawled.
Frequently asked questions
Is llms.txt an official standard?
No. It is a community proposal. It is not an IETF or W3C standard and no standards process is underway, so no crawler or AI product is obliged to read it.
Does llms.txt help my site get cited by AI answer engines?
Current data says no. In a 90-day window of about 500 million AI bot visits, 408 requested llms.txt, and having one did not measurably change citation rates. Being in the search index and answering the question early on the page matter more.
Who benefits from an llms.txt file?
Mainly coding agents and AI editors pointed at documentation sites. They use it to choose which pages to load into a limited context window, which is most useful if you publish developer docs.
How long should llms.txt be?
Short. A few dozen curated links with descriptions of what each page contains, plus an Optional section for material to skip when context is tight.