01What changed, precisely
As of 15 September 2026, Cloudflare blocks mixed-use AI crawlers by default on ad-supported pages across its network. Site owners do not have to configure anything or write a line of robots.txt: the rules ship as managed policy, and Cloudflare maintains the crawler lists and the heuristics behind them.
Two words in that sentence are doing the work.
- Mixed-use
- A crawler that both trains models and answers live questions. The distinction publishers care about is training versus answering — answering sends traffic back, training does not — and a crawler that does both has, until now, been treated as one thing.
- By default
- The important half. Blocking AI crawlers has been possible for two years; almost nobody did it, because it required a decision. Making it the default inverts who has to act, and defaults are the only setting that moves aggregate behaviour.
Alongside the block sits pay-per-crawl: a mechanism for publishers to charge AI companies for access rather than simply refusing them. That is the more consequential half in the long run, because it turns an allow/deny question into a licensing one.
The web did not close on 15 September. It acquired a price list.
02The numbers, before and after
This was not sudden. The default-block is the visible step in a trend that has been running all year.
| Metric, AI crawlers | July 2025 | July 2026 |
|---|---|---|
| 4xx response rate | 10.15% | 15.38% |
| 403 Forbidden rate — deliberate blocks | 5.67% | 9.64% |
A 403 rate approaching one in ten is not yet an existential problem, but the direction is unambiguous, and the interesting part is the composition. Analyses of robots.txt across Cloudflare's network show publishers converging on a consistent position: block the training bots, allow the answering bots. They are not trying to disappear from AI. They are trying to be cited rather than absorbed.
That is good news if what you are building is retrieval. A pipeline that fetches a page, answers a question from it and links back is on the side of that line publishers are keeping open.
03robots.txt is doing a job it was never designed for
robots.txt is thirty years old. It was built to keep a search indexer out of /cgi-bin/. It is now being asked to referee a question it has no vocabulary for: not may you fetch this, but what may you do with it once you have.
Those are different permissions and the file can only express the first. A publisher who is happy for Perplexity to quote them and unhappy for a training run to ingest them has to express that through a list of user-agent strings, which requires them to know every crawler by name, forever, and update the list as new ones appear. That does not scale, which is precisely why managed lists appeared.
It is also advisory. A crawler that ignores robots.txt is not breaking anything technical — which is why enforcement has moved to the network layer, where a 403 is not a request.
04What a crawler has to do now
None of this is new advice. It is the same advice, now with consequences attached.
- Honour robots.txt, including the AI-specific agents. Not because it is enforced, but because the sites that notice you ignoring it are the ones with the capability to stop you permanently.
- Identify yourself honestly. A named user-agent with a URL explaining what you do is what gets you on an allowlist. A crawler wearing a Chrome string is, by Cloudflare's own classification, an automated bad bot — and that classification is applied by heuristics you cannot see or appeal.
- Rate-limit yourself below what the site would tolerate. Politeness delays cost you hours and buy you years.
- Prefer the sitemap. A sitemap crawl is one request per page you actually want, instead of the full link graph with its tag pages, locale variants and printable duplicates. It is cheaper for you and dramatically cheaper for them.
- Cache aggressively and re-crawl incrementally. Hash what you fetched; on the next run, only re-fetch what plausibly changed. Most of the load a crawler generates is re-fetching pages that are identical to last time.
The last two are worth more than the rest combined, because they are the ones that reduce the traffic you generate rather than merely making it better-mannered. A crawler that re-reads a 40,000-page docs site nightly to find the eleven pages that changed is the reason these defaults exist.
The mechanics of doing that properly — sitemap seeding, three-layer dedup, content hashing between runs — are in building a RAG knowledge base from a documentation site.
05The other half: traffic that is not human any more
The mirror image of the blocking story is that a large and growing share of web traffic is agentic. Sites are tightening defences faster than at any point in the web's history precisely because so much of what arrives is automated.
This puts publishers in an awkward position. The same automation they are blocking is increasingly how people find them. Blocking everything is a way to become invisible to the surfaces that are eating search.
Hence the split: training bots out, answering bots in. And hence the emerging practice of serving agents a different representation — returning markdown to a client that sends Accept: text/markdown, rather than a page of navigation, cookie banners and ad slots. It is cheaper for both sides, and there is early evidence that exposing markdown variants improves citation rates.
The related fashion, llms.txt, is a weaker story than the marketing suggests. We looked at what the adoption data actually shows in llms.txt, measured.
06What to actually do about it
If you run a crawler:
- Check your 403 rate as a time series, not a total. A step change on a date is a policy change; a slow climb is you getting louder.
- Make sure a block is visible as a block. A crawler that treats 403 as "page empty" will quietly fill a knowledge base with nothing and report success — which is worse than failing, because a search over it returns confident answers built from navigation chrome.
- Put the site's own budget above your convenience. Sitemap over link-following, incremental over full, off-peak over immediate.
If you publish a site and are deciding where to sit:
- The default is now blocking, so doing nothing is a choice — and it is the choice to be absent from answer engines.
- "Block training, allow answering" is where most publishers have landed and it is a defensible place to be.
- If you have documentation, the cheapest thing you can do for an agent is serve clean markdown. It costs you nothing and removes the reason for anyone to parse your HTML badly.
And if you are choosing a vendor: ask what happens to a 403 in their pipeline. A crawler that silently records blocked pages as empty ones is a data-quality problem wearing a networking costume.
The wider context — where fetching sits in a retrieval pipeline and what happens downstream of it — is in web scraping for AI in 2026.