AI crawlers in more depth
Search engines have crawled the web for decades in exchange for sending visitors back. AI crawlers break that bargain in different ways depending on their job, which is why the major AI companies now publish several user agents rather than one, so site owners can make separate choices.
Three jobs are worth telling apart. Training crawlers collect pages to train models. Search crawlers build an index that an AI assistant searches when it answers, and usually link to the source. User-triggered fetchers retrieve one page because a person asked the assistant to read it, so they behave more like a browser than a crawler.
Common AI user agents
| Operator | Training | Search | User-triggered |
|---|---|---|---|
| OpenAI | GPTBot | OAI-SearchBot | ChatGPT-User |
| Anthropic | ClaudeBot | Claude-SearchBot | Claude-User |
| Perplexity | — | PerplexityBot | Perplexity-User |
Google-Extended (a robots.txt token, not a separate crawler) | Googlebot | — | |
| Common Crawl | CCBot (an open dataset many models are trained on) | — | — |
Google-Extended is unusual. Pages are still fetched by Googlebot; the token only tells Google whether that content may be used for its Gemini models. Blocking it does not affect Google Search. Apple uses the same pattern with Applebot-Extended.
Blocking AI crawlers with robots.txt
Each agent reads its own group in robots.txt, so you can opt out of training while staying visible in AI search:
Robots.txt is a request, not a wall. Compliant crawlers honour it; others do not, and some operators say their user-triggered fetchers may not apply robots.txt because a person asked for the page. Sites that want enforcement use their CDN or firewall to block by user agent and IP range; see the Cloudflare AI crawler wall for how that has played out.
Common pitfalls
- Blocking everything by accident. A
Disallow: /meant for a training bot placed underUser-agent: *blocks search engines too. - Assuming blocking removes past data. robots.txt affects future fetches; it does not pull pages out of datasets already collected.
- Blocking search agents and expecting citations. If an AI search crawler cannot fetch your pages, the assistant cannot link to them.
- Trusting the user-agent string alone. Anyone can claim to be GPTBot. Operators that publish IP ranges let you verify real traffic.
How SarvCrawl handles it
SarvCrawl is a crawling API you run on your own behalf, not a crawler that builds an index of the web. Crawl jobs respect robots.txt, stay on the start URL’s domain and path, and are bounded by max_depth and limit; delay adds a pause between pages on full-site crawls. Scrapes fetch the one URL you name. Whether your use of a site’s content is permitted is a question for its robots.txt and terms, and that responsibility stays with you.
Frequently asked questions
What is GPTBot?
GPTBot is OpenAI’s crawler for collecting training data. OpenAI publishes separate agents for search (OAI-SearchBot) and for pages a ChatGPT user asks it to open (ChatGPT-User).
Does blocking Google-Extended remove my site from Google Search?
No. Google-Extended only controls whether content Googlebot fetches may be used for Google’s Gemini models. Search crawling and ranking are unaffected.
Can I block AI training but allow AI search?
Yes, for operators that publish separate agents. Disallow the training agents and allow the search agents in robots.txt, as in the example above.
Do all AI crawlers obey robots.txt?
No. The major operators state that their crawlers honour it, but compliance is voluntary, and user-triggered fetchers are sometimes treated differently. Enforcement needs a firewall or CDN rule.