Robots.txt in more depth
Every host has at most one robots.txt, served at /robots.txt. A crawler fetches it before crawling and applies the group of rules that matches its user agent. The rules apply per host and protocol, so https://example.com/robots.txt does not govern https://docs.example.com.
Robots.txt controls crawling, not indexing or access. A disallowed URL can still appear in search results if other sites link to it, and anyone can still open it in a browser. Use authentication for private content and a noindex directive for pages that should not be indexed.
How robots.txt rules are matched
User-agentstarts a group. A crawler uses the group with the most specific matching name, or*if none matches.DisallowandAllowtake a path prefix. When several rules match, the longest (most specific) rule wins; on a tie,Allowwins under RFC 9309.*matches any sequence of characters and$anchors the end of the URL.Sitemaplines point to sitemap.xml files and apply to the whole file, not one group.Crawl-delayis not part of the standard. Some crawlers honour it; Google ignores it.
If robots.txt returns 404, crawlers treat the whole site as allowed. If it returns a server error, RFC 9309 says to assume everything is disallowed for a while.
Example robots.txt
Here every crawler is kept out of search results and the cart, except the cart help page; one named AI crawler is blocked entirely; and the sitemap is advertised to all. Many sites now add groups for AI crawlers specifically; see the Cloudflare AI crawler wall for how that has played out.
Common pitfalls
- `Disallow: /` left over from staging blocks the whole site after launch.
- Blocking CSS and JavaScript stops search engines rendering the page properly.
- Using it to hide pages. The file is public, so listing secret paths advertises them.
- Expecting one group to inherit another. A crawler follows only its own group; rules under
*do not apply to a crawler that has a named group. - Forgetting subdomains. Each host needs its own file.
How SarvCrawl handles it
When you crawl a site, exclude_paths lets you keep the crawl out of whole sections, which is the simplest way to mirror a site’s Disallow rules or skip areas you do not need, and limit and max_depth keep the request count bounded. The map endpoint reads the URLs a site lists in its sitemaps along with its links, without fetching page content. Whether your use of a site is permitted is a question for its terms and robots.txt, and that responsibility stays with you.
Frequently asked questions
Is robots.txt legally binding?
It is a voluntary protocol, not a contract or a security control. Whether ignoring it has legal consequences depends on jurisdiction and the site terms, so treat it as the owner’s stated wishes.
Does robots.txt stop a page from being indexed?
No. It stops compliant crawlers fetching the page, but the URL can still be indexed from links elsewhere. Use a noindex meta tag or header, and allow crawling so the crawler can see it.
Where does robots.txt have to be?
At the root of the host, for example https://example.com/robots.txt. A file in a subfolder is ignored.
Can I block AI crawlers with robots.txt?
You can add Disallow rules for the user agents that AI companies publish. Compliant crawlers will honour them; crawlers that ignore the protocol will not.