What Is Polite Crawling? Crawl Delay, Rate Limits and robots.txt

Polite crawling in more depth

A crawler can issue requests far faster than a person browsing. Pointed at a small site with no restraint, it can slow the site for real visitors or knock it over. Politeness is the set of habits that prevents that, and it is also practical: impolite crawlers get rate-limited, blocked or served HTTP 429 responses, and finish with less data.

The usual elements are: honour robots.txt; keep a small number of connections per host; leave a gap between requests to the same host; slow down or stop on 429 and 503 responses, respecting any Retry-After header; and identify the crawler with a clear user agent.

Crawl delay

Crawl delay is the pause a crawler waits between requests to the same host. It can be set by the crawler operator, or requested by the site with a Crawl-delay line in robots.txt.

User-agent: *
Crawl-delay: 10
Disallow: /search

Choosing a pace

Small or unknown sites: one request at a time with a pause of a second or more is a safe default.
Large sites behind a CDN: can usually take more, but adaptive throttling, slowing as response times rise, is kinder than a fixed high rate.
Any 429 or 503: back off, ideally exponentially, and honour Retry-After.
Off-peak timing helps for sites whose traffic follows one time zone.

Common pitfalls

Parallel jobs on one host. Ten polite crawlers at once are not polite.
Retrying instantly on errors. Hammering a struggling server makes the outage worse.
Assuming robots.txt rules cover pace. Allow and Disallow say where, not how fast.
Treating politeness as optional on big sites. Large sites are the most likely to detect and block aggressive crawlers.

How SarvCrawl handles it

Crawls respect robots.txt rules. The robots.txt Crawl-delay directive is not honoured, so if a site asks for one, set the pace yourself: with crawl_entire, the delay parameter (0 to 60 seconds) makes the crawl fetch one page at a time with that pause between pages. delay has no effect on crawls without crawl_entire. Each user can run one crawl at a time, and starting another while one is active returns a 429 CONCURRENCY_LIMIT error.

curl -s -X POST https://crawl.sarv.com/api/jobs/crawl \
  -H "x-api-key: $KB_API_KEY" -H "Content-Type: application/json" \
  -d '{"kb_id":"kb_abc123","url":"https://small-site.example",
       "crawl_entire":true,"delay":5}'

Frequently asked questions

What is a good crawl delay?

There is no universal figure. One request at a time with a pause of a second or more suits most small sites; if the site publishes a Crawl-delay, use at least that.

Does Google follow Crawl-delay?

No. Google ignores the Crawl-delay line and adjusts its crawl rate automatically based on how the server responds.

Is Crawl-delay part of the robots.txt standard?

No. RFC 9309 defines User-agent, Allow and Disallow. Crawl-delay is a widely seen extension that each crawler interprets in its own way, if at all.

What should a crawler do on a 429 response?

Slow down. Wait for the time given in Retry-After if present, otherwise back off with increasing pauses before trying again.

What is polite crawling?