What Is Crawl Budget? How It Works and How to Use It

Crawl budget in more depth

The term comes from search engines. Google describes crawl budget as the combination of a crawl capacity limit, how many parallel connections and requests a site can take without slowing down, and crawl demand, how much the engine wants to crawl the site based on popularity and how often it changes. Only large or fast-changing sites usually need to think about it.

The same idea applies when you run a crawler yourself. Every crawl has a budget, whether it is a page limit, a time limit, a token or credit cost, or the politeness you owe the target server. Spending it on duplicates and junk means the pages you wanted may never be reached.

What spends crawl budget

Duplicate URLs: Tracking parameters, session ids, trailing slashes and mixed case make one page look like many. URL normalization fixes most of it.
Faceted navigation: Filter and sort combinations on listing pages can generate millions of URLs with near-identical content.
Soft 404s and redirects: Pages that return 200 with “not found” content, and long redirect chains, each cost a fetch.
Slow responses: When a server slows down or returns HTTP 429, polite crawlers back off and fewer pages fit in the same window.
Low-value sections: Tag archives, print views and login pages rarely carry content worth indexing.

Example: spending a budget well

A practical order of operations for your own crawls:

List the site’s URLs first with a sitemap or a map request, without fetching content, and see what is actually there.
Exclude sections you do not need, such as /tag/, /search or /login.
Ignore query parameters unless the site genuinely uses them for distinct content.
Set a page limit a little above the count you expect, so a trap cannot run away with the crawl.

curl -s -X POST https://crawl.sarv.com/api/jobs/map \
  -H "x-api-key: $KB_API_KEY" -H "Content-Type: application/json" \
  -d '{"kb_id":"kb_abc123","url":"https://shop.example.com","limit":5000}'

Common pitfalls

Treating robots.txt as an index control. Blocking a URL stops crawling it but does not remove it from search results; use noindex for that.
Optimising budget on a small site. A few thousand pages rarely hit search engine limits; content quality and internal linking matter more.
No page limit. One crawler trap can consume an entire run.

How SarvCrawl handles it

A crawl’s budget is set with limit and max_depth, and exclude_paths keeps it out of sections you do not want. Crawls run with ignoreQueryParameters, so URL variants that differ only in their query string are fetched once. The map endpoint lists URLs from sitemaps and links without fetching content, which lets you size a crawl first, and the console reports redundant and external link counts for every crawl so waste is visible. To keep content fresh without recrawling everything, the monitor endpoint re-checks chosen pages on a schedule and sends a webhook when one changes.

Frequently asked questions

Does crawl budget matter for small sites?

Rarely for search engines. Google says most sites with fewer than a few thousand URLs are crawled efficiently. It always matters for your own crawls if you pay per page or token.

Does blocking pages in robots.txt save crawl budget?

It stops compliant crawlers from fetching those URLs, so yes, but it does not remove already-indexed pages from search results.

How do I find what is wasting crawl budget?

Look at the URLs a crawl actually fetched. Clusters of query-string variants, filter combinations and redirects are the usual culprits.

What is crawl budget?