What Is Crawl Scope? Domain, Path and Depth Rules

Crawl scope in more depth

A page on almost any site links out to social networks, analytics vendors, partner sites and the rest of the web. Without scope rules a crawler would follow them all. Scope is the filter applied before a URL enters the frontier: does this link belong to the crawl I asked for?

Scope usually combines a where rule (which hosts and paths) with a how much rule (depth and page limits). The where rule keeps the crawl on target; the how-much rule keeps its cost bounded even inside the target.

Common scope rules

Rule	What it keeps	Typical use
Same host	`example.com` only	A single site
Subdomains	`docs.`, `blog.`, `help.` as well	A company with several properties
Path prefix	URLs under `/docs/`	One section of a large site
Exclusions	Everything except `/blog/`, `/tag/`	Skipping low-value areas
Depth and limit	Pages within N hops, up to M pages	Bounding cost

Redirects are a scope question too. A link inside the scope can redirect to a host outside it, so a careful crawler checks scope again after following a redirect.

Example: a docs crawl across subdomains

curl -s -X POST https://crawl.sarv.com/api/jobs/crawl \
  -H "x-api-key: $KB_API_KEY" -H "Content-Type: application/json" \
  -d '{"kb_id":"kb_abc123","url":"https://example.com/",
       "allow_subdomains":true,"max_depth":3,"limit":1000,
       "exclude_paths":["/blog","/careers"]}'

This follows links on example.com and its subdomains, skips the blog and careers sections, and stops at three hops or a thousand pages.

Common pitfalls

`www` versus bare domain. If the site links to both, a strict host match may treat half the site as external.
Content on another subdomain. Docs on docs.example.com are missed by a same-host crawl of example.com.
Path prefixes that are too tight. Seeding at /docs/v2/ misses shared pages under /docs/ that every version links to.
Scope without limits. A tight domain rule does not stop a faceted shop from generating millions of in-scope URLs.

How SarvCrawl handles it

A crawl stays on the seed’s domain by default and only follows URLs under the seed’s path. allow_subdomains (default false) widens it to subdomains, and external links are never followed. exclude_paths removes sections; there is no include-paths option on crawl, so to narrow to one section, seed inside it. max_depth (default 5, up to 20) and limit (default 500, up to 2000) bound the crawl, and crawl_entire lifts both, within your plan limits. If a page redirects outside the scope, it is rejected with an error explaining why.

Frequently asked questions

Does a crawler follow external links?

Most site crawlers do not by default. They may record external links for reporting but only fetch URLs within the configured scope.

Are subdomains in scope by default?

It varies by tool. Many treat each subdomain as a separate site unless you enable subdomains explicitly, because blogs and help centres often run on different platforms.

What is the difference between crawl scope and crawl depth?

Scope decides which URLs are eligible; depth is one of the rules within it, limiting how many link hops from the seed a crawl may go.

How do I crawl only one section of a site?

Seed inside the section and use a crawler that restricts to the seed’s path, or exclude the sections you do not want.

What is crawl scope?