What Is a Seed URL? Choosing Where a Crawl Starts

Seed URLs in more depth

Every crawl needs somewhere to begin. The seed is placed in the crawler’s frontier at depth 0, fetched, and parsed for links. Those links become depth 1, their links depth 2, and so on until a depth or page limit stops the crawl.

Search engines run with millions of seeds and keep adding more. A focused crawl of one site usually has a single seed, which makes the choice matter: the seed decides what the crawler can reach, how deep each page sits, and often what counts as in scope.

What the seed decides

Depth. Crawl depth is counted in link hops from the seed, so the same page can be depth 1 from one seed and depth 4 from another.
Scope. Many crawlers use the seed’s host, and sometimes its path, to decide which links to follow. See crawl scope.
Reachability. A crawler only finds pages linked, directly or indirectly, from the seed. Orphan pages need a sitemap or a second seed.
Robots rules. The seed’s host determines which robots.txt file applies.

Example: choosing a seed

Goal	Good seed	Why
Whole marketing site	`https://example.com/`	The homepage links to most sections.
Only the docs	`https://example.com/docs/`	Starts inside the section and keeps depth shallow.
One API version	`https://example.com/docs/v2/`	Avoids spending the page limit on older versions.

curl -s -X POST https://crawl.sarv.com/api/jobs/crawl \
  -H "x-api-key: $KB_API_KEY" -H "Content-Type: application/json" \
  -d '{"kb_id":"kb_abc123","url":"https://example.com/docs/",
       "max_depth":3,"limit":500}'

Common pitfalls

Seeding at a leaf page. An article that only links back to the homepage gives the crawler little to work with.
Seeding a URL that redirects. If http://example.com redirects to https://www.example.com/en/, the crawl may start somewhere other than you expected. Use the final address.
Seeding a login or consent page. The crawler sees the wall, not the content behind it.
Assuming the seed reaches everything. Pages with no inbound links are invisible to link-following. Check a sitemap or a map of the site.

How SarvCrawl handles it

The seed is the required url field of a crawl job. max_depth (default 5, range 1 to 20) counts link hops from it, and only URLs on the seed’s domain and under the seed’s path are crawled, so seeding at /docs/ keeps the crawl inside /docs/. Sitemaps are read automatically to discover URLs as well as links. After the crawl, the console shows a link tree rooted at the seed, so you can see which page brought each URL into the crawl.

Frequently asked questions

Can a crawl have more than one seed URL?

Large crawlers often start from many seeds. For a single site, one well-chosen seed plus the site’s sitemap is usually enough; separate sections can be crawled as separate jobs.

Should the seed be the homepage?

Only if you want the whole site. If you need one section, seed inside it so the page limit is spent on the pages you want.

Does a trailing slash on the seed matter?

It can. Some sites serve different content or redirect between `/docs` and `/docs/`, and path-based scope rules may treat them differently. Use the form the site itself links to.

Is a seed URL the same as a start URL?

Yes. Seed, start URL and root URL are used interchangeably for the address a crawl begins from.

What is a seed URL?