Crawl depth in more depth
Depth describes how far a page sits from the seed in the link graph, not in the URL. A page at /docs/api/v2/auth/tokens can be depth 1 if the homepage links to it directly, and a page at /pricing can be depth 4 if the only route to it is through three other pages.
Because crawlers usually go breadth-first, a page’s depth is the length of the shortest path found to it. If the homepage links to page B and page A also links to B, B is depth 1, not 2.
For SEO, “click depth” is the same idea seen from a visitor’s side: pages many clicks from the homepage tend to be crawled less often and treated as less important.
How max depth limits a crawl
Each URL in the crawler’s frontier carries the depth at which it was found. When a page at depth d is fetched, its links are queued at depth d + 1, and links that would exceed the maximum are dropped.
| Max depth | What gets crawled |
|---|---|
| 0 | Only the start URL. |
| 1 | The start URL and every in-scope page it links to. |
| 2–3 | Typical for documentation and marketing sites; reaches most content reachable from navigation. |
| 5+ | Deep archives and paginated lists; usually needs a page limit too. |
Site navigation flattens depth. A docs site with a full sidebar on every page puts most pages at depth 1 or 2, so a small max depth often covers far more of a site than people expect.
Example: depth and limit together
Depth and page limits solve different problems. Depth stops the crawl wandering too far; a limit caps the total cost. Use both.
Common pitfalls
- Confusing link depth with path depth. A max depth of 2 does not mean “two folders deep”. Use path filters for that.
- Paginated content. Page 40 of a blog archive is depth 40 if each page only links to the next. A sitemap or a map of URLs reaches it directly.
- Depth alone is not a budget. On a site with large menus, depth 2 can still mean thousands of pages. Set a limit.
- Starting too deep. Seeding at a leaf page that does not link back to the section you want returns very little.
How SarvCrawl handles it
Crawl jobs take max_depth alongside limit and exclude_paths. After a crawl, the console shows it as a link tree where each page sits under the first page that linked to it, so you can see exactly which path brought a page into the crawl and at what depth. If a section is missing, the map endpoint lists URLs from sitemaps and links without fetching content, and the free website tree tool shows a site’s structure.
Frequently asked questions
What is a good max crawl depth?
For most documentation and marketing sites, 2 or 3 reaches nearly everything linked from navigation. Pair it with a page limit and adjust after looking at what the first crawl found.
Is the start page depth 0 or depth 1?
Most crawlers treat the start URL as depth 0, so a max depth of 1 means the start page plus the pages it links to. Check the convention of the tool you use.
Does crawl depth matter for SEO?
Pages that take many clicks to reach from the homepage tend to be discovered and recrawled less often. Linking important pages from navigation or hub pages keeps them shallow.