The URL frontier in more depth
A crawler is a loop: take a URL from the frontier, fetch it, extract its links, add the new ones to the frontier. The frontier starts with the seed URL and grows as pages are parsed. The crawl ends when the frontier is empty or a limit is reached.
Two decisions live in the frontier. Ordering decides which URL is fetched next: first-in first-out gives breadth-first traversal, while priority queues let important or fresh pages jump ahead. Admission decides whether a URL enters at all: is it in scope, and has it been seen before?
How a frontier avoids duplicates
Without a visited set, a crawler would loop forever on sites where pages link to each other. Before adding a URL, the frontier checks whether it has already been queued or fetched. That check is only as good as the comparison: https://example.com/a, https://www.example.com/a/ and http://example.com/a?utm_source=x are often the same page.
So frontiers compare URLs after URL normalization: lowercasing the host, dropping fragments, handling trailing slashes, and often stripping tracking parameters. Large crawlers also split the frontier per host so they can space out requests to each server, which is the basis of polite crawling.
Example: one step of the loop
- Frontier holds
/, depth 0. Fetch it; it links to/docs,/pricingand/docs#install. - Normalise:
/docs#installbecomes/docs, already queued, so it is dropped. - Queue
/docsand/pricingat depth 1 and mark/as visited. - Take the next URL from the frontier and repeat.
Common pitfalls
- Weak normalisation. Session ids and tracking parameters make one page look like thousands and flood the frontier.
- Crawler traps. Calendars, infinite filters and relative-link loops generate endless new URLs. A page limit is the backstop.
- Unbounded growth. On large sites the frontier can outgrow memory; production crawlers keep it in a database or queue service.
- Ignoring order. A depth-first frontier with a page limit can spend the whole budget down one branch.
How SarvCrawl handles it
Each crawl keeps its own visited-URL set. www and scheme variants are collapsed, query parameters are ignored, and a second check on host and path, with query, fragment and trailing slash dropped, catches remaining duplicates. Pending pages sit in a priority job queue ordered by priority and then creation time, which in practice behaves roughly breadth-first. You can follow progress with GET /api/jobs/:id/logs and see the resulting link structure with GET /api/jobs/:id/tree.
Frequently asked questions
Is the URL frontier the same as a crawl queue?
Broadly, yes. Frontier is the term used in crawler research; queue is the everyday word. A frontier usually also includes the logic for ordering and de-duplicating URLs.
What happens when the frontier is empty?
The crawl is complete: every reachable, in-scope URL has been fetched. Most crawls stop earlier because they hit a depth or page limit.
Why does a crawler fetch the same page twice?
Usually because two URL variants were not recognised as the same page, for example with and without a trailing slash or with different query parameters.
How do search engines prioritise their frontier?
They weigh signals such as a page’s importance, how often it changes and how much load the host can take. The exact formulas are not published.