What is a web crawler?

1. 01Seed. The crawl starts from one or more seed URLs, often a homepage or a sitemap.xml.
2. 02Frontier. URLs waiting to be fetched sit in a queue called the frontier. The order it is drained in, breadth-first or by priority, decides which pages are reached first.
3. 03Fetch. The crawler requests each URL over HTTP. Sites that build their content with JavaScript need JavaScript rendering in a headless browser before the content exists.
4. 04Parse and extract links. The response is parsed and every <a href> is resolved to an absolute URL. See link extraction.
5. 05Filter and deduplicate. Each new link is normalized, checked against the seen set, and kept only if it is in scope: same domain, allowed path, within the crawl depth limit and allowed by robots.txt.
6. 06Repeat until the frontier is empty or a page limit is hit.

Scope is the whole game. A crawler with no page limit, no depth limit and no path filters will happily follow calendar links, faceted search and session-tagged URLs forever.

A minimal crawler in Python shows the moving parts. It keeps to one host, caps the page count and skips URLs it has already seen.

from collections import deque
from urllib.parse import urljoin, urlparse
import re, requests

def crawl(seed, limit=100):
    host = urlparse(seed).netloc
    seen, frontier = {seed}, deque([seed])
    while frontier and len(seen) <= limit:
        url = frontier.popleft()
        html = requests.get(url, timeout=10).text
        for href in re.findall(r'href="([^"#]+)"', html):
            link = urljoin(url, href)
            if urlparse(link).netloc == host and link not in seen:
                seen.add(link)
                frontier.append(link)
    return seen

A production crawler adds politeness delays, robots.txt checks, retries with backoff on HTTP 429, a real HTML parser and rendering for script-heavy pages.

• Crawler traps. Infinite calendars, sort and filter parameters, and relative links that keep nesting deeper produce unlimited unique URLs. Page limits and path exclusions are the defence.
• Duplicate URLs. The same page under ?utm_source= variants, trailing slashes or mixed case wastes the crawl budget.
• Empty pages. A plain HTTP fetch of a single-page app returns a shell with no content.
• Going too fast. Hammering a server gets you rate limited or blocked and is unfair to the site owner.

The crawl endpoint takes a start URL plus max_depth, limit and exclude_paths to keep the crawl in scope, and crawls run with query parameters ignored so tracking variants collapse into one page. Every page is rendered in a real browser and saved as clean markdown into a knowledge base. With include_docs, linked PDFs and Office files are parsed too.

The console draws each crawl as a link tree, with every page placed under the first page that linked to it, plus counts of redundant and external links. To see what a crawl would reach before you run it, the map endpoint lists URLs from sitemaps and links without fetching content.