What Is Incremental Crawling? Keeping Crawls Fresh

Incremental crawling in more depth

A full recrawl is the simplest way to stay current and the most wasteful: on most sites the bulk of pages have not changed since last time. Incremental crawling treats the previous crawl as state. It remembers each URL, when it was fetched and what it contained, and on the next pass spends effort only where something is likely to differ.

For search indexes and knowledge bases this matters twice: fewer fetches, and fewer chunks to re-embed. Stale content is a quiet failure, because answers built from last month’s docs look just as confident as correct ones.

How change detection works

Sitemap lastmod: If the site keeps <lastmod> accurate in its sitemap, compare dates and fetch only pages newer than your copy. Cheap, but only as good as the site’s dates.
Conditional requests: Send If-None-Match with the stored ETag or If-Modified-Since with the stored date. The server answers 304 Not Modified with no body if nothing changed.
Content hashing: Fetch the page, hash the extracted main content and compare with the stored hash. Works everywhere, and ignores changes to ads or timestamps if you hash the cleaned text.
Discovery pass: Re-list URLs from sitemaps and links to find pages that are new or gone.

Scheduling then decides how often to check each page. Pages that changed recently tend to change again, so many crawlers revisit them more often and back off on pages that never change.

Example: a conditional fetch

import hashlib, requests

def refresh(url, stored):
    headers = {"If-None-Match": stored["etag"]} if stored.get("etag") else {}
    r = requests.get(url, headers=headers, timeout=30)
    if r.status_code == 304:
        return None  # unchanged
    digest = hashlib.sha256(r.text.encode()).hexdigest()
    if digest == stored.get("hash"):
        return None  # same content, new response
    return {"etag": r.headers.get("ETag"), "hash": digest, "html": r.text}

Common pitfalls

Hashing raw HTML. Rotating ads, CSRF tokens and timestamps change on every request, so every page looks changed. Hash the extracted content.
Missing deletions. Pages that now 404 must be removed from the index, not kept forever.
Trusting lastmod blindly. Some sites set it to the build time on every page.
Re-embedding everything. Only changed chunks need new embeddings.

How SarvCrawl handles it

The monitor endpoint re-checks the pages you choose on a schedule and sends a webhook when one changes, so you can re-scrape just that page into its knowledge base rather than recrawling the site. The map endpoint lists a site’s current URLs from sitemaps and links without fetching content, which is a cheap way to spot new sections worth crawling.

Frequently asked questions

How is incremental crawling different from a full recrawl?

A full recrawl fetches every page again. An incremental crawl uses stored state and change signals to fetch only pages that are new or likely to have changed.

What is a 304 Not Modified response?

The reply to a conditional request when the resource has not changed since the version identified by the ETag or date the client sent. It has no body, so it is cheap for both sides.

How often should pages be recrawled?

It depends on how often they change. Revisit frequently changing pages often and back off on stable ones, adjusting based on what each check finds.

What is incremental crawling?