Website Change Detection: How Page Monitoring Works

Change detection in more depth

Pages change without notice: prices move, docs gain a section, a policy page is quietly rewritten, an article is taken down. Change detection watches a fixed set of URLs, or a whole site, and tells you when something moved.

The hard part is not fetching; it is deciding what counts as a change. Raw HTML differs on almost every load because of timestamps, ad slots, session tokens and reordered scripts. A useful monitor compares the content a reader would see and ignores the rest.

How change detection works

On each scheduled run, fetch every watched URL, rendering JavaScript where needed.
Reduce the page to its main content, usually as text or markdown, and drop navigation and boilerplate.
Normalise it so cosmetic differences, such as whitespace, do not register as changes.
Compare with the stored copy. If it differs, produce a diff and a short summary of what was added and removed.
Store the new copy and send an alert for anything new, changed, removed or failing.

Monitoring a whole website adds discovery to each run: list the site’s URLs, so pages that appear for the first time are reported as new and pages that disappear as removed.

What it is used for

Competitive intelligence. Pricing, plans and feature pages of other companies.
Keeping a knowledge base fresh. Re-ingest only the pages that changed, rather than recrawling a site. This is incremental crawling driven by alerts.
Compliance and policy. Terms, regulatory notices and published guidance.
News and announcements. Press pages and changelogs without RSS feeds.

Common pitfalls

Diffing raw HTML. Nearly every check looks like a change, and real changes get lost in the noise.
Checking too often. Hourly checks on a page that changes monthly cost the site requests and give you nothing.
Ignoring errors. A page that starts returning 404 or 403 is a change worth knowing about, not a silent skip.
Alerting on every edit. A summary of what was added and removed is far easier to act on than a full page diff.

How SarvCrawl handles it

The monitor endpoint watches either a list of pages (monitor_type: "page", up to 50 urls) or a whole site (monitor_type: "website", one url, a limit of up to 2000 pages and optional include_paths and exclude_paths). schedule is plain text, such as "every 30 minutes", "hourly", "daily at 9:00 am" or "weekly". Comparison is on the page’s markdown, and reordered or whitespace-only changes count as the same. Each URL ends a run as same, new, changed, removed or error; everything except same is POSTed to your notify_url with a unified diff and a plain-English summary of what was added and removed.

curl -s -X POST https://crawl.sarv.com/api/jobs/monitor \
  -H "x-api-key: $KB_API_KEY" -H "Content-Type: application/json" \
  -d '{"kb_id":"kb_abc123","monitor_type":"page",
       "urls":["https://example.com/pricing","https://example.com/terms"],
       "schedule":"daily at 9:00 am",
       "notify_url":"https://hooks.example.com/sarvcrawl"}'
# webhook body: {job_id, kb_id, url, status, diff_summary, diff, checked_at}

Frequently asked questions

How often should I check a page for changes?

Match the schedule to how often the page really changes and how fast you need to know. Daily suits most pricing and policy pages; minutes suit only pages where an hour’s delay matters.

Why does my monitor report changes when nothing visible changed?

It is probably comparing raw HTML, which shifts on every load. Compare the extracted main content instead, and normalise whitespace.

Can change detection find new pages on a site?

Yes, if it monitors the whole site rather than a fixed list. Each run discovers the current URLs, so additions are reported as new and missing pages as removed.

Is change detection the same as incremental crawling?

They are closely related. Change detection tells you what changed; incremental crawling uses that to re-fetch only those pages into an index.

What is website change detection?