What Is URL Normalization? Rules, Examples and Pitfalls

URL normalization in more depth

The same page is routinely reachable under many URLs: HTTP://Example.com:80/a/./b/, http://example.com/a/b/, and http://example.com/a/b/?utm_source=newsletter may all return identical content. To a naive crawler these are three pages. Normalization maps them to one key before the crawler checks whether it has seen the URL.

RFC 3986 defines rewrites that never change meaning. Beyond those, crawlers apply heuristic rules that are usually right but can occasionally merge pages that really are different.

Common normalization rules

Rule	Before	After	Safe?
Lowercase scheme and host	`HTTPS://Example.COM/A`	`https://example.com/A`	Yes
Remove default port	`https://example.com:443/`	`https://example.com/`	Yes
Resolve dot segments	`/a/./b/../c`	`/a/c`	Yes
Uppercase percent-escapes	`%7e` → `%7E`		Yes
Drop the fragment	`/page#section`	`/page`	Usually
Strip tracking parameters	`?utm_source=x`	(removed)	Usually
Sort query parameters	`?b=2&a=1`	`?a=1&b=2`	Usually
Trailing slash	`/docs/`	`/docs`	Site-dependent
Drop all query parameters	`?page=2`	(removed)	Site-dependent

The path is case-sensitive in general, so lowercasing it is not safe even though many servers treat it as insensitive.

Example in Python

from urllib.parse import urlsplit, urlunsplit, parse_qsl, urlencode

TRACKING = {"utm_source", "utm_medium", "utm_campaign", "gclid", "fbclid"}

def normalize(url):
    s = urlsplit(url)
    host = s.hostname or ""
    if s.port and s.port not in (80, 443):
        host += f":{s.port}"
    query = sorted((k, v) for k, v in parse_qsl(s.query) if k not in TRACKING)
    return urlunsplit((s.scheme.lower(), host, s.path or "/", urlencode(query), ""))

Pages can also declare their own preferred URL with <link rel="canonical">, which is a strong hint about which variant to keep when several are fetched.

Common pitfalls

Over-normalizing. Dropping every query string breaks sites that use ?id= or ?page= for genuinely distinct pages.
Hash routing. Some single-page apps put the route in the fragment (/#/settings), so dropping it merges distinct views.
Under-normalizing. Missing session ids or tracking parameters lets one page multiply into thousands and blows the crawl budget.
Normalization is not deduplication. Different URLs can still serve the same content; near-duplicate detection catches what URL rules miss.

How SarvCrawl handles it

Crawls run with ignoreQueryParameters, so URLs that differ only in their query string are treated as the same page and fetched once. That removes tracking-parameter duplicates; for sites that put distinct content behind query strings, crawl those pages individually with the scrape endpoint. The console’s link tree reports how many links on each crawl were redundant, which shows how much duplication normalization saved.

Frequently asked questions

Is URL normalization the same as a canonical URL?

Related but different. Normalization is a rule-based rewrite done by the crawler. A canonical URL is the version the site itself declares with a rel=canonical link.

Should a crawler remove trailing slashes?

Only if the site serves the same content with and without them. On some servers /docs and /docs/ are different resources, so check before merging them.

Are URL paths case-sensitive?

Under the URL standard, yes. The scheme and host are case-insensitive, but the path and query may not be, so only lowercase the scheme and host.

What is URL normalization?