What is near-duplicate detection?

Deduplication works best at three levels, each catching what the last cannot:

1. 01URL normalisation, before fetching: strip tracking parameters and fragments, unify trailing slashes and case, respect rel=canonical. Cheapest, because it prevents the request.
2. 02Exact content hashing, after fetching: SHA-256 of the extracted text, not the raw HTML, which differs on every render.
3. 03Near-duplicate detection: compare sets of overlapping word sequences.

The usual technique is shingling: split text into overlapping runs of k words and compare the sets with Jaccard similarity. Comparing every pair is too slow at scale, so MinHash compresses each set into a short signature whose agreement estimates Jaccard, and locality-sensitive hashing buckets likely matches. SimHash is an alternative that produces a single fingerprint where similar texts differ in few bits.

def shingles(text, k=5):
    words = text.lower().split()
    return {" ".join(words[i:i + k]) for i in range(len(words) - k + 1)}

def jaccard(a, b):
    return len(a & b) / len(a | b) if a | b else 1.0

if jaccard(shingles(page_a), shingles(page_b)) > 0.9:
    print("near-duplicate")

• Hashing HTML. CSRF tokens, build hashes and footer years change on every render, so no two fetches match. Hash the extracted text.
• Comparing whole pages including boilerplate. Shared navigation and footers make unrelated pages look similar. Extract main content first.
• Threshold too low. Legitimately different pages from one template, say two product pages, can be merged by mistake. Inspect matches before deleting.
• Deduplicating versions blindly. /v2/ and /v3/ may be near-identical yet differ in exactly the line that matters. Usually the right move is to exclude old versions at crawl time.

URL normalisation saves money. Near-duplicate detection saves your search results.

In SarvCrawl the first line of defence is the crawl itself: exclude_paths keeps version archives, locale prefixes, tag pages and changelogs out of a knowledge base, and a map of the site first shows which directories are worth crawling. Pages are stored as clean markdown rather than raw HTML, so render-to-render noise such as tokens and build hashes does not make one page look like many.

At query time, hybrid search caps how many chunks any one page can contribute to a result, two for five or fewer results and three otherwise, so near-identical chunks cannot fill the top of the list. The full three-level approach is in build a RAG knowledge base from a documentation site.