01The shape of the job
Documentation sites are the best possible first corpus. They are structured, they are written to be read, they use headings properly, and — unlike a marketing site — almost every page is worth having.
They also change constantly, which is the part tutorials skip. A knowledge base built from docs in January and never refreshed is a knowledge base confidently answering questions about an API that has since changed twice.
So the job has five parts, and only the first is the one people write about:
- Discover what is there, cheaply, before fetching anything.
- Crawl the part that matters.
- Deduplicate — at three separate levels, because each catches what the last missed.
- Chunk and embed.
- Keep it current incrementally, rather than by re-doing all of the above nightly.
A knowledge base is not a thing you build. It is a thing you keep, and the keeping is most of the cost.
02Step 1: look before you fetch
Map the site first. It returns URLs and no content, which means it costs nothing to run and tells you the two things that determine everything downstream: how big this is, and what the URL shape looks like.
The output of that last line is the thing to look at. On a typical docs site you will find a large directory nobody wants: /blog/, /changelog/, /tag/, a /v1/ archive of the previous major version, and — the classic — every page again under three locale prefixes.
Version archives deserve a specific warning. A docs site that keeps /v1/, /v2/ and /v3/ online has three near-identical copies of every page. Ingest all three and every query retrieves three variants of the same answer, two of which are wrong for the current release, and nothing in the chunk says which is which. Exclude old versions explicitly. This is the single most common way a docs knowledge base ends up worse than useless.
03Step 2: crawl, bounded by what you just learned
include_docs is on because documentation links to PDFs — spec sheets, compliance documents, migration guides — and a knowledge base that has the page describing the spec but not the spec is a knowledge base that will confidently paraphrase a summary.
When the job finishes, look at its tree before trusting it. GET /api/jobs/:id/tree shows what was found from where, and it is where you discover that 400 of your 1,200 pages came from one paginated index.
04Step 3: deduplicate, at three levels
Deduplication is not one thing. It is three, applied at different points, and each catches what the previous one structurally cannot.
Level 1 — URL normalisation, before fetching
The cheapest by a wide margin, because it prevents the request. ?utm_source=, trailing slashes, #anchors, uppercase paths, index.html versus the bare directory, and any <link rel="canonical"> the page declares — all of these produce the same document at different addresses.
On a large docs site this alone typically removes a double-digit percentage of the crawl. It is also the only level that saves you fetching cost rather than storage cost.
Level 2 — content hashing, after fetching
SHA-256 of the extracted text, not the raw HTML. This distinction is the whole trick: two renders of the same page differ in a CSRF token, a build hash in an asset URL, a rotating footer year and a timestamp, so the HTML hashes differ every time and the text hash does not.
Hash the markdown after conversion and you get a stable identity for a page's content, which is both a deduplicator now and the basis of incremental re-crawling later.
Level 3 — near-duplicate detection
The residue: pages that are 95% the same. Two locale variants where only the nav translated. An API reference page generated for each of forty endpoints from one template. A "print version" of every page.
These pass both earlier filters — different URLs, different hashes — and they are the ones that hurt retrieval most, because they cluster tightly in embedding space and crowd out genuine variety in the top-k. A shingle or MinHash comparison over extracted text catches them.
URL normalisation saves money. Near-duplicate detection saves your search results.
There is a fourth line of defence at query time — a per-page diversity cap in the fused results — but that is damage control. Cleaning the corpus is the real fix. See hybrid search in production for what junk in the index actually does to ranking.
05Step 4: chunk and embed
In this pipeline this stage is not yours to run — crawling into a knowledge base parses, chunks and embeds on the way through, and the job is not complete until all of it has happened. If you are assembling from parts instead, this is the seam where you would be wiring a chunker and a vector store together.
Either way, the decisions are the same and they matter more than the retrieval tuning that comes after: split on the document's own headings, keep tables and fenced code atomic, budget a few hundred tokens with a small overlap, and count tokens with the encoder your model uses. Ours is 800 with 100 of overlap, counted with tiktoken cl100k_base. The reasoning is in chunking markdown without an LLM.
Then check it, which takes one query:
The second query is the more informative one. A retrieval system that cannot tell you when it has nothing relevant will hand an agent five chunks of your API docs in response to a baking question, and the agent will use them.
06Step 5: staying current without re-crawling everything
The naive schedule is a nightly full re-crawl. It is wasteful, it is impolite, and it rewrites your entire index to reflect eleven changed pages.
The incremental version uses the content hashes from level 2:
- Re-fetch the page (or check
Last-Modified/ETagfirst, when the server is honest about them). - Convert to markdown and hash the text.
- Unchanged hash — stop. Nothing to re-chunk, nothing to re-embed, nothing to write.
- Changed hash — re-chunk and re-embed that page only, and replace its chunks.
This turns the nightly job from O(corpus) to O(changed), which on a mature docs site is usually two orders of magnitude smaller. It also makes the refresh cheap enough to run daily rather than monthly, which is the actual win: freshness is a function of how cheap the refresh is.
Or let the monitor do it
A scheduled monitor does this as a first-class job type, with a schedule in plain language and a webhook when something moves:
Two design details worth copying if you build your own. A monitor's output is always markdown, with no format options — because a diff needs a single canonical representation, and diffing HTML would report a changed build hash as a content change every night. And a monitor never completes: it sits at running indefinitely by design, so cancelling is how you stop one, and a delete on a running monitor is refused rather than silently doing something.
The unified diff in the payload is the part to actually use. It means your alert can say "the API key page now says keys expire after 90 days" rather than "something changed", which is the difference between a notification somebody acts on and one they filter to a folder.
07The five mistakes, in the order people make them
- Crawling unbounded. No
exclude_paths, nolimit, and a corpus that is 60% tag archive. Map first. - Ingesting every version of the docs. Three copies of every page, two of them wrong, none of them labelled.
- Hashing HTML instead of text. Every page appears to change every night; incremental re-crawling degrades to a full one and nobody notices because it still works.
- Skipping the negative test. Nobody checks what the system does with an unanswerable question until a user does.
- Building once. The most expensive mistake, and the slowest to show up: everything works, and then a year later the answers are quietly about last year's API.
The first two are covered in crawl, scrape or map?. The wider pipeline this sits in is web scraping for AI in 2026.