01The gap nobody mentions in the demo
Every web-scraping-for-AI demo ends at the same place. You call an API, a page comes back as clean markdown, and the post says "now just feed it to your LLM."
That last sentence is doing an enormous amount of work. Between "clean markdown" and "a system that answers questions" sits a chunking strategy, an embedding model, a vector index, a retrieval policy, a re-crawl schedule, a deduplication story and a way to tell when the answer is wrong. That is not a weekend. That is the project.
Scraping is the part of a retrieval system that is easy to demo and the smallest part of the work. Everything expensive happens after the markdown comes back.
This guide walks the whole path, stage by stage — what each one is for, what actually goes wrong in it, and what it costs to own. It is the map; the posts it links to are the territory.
It is written by people who run this pipeline, so it is not neutral. Where a claim is about our own system it is stated as such, with the constant or the endpoint, so you can check it rather than take it.
02Three kinds of tool, and which half of the work they leave you
The market looks crowded until you notice it splits cleanly into three groups by where they stop.
| Category | Examples | They give you | You still build |
|---|---|---|---|
| Scraping APIs | Firecrawl, Apify, Bright Data, Zyte, ScrapingBee, Crawl4AI | Clean markdown or JSON from a URL, at scale, past most anti-bot | Chunking, embeddings, a vector store, retrieval, freshness |
| RAG-as-a-service | Ragie, Bedrock Knowledge Bases, Azure AI Search, Vertex AI Search | Ingestion, embedding, a managed index, a retrieval endpoint | The crawler, the document pipeline, the change detection |
| End-to-end | SarvCrawl | Both halves behind one key — crawl and parse, then chunk, embed and search | The application |
The scraping group is the largest and the most mature. If your problem is "this site fights back" — residential proxies, fingerprinting, CAPTCHA — that is a specialised business and Bright Data, Zyte and Oxylabs are very good at it. Crawl4AI, at 78,000-plus GitHub stars, owns the self-hosted end of the same space.
The RAG group solves the opposite problem. You already have the documents; you want retrieval without operating an index. The cautionary note is that this is a harder business than it looks — Vectara, one of the better-known names, has shut down its self-serve tier.
The thing to decide early is not which vendor. It is which of the two seams you want to own, because the seam is where the work lives: the handoff between "I have markdown" and "I have retrievable chunks" is exactly where freshness, deduplication and chunk quality all have to be solved, and neither group solves it for you.
03Stage one: fetching, and the three ways it goes wrong
Fetching a page is trivial. Fetching a hundred thousand pages, from sites that would rather you did not, without getting the wrong thing, is not.
The page is JavaScript
A large fraction of the web renders client-side. curl on a React documentation site returns an empty shell — sometimes literally a few hundred bytes of <div id="root">. Any crawler worth using runs a headless browser pool for these, which is where most of the compute cost in a crawl actually goes. It is also why self-hosting a crawler is heavier than people expect: a browser pool is the memory-hungry part.
The site is blocking you — and, since September 2026, by default
This changed materially this year. Cloudflare now ships managed robots.txt and default blocking for mixed-use AI crawlers on ad-supported pages, plus a pay-per-crawl licensing path. Across their network the 403 rate for AI crawlers roughly doubled year on year, from 5.67% to 9.64%.
The practical consequence is that "can I fetch this" is now a question with a commercial answer as well as a technical one. We wrote that up separately in what Cloudflare's AI-crawler wall changed for your pipeline.
You fetched too much
The quiet one. Point a crawler at a docs site with no bounds and it will happily pull the changelog archive, every tag page, every locale, and the printable version of everything — and you will pay to parse, chunk and embed all of it. Depth limits, page limits and path exclusions are not a nicety, they are the cost control. The cheapest fix is to look before you leap: a URL-discovery pass that returns the link graph without fetching content lets you decide what is worth crawling for effectively nothing. That is the argument of crawl, scrape or map?.
04Stage two: parsing, where markdown pays for itself
HTML is a terrible container for model input. It is mostly navigation, script tags, cookie banners and styling hooks, and a language model pays for every token of it.
The commonly cited measurement is a single blog post at 16,180 tokens as HTML and 3,150 as markdown — better than five to one. At a hundred thousand pages that ratio is the difference between a viable product and a finance conversation.
Markdown is not a nicer format for humans. It is a five-times-cheaper format for models, and that is why every serious tool in this market converts to it by default.
Markdown also survives chunking in a way HTML does not: its headings are a real document outline, so a chunker can split on structure instead of on character counts. That matters more than the token saving. The how-to is in scraping a website to clean markdown.
Documents are not pages
Most real corpora are half web and half files — PDFs, DOCX, XLSX, PPTX, CSV, and scans. A pipeline that handles only HTML handles about half of your problem.
PDFs split into two populations that need different treatment. One has a text layer and parses deterministically. The other is a photograph of a document and needs OCR, which means the output is a reconstruction and its quality varies by page. Conflating the two is how you end up with a knowledge base that is quietly 8% hallucinated table cells. Measuring that gap is its own discipline — see scoring your own PDF conversion.
05Stage three: chunking, the stage that decides your ceiling
Chunking is where most retrieval quality is won or lost, and it gets the least attention because it looks like a text-splitting problem.
It is not. Split a 40-row pricing table down the middle and neither half is answerable. Split a code block and you have produced two fragments that are syntactically meaningless. Split a document into 200-character pieces and every chunk loses the context that made it mean anything — research on chunking for retrieval keeps finding the same failure, that over-fragmentation destroys discriminative power because the evidence for an answer ends up spread across units that are individually unconvincing.
The defaults worth starting from: split on the document's own headings, keep tables and fenced code atomic, use a budget of a few hundred tokens with a small overlap, and count tokens with the same encoder your model uses rather than counting characters and hoping. Ours is 800 tokens with 100 of overlap, counted with tiktoken cl100k_base, and — deliberately — with no LLM in the loop. The reasoning is in chunking markdown without an LLM.
06Stage four: retrieval, where one index is not enough
The default architecture — embed everything, do nearest-neighbour search — fails on a specific and very common class of query: literals. Version numbers, error codes, identifiers, exact phrases. A dense vector has no notion that those exact characters were the point.
Production retrieval runs two legs, a lexical one and a vector one, and fuses them. Reciprocal Rank Fusion is the standard way: throw away both scores, keep only the ranks, and add reciprocals. On the WANDS benchmark, plain RRF scores 0.7068 NDCG against 0.6983 for BM25 alone and 0.6953 for pure KNN; a tuned hybrid reaches 0.7497.
The fusion is the easy part. The decisions that actually move quality are the ones around it — how big a candidate pool to fetch, whether to weight the legs by what the query looks like, what junk to filter before fusing, how many chunks from one page to allow in the top ten, and whether the system is able to say it does not know. All of those, with the constants, are in hybrid search in production.
A nearest-neighbour search always returns neighbours. Ask an irrelevant question and it returns the ten least-irrelevant chunks it has, with nothing in the response to say so.
07Stage five: freshness, the one everyone skips
A knowledge base built once is a knowledge base that is wrong within a quarter. This is the stage that is missing from almost every tutorial, and it is the one that decides whether the thing is still trusted a year later.
A nightly full re-crawl is the obvious answer and the wrong one: it is expensive, it is rude to the site, and it rewrites your whole index to change forty pages. Incremental is the pattern — hash the extracted text of each page, compare against the previous run, and only re-chunk and re-embed what actually changed. That turns a nightly job from O(corpus) to O(changed), which is usually two orders of magnitude smaller.
Deduplication belongs at three levels, and each catches what the last missed: URL normalisation before you fetch (the cheapest, because it prevents the request), content hashing on extracted text rather than raw HTML after you fetch, and near-duplicate filtering for pages with heavy textual overlap. Both are worked through in building a RAG knowledge base from a documentation site.
08Build, assemble or buy
There is no universal answer, but the decision is more legible than it looks once you count the pieces rather than the vendors.
- Build it all
- Right when crawling is your product, or when the corpus is small, static and yours already. You are taking on a browser pool, a document parser, an OCR path, a queue, a vector index and a retrieval policy. Every one of those is tractable; there are six of them.
- Assemble from parts
- The common path: a scraping API plus a vector database plus your own glue. Maximum flexibility, and the glue is permanent — the seam between parsing and chunking is where freshness and dedup live, and it is now yours to maintain.
- Buy end-to-end
- Right when retrieval over web and document content is a feature of your product rather than the product. One integration, one meter, and the trade is that the pipeline's opinions become yours.
The honest version of the self-hosting question: it is genuinely available and genuinely more work than a compose file suggests. Our own stack is 23 services and wants 60 GB of disk before it starts; Firecrawl's self-hosted path asks for 8–12 GB of RAM and does not include the anti-bot capability of its cloud. "Open source" and "cheap to operate" are different properties and the second one is rarer.
An honest comparison of the managed options, with real prices, is in 8 best web scraping APIs in 2026.
09What the whole thing looks like as one call
For concreteness, here is the end-to-end path in our own API: create a knowledge base, crawl into it, poll until the pipeline has finished parsing, chunking and embedding, then search it. Six lines of shell.
Ingestion is asynchronous because crawling is; search is synchronous because waiting on a job id to ask a question would be absurd. That asymmetry is the one thing to internalise about any API in this space.
The same six steps are available from the CLI, from a Node or Python SDK, from an n8n node, and from 21 MCP tools — which means an agent in Claude Code or Cursor can drive the whole pipeline without any glue code, because each tool description tells the model when to reach for it and to poll job status before searching.
10Questions that come up before anyone builds this
Do I need a vector database?
You need a vector index, which is not the same thing. Elasticsearch, Postgres with pgvector and OpenSearch all index dense vectors alongside a lexical index, and having both in one engine is what makes hybrid search one round trip instead of two systems to keep in sync. A dedicated vector database is worth it when vectors are the only thing you are searching.
How big should a chunk be?
A few hundred tokens, split on the document structure rather than a character count, with a small overlap. We use 800 tokens with 100 of overlap. The number matters far less than the rule that tables and code blocks are never split, and that the split points are the document own headings.
Is scraping a public website legal?
Publicly accessible content is generally fetchable, but that is a starting point rather than an answer: terms of service, copyright, personal data law and — since 2026 — explicit licensing regimes like pay-per-crawl all apply. Honour robots.txt, identify your crawler, rate-limit yourself, and get advice for anything commercial. Technical capability is not permission.
How often should I re-crawl?
Match it to how fast the source actually changes, and make it incremental. Hash each page extracted text, compare against the last run, and only re-chunk and re-embed what moved. A nightly full re-crawl costs the same whether forty pages changed or none did.
What happens to a scanned PDF?
It goes through OCR, and the output is a reconstruction rather than an extraction. Treat its quality as a number you measure per document rather than a property you assume. Scoring OCR output against the original text layer does not work either, because a scan has no text layer — it has to be judged on its own.
Can I run all of this myself?
Yes, and it is heavier than the compose file makes it look. Our stack is 23 services and asks for 60 GB of free disk, with five dependencies checked at boot: PostgreSQL, Redis, RabbitMQ, MinIO and Elasticsearch. That is normal for this class of system rather than unusual.
11Where to go next
The nine posts this one links to are each a stage of the pipeline done properly:
- Fetching — crawl, scrape or map? and Cloudflare's AI-crawler wall
- Parsing — scrape a website to clean markdown and scoring your PDF conversion
- Chunking — chunking markdown without an LLM
- Retrieval — hybrid search in production
- Freshness — build a RAG knowledge base from a documentation site
- Choosing — 8 best web scraping APIs in 2026
- The agent web — llms.txt, measured
Or skip the reading: the API reference has every route with a live playground, and an account starts with a billion tokens and no card.