How to scrape a website to clean markdown for LLMs

01Why markdown, and not just stripped HTML

Feed a language model raw HTML and it pays for the navigation, the cookie banner, four analytics snippets, the footer sitemap, every class="flex items-center" and the script tag that loads the chat widget. The article is somewhere in there, at maybe a fifth of the total.

The commonly cited measurement on a single blog post is 16,180 tokens as HTML against 3,150 as markdown — a bit over five to one. That ratio is stable enough across ordinary content-heavy pages to plan with.

	Raw HTML	Markdown
Tokens, one blog post	16,180	3,150
Cost at 100k pages, indicative	~1.6B tokens	~315M tokens
Useful signal per token	low	high
Splittable on document structure	no	yes

The cost saving is the argument people lead with. The last row is the one that actually matters.

Markdown headings are a real document outline. That is what lets a chunker split on meaning instead of on character counts — and chunking is where retrieval quality is won.

HTML has headings too, in principle. In practice a <div class="text-2xl font-bold"> is a heading to a human and nothing to a parser, and most of the web is built that way. Conversion to markdown is where the guesswork happens; doing it once, well, at ingest, beats every downstream consumer guessing separately.

02Four ways to do it

Same page, four surfaces. Pick whichever fits where you are standing.

curl -X POST https://crawl.sarv.com/api/jobs/scrape \
  -H "content-type: application/json" \
  -H "x-api-key: $KB_API_KEY" \
  -d '{
    "kb_id": "kb_abc123",
    "url": "https://example.com/docs/getting-started",
    "formats": ["markdown", "links"],
    "include_docs": true
  }'

# => { "job_id": "...", "status": "queued" }

There is a fifth surface worth knowing about if you live in an editor: an agent in Claude Code or Cursor can do all of the above through 21 MCP tools, and the tool descriptions tell the model to poll job status before it searches, so it sequences the pipeline without any glue code from you.

03The two options that decide whether the output is usable

`only_main_content`, and when to turn it off

Main-content extraction drops the chrome — nav, sidebar, footer, cookie bar — and keeps the article. It is the right default for almost everything, and it is wrong in two specific cases.

You are crawling a documentation site whose sidebar is the structure. Sometimes the nav is the only place the hierarchy exists, and losing it loses the map.
You are scraping a listing or index page, where the "chrome" is the content — a pricing table, a directory, a search-results page.

The failure mode is quiet: main-content extraction on a page it misreads returns a short, clean, plausible document that is missing the part you wanted. Spot-check the first ten pages of any new crawl by eye. It takes four minutes and it is the highest-value four minutes in the whole exercise.

`formats`, and asking for more than markdown

A scrape can return several representations of the same page: markdown, html, links, json and images. Asking for extras costs storage, not a second fetch, and two of them earn their keep more often than people expect.

links: The page's outbound link graph. This is what lets you decide what to crawl next without crawling it first, and it is how a one-page scrape becomes a bounded crawl.
html: Keep it when you might change your mind about conversion. Re-fetching a page six months later gets you a different page; re-converting stored HTML gets you the same page, converted better.

Keeping the raw HTML alongside the markdown is the cheapest form of insurance in an ingestion pipeline. Conversion heuristics improve; the web does not stand still while you improve them.

04The half of the web that is not HTML

Most real corpora are part web page and part file. include_docs follows linked documents — PDF, DOCX, XLSX, PPTX, CSV, RTF, plain text — and parses them into markdown through the same pipeline, so a spec sheet linked from a product page ends up in the index beside it.

PDFs split into two populations. One has a text layer and converts deterministically. The other is a photograph and needs OCR, where the output is a reconstruction whose quality varies page by page. Treating those as the same thing is how a knowledge base ends up quietly wrong about numbers in tables.

Because the second case cannot be checked against an original text layer — there is not one — it needs measuring on its own terms. That is a whole discipline and it is in scoring your own PDF conversion.

05Checking that the markdown is actually good

Four things to look at before you ingest a hundred thousand pages, in order of how often they are wrong:

Do the headings survive? If the whole document came out as flat paragraphs, the converter did not find a heading structure and every chunker downstream will be splitting blind.
Do the tables survive as tables? Tables are where conversion most often degrades to a run-on sentence, and a table flattened into prose is unanswerable.
Did the code blocks keep their fences? An unfenced code block is indistinguishable from prose to a chunker, which will happily split it in half.
Is the boilerplate gone? Grep a sample for "Accept cookies" and your own footer text. If they are there, main-content extraction missed, and you are about to embed the same 200 tokens onto every page in the corpus.

That last one is worth dwelling on. Boilerplate repeated on every page is not merely wasted tokens — it makes every page look similar to every other page in embedding space, which is a direct hit to retrieval precision. It is a surprisingly common cause of "our vector search returns random pages".

If every page in your corpus ends with the same 200 tokens of footer, every page in your corpus is a little bit similar to every other one. Vector search notices.

When the conversion is right, the next decision is how to cut it up — which is chunking markdown without an LLM. If you are still deciding how much of a site to fetch in the first place, start with crawl, scrape or map?.