What LLM-ready content is
A web page as served is mostly not content. The HTML carries menus, cookie banners, footers, inline scripts, tracking tags and layout markup, and the paragraph you want is a small part of it. Give that to a model and you pay tokens for the noise, and the noise competes with the answer for attention.
LLM-ready content is the same page after cleanup: the main text, with its headings, lists, tables and code blocks intact, in a format a model reads well. In practice that format is markdown, because it keeps structure in a few characters and models have seen a great deal of it.
What makes content LLM-ready
| Property | Why it matters |
|---|---|
| Main content only | Navigation and footers repeat on every page and pollute search results. |
| Structure preserved | Headings, lists and tables tell the model what belongs together. |
| Plain text format | Markdown costs far fewer tokens than the equivalent HTML. |
| Source URL and title kept | Without them an answer cannot cite where it came from. |
| Documents converted too | PDFs and Office files hold much of the useful material on many sites. |
| Chunked on structure | Retrieval works on passages, and a passage cut mid-table is useless. |
The first two are what main content extraction and HTML to markdown conversion do. The last is document chunking, which matters once the content goes into a retrieval system rather than straight into a prompt.
Raw HTML versus LLM-ready markdown
The markdown keeps everything a reader needs, the heading and the emphasis included, and none of what a browser needs.
Common mistakes
- Stripping all tags to plain text. Headings and tables collapse into one run of words, and the model loses which value belongs to which row.
- Dropping the source URL. Content without provenance cannot be cited or refreshed.
- Ignoring JavaScript-rendered pages. A plain HTTP fetch of a single-page app returns an empty shell. See JavaScript rendering.
- Skipping linked documents. The specification is often a PDF linked from the page, not the page itself.
- Chunking by character count. Fixed-size cuts split code blocks and tables. Split on structure first.
How SarvCrawl handles it
Every scrape and crawl renders the page in headless Chromium, falling back to a plain HTTP fetch, and always runs main-content extraction: nav, footer, aside, script and style are removed. Output formats are markdown, html, links and json, where json is the page object with its metadata and HTTP status code. Linked documents found during a crawl, and files you upload, go through the document pipeline, with scanned PDFs sent to OCR.
The markdown is then chunked with structure kept whole (prose, lists, tables and code fences) at 800 tokens with 100 tokens of overlap, embedded and indexed into a knowledge base. For a single page, the Chrome extension copies the open tab as markdown to paste into ChatGPT or Claude. The free HTML to markdown and PDF to markdown tools show the conversion on your own input.
Frequently asked questions
Why is markdown preferred over HTML for LLMs?
It keeps headings, lists, tables and emphasis with a few characters of syntax, so it costs far fewer tokens than the same content as HTML, and it contains none of the layout and script markup that is irrelevant to a reader.
Is plain text LLM-ready?
Partly. It removes the noise but also the structure, so tables and headings become an undifferentiated block. Markdown keeps that structure at almost the same cost.
Do I need to chunk LLM-ready content?
Only if it goes into a retrieval system. A single page pasted into a prompt can be used whole; a site of thousands of pages needs chunking so search can return the relevant passage.
Does LLM-ready content include PDFs?
It should. Documents linked from a site often hold the most specific information, and they need their own conversion, including OCR for scanned pages.