Use case · Training data

Clean web corpora for fine-tuning and evals

Crawl sites and parse documents into clean markdown with source URLs and metadata, then export the whole knowledge base as JSONL or CSV for fine-tuning and evaluation sets.

Start free CLI docs

~500
tokens per page, on average
2
export formats: JSONL and CSV
1B
free tokens on sign-up, no card

// For ML engineers building fine-tuning, eval and retrieval datasets

FROM SITE TO JSONL

Terminal · sarv-kbCLI

$ sarv-kb kb create "support-corpus"
✓ Created kb_8f2c1a
$ sarv-kb crawl https://docs.example.com --kb kb_8f2c1a \
    --limit 2000 --max-depth 4 --exclude-paths /blog,/tag \
    -f markdown,json --wait
✓ Crawl completed · 1,912 pages
$ sarv-kb upload manuals/installation-guide.pdf --kb kb_8f2c1a --wait
✓ Upload completed · 1 document
$ sarv-kb export --kb kb_8f2c1a --format jsonl -o corpus.jsonl
✓ Exported 1,913 pages → corpus.jsonl
$ head -c 180 corpus.jsonl
{"markdown":"# Installing the agent\n\nThe agent runs on Linux…",
 "metadata":{"title":"Installing the agent","sourceURL":"https://docs.example.com/install"…

Perfect for

Domain fine-tuning sets

Turn a vertical’s documentation, manuals and articles into one clean corpus instead of a folder of HTML dumps.

Evaluation and QA sets

Build eval sets from pages with stable source URLs, so every expected answer traces back to a page.

Document-heavy corpora

Mix crawled pages with PDFs, Office files, EPUBs and scans; OCR handles the scanned ones.

Repeatable dataset builds

Script the crawl and export with the CLI or SDK so a dataset can be rebuilt from scratch.

How it works

01 / 06

Find the sources

Map a domain to list its URLs before you spend tokens, or run a web search to collect sources on a topic.

Map Search

02 / 06

Crawl with limits

Set max depth, page limits and excluded paths so the corpus holds content pages, not tag archives and logins.

Crawl

03 / 06

Parse documents

Upload PDF, Office, EPUB, image and audio files; they come back as markdown alongside the web pages.

Parse

04 / 06

Check conversion quality

Check conversion audit scores for parsed documents and read the stored markdown in the console before it goes into a dataset.

Try the playground

05 / 06

Export JSONL or CSV

Stream a whole knowledge base or a single job out as JSONL or CSV from the console, CLI or API.

CLI

06 / 06

Automate the pipeline

Drive the same steps from the Node or Python SDK and rebuild the dataset whenever the sources change.

API reference

How SarvCrawl compares

Feature	SarvCrawl	Raw HTML dumps	Hand-written scrapers	Public web corpora
Clean markdown, boilerplate removed
Source URL kept on every page
JavaScript-rendered pages
PDFs and scans in the same corpus
You choose the exact sources
One-command JSONL or CSV export

Build the corpus, not the scraper

Crawl, parse and export clean markdown datasets. 1 billion free tokens, no card.

Start free CLI docs