Crawl

Crawl a whole website into a searchable knowledge base

Start from one URL. SarvCrawl follows the links, renders every page, parses the PDFs it finds, and turns the lot into markdown that is chunked, embedded and ready to search.

Scrape Crawl Map Search Parse Monitor Crawl tree

20 pages
Crawled live in the demo above, free and without an account
Tree
Every crawl is drawn as a link tree, so a dead branch is obvious
1B
Free tokens on sign-up, on the order of two million pages

Built for whole sites, not single pages

Follows links for you

Breadth-first from the start URL, within the depth and page limits you set, each URL fetched once.

Path rules

exclude_paths keeps admin screens, tag archives and other noise out. Query strings are ignored so one page is one page.

Documents along the way

With include_docs, linked PDFs and office files are parsed into the same knowledge base, with OCR for scanned ones.

A crawl tree

Each page sits under the first page that linked to it, with counts for duplicate and external links. Try it with the free website tree tool.

Search what you crawled

Keyword and semantic search over every page on paid plans, through the API, the console or MCP.

Export any time

Download a crawl or a whole knowledge base as JSONL or CSV, chunks included.

How a crawl runs

01 / 03
Point it at a site
POST /api/jobs/crawl with a start URL, max_depth, limit and any paths to skip.
02 / 03
Pages stream in
Pages are rendered and converted as they are discovered; the console shows progress and the tree as it grows.
03 / 03
Search and export
When the job completes, everything is chunked and embedded, ready for search or export.

Crawl from your code

A crawl is a job. Poll it, or watch it in the console; its pages are readable while it runs.

curl -X POST https://crawl.sarv.com/api/jobs/crawl \
  -H "x-api-key: YOUR_API_KEY" -H "Content-Type: application/json" \
  -d '{"kb_id":"kb_abc123","url":"https://docs.example.com",
       "max_depth":5,"limit":500,"exclude_paths":["/admin"],
       "formats":["markdown"],"include_docs":true}'

Crawl API reference

Questions

What does a crawl API do?

It starts at a URL, discovers the pages a site links to, and fetches each one, so you get a whole site in one request instead of scripting a loop of single-page scrapes.

How do I keep a crawl small?

Set limit (pages) and max_depth (link hops from the start URL), and list paths to skip in exclude_paths. Running a map first shows what is there before you spend tokens.

Does the crawl include PDFs?

Yes with include_docs: linked PDFs and office files are parsed into the same knowledge base, and scanned pages go through OCR.

What does the free crawl above do?

It runs a real crawl of up to 20 pages and shows each one as markdown. Without an account you see the start of every page; signed in you read them in full and can crawl the whole site into a knowledge base from the same box.

Crawl your docs into search today

1 billion tokens free on sign-up, no card. Roughly 500 tokens a page.

Start with 1 billion free tokens Read the API docs