Web Crawler API — Crawl Whole Sites to Markdown

How a crawl works

Map (optional). POST /api/jobs/map lists the URLs a site exposes without fetching their content — the cheap way to decide what is worth crawling.
Crawl. POST /api/jobs/crawl follows links from the root URL, bounded by max_depth, a page limit and an exclude_paths list.
Parse. Each page is rendered and converted to markdown. With include_docs, linked PDFs and Office files are parsed too; scans go through OCR.
Index. Pages are chunked on their own headings and embedded into the knowledge base.
Inspect. GET /api/jobs/:id/tree returns the tree of what the crawl found, and every page is readable in the console or over the API.

Map, then crawl

# Map first: discover URLs without fetching content
curl -s -X POST https://crawl.sarv.com/api/jobs/map \
  -H "x-api-key: $KB_API_KEY" -H "Content-Type: application/json" \
  -d '{"kb_id":"kb_abc123","url":"https://docs.example.com","limit":1000}'

# Then crawl what is worth having
curl -s -X POST https://crawl.sarv.com/api/jobs/crawl \
  -H "x-api-key: $KB_API_KEY" -H "Content-Type: application/json" \
  -d '{"kb_id":"kb_abc123","url":"https://docs.example.com",
       "max_depth":3,"limit":500,"exclude_paths":["/blog"],
       "formats":["markdown"],"include_docs":true}'

Not sure whether you need a crawl, a scrape or a map? Crawl vs scrape vs map walks through the decision.

Keeping a crawl fresh

Sites change. A Monitor job watches pages on a schedule and posts to your webhook when one changes, so a knowledge base built from a docs site does not quietly go stale.

Re-crawling is the same call again. Because everything lives in one knowledge base, the search your app or agent runs always reads the latest version.

What teams crawl

Documentation sites: Turn a product's docs into a RAG knowledge base an assistant can answer from.
Help centres: Ground a support bot in the articles your team already wrote.
Company and prospect sites: Pull structured signals for lead enrichment.
Your own editor: Let Claude or Cursor crawl on demand through the MCP server.

Questions

How do I stop a crawl from wandering off?

Set max_depth for how many links deep it may go, limit for the total number of pages, and exclude_paths for sections you do not want, such as /blog or /admin.

Does the crawler follow PDFs?

Yes, with include_docs set. Linked PDFs and Office documents are parsed to markdown in the same job, and scanned PDFs are OCRed.

Is crawling included on the free plan?

Yes. Crawl, scrape, map, web search, upload and monitor are on every plan, and the free plan starts with 1 billion tokens.

What does a crawl cost?

Crawls bill in tokens, which parsing, chunking and embedding consume. At roughly 500 tokens to a page of text, the free allowance covers on the order of two million pages.

A web crawler API for whole sites, not single pages