New1 billion tokens free on sign-up — crawl, parse, chunk and embed on the houseSee pricing
SarvCrawlSarvCrawl
1 billion tokens free on sign-up

The web crawling API that gives your agents a search engine.

Point SarvCrawl at a URL or drop in a PDF. It comes back crawled, parsed to clean markdown, indexed, chunked and embedded — ready to query by keyword or by meaning.

https://
One page to clean markdown · no sign-up
  • No card to start
  • 1 billion tokens included
  • Self-host or cloud

Called from Claude Code · Cursor · VS Code · n8n · Node · Python · cURL

From URL to answer

One request, six steps, all of them yours to inspect.

Crawled, stored in every format, chunked, embedded — then answered, with the page it came from.

01Give it a URL

new crawl

https://docs|
depth 2limit 25 pagesinclude linked documents
Start crawl

Same call from the API: POST /api/jobs/crawl

https://docs.example.comcrawling
/200
markdownfetching18 KB
linksfetching12 found
/authentication200
markdownfetching18 KB
screenshotfetching240 KB
security-notes.pdffetchingOCR
/v1/legacy301
redirectfetching→ /v2

/authentication → stored

authentication.mdmarkdown18 KB
authentication.htmlraw html61 KB
links.json37 links2 KB
page.jsonstructured4 KB
screenshot.pngimages240 KB
security-notes.pdfparsed + OCR1.2 MB

Kept in object storage, with the page row and its job lineage in Postgres — and indexed for keyword search.

authentication.md

# Authentication
Every request carries an x-api-key header…

## Rotating a key
Issue a second key, move traffic, then revoke…
markdown_router · split on headings
chunk 01Authentication180 tokens
chunk 02Rotating a key164 tokens
chunk 03Rate limits212 tokens
embedding chunks103 / 412
chunk 01[ 0.021, −0.118, 0.334, … ]
chunk 02[ −0.067, 0.245, 0.019, … ]
chunk 03[ 0.132, 0.008, −0.201, … ]

Stored beside the chunk, so a hit can always be traced back to its page.

Your data is ready to query

24

pages

3

documents

412

chunks

412

vectors

how do I rotate a key without downtime?hybrid

“Issue a second key, move traffic to it, then revoke the first. Keys never expire on their own.”

/authentication — Rotating a key · chunk 02 · 0.91

Paste a site, set how deep to go, and start. That is the whole setup.

How it works

Crawl it, read it, search it.

One pipeline from a URL to an answer, behind one API key.

[ 01 / 04 ]·The pipeline

A complete web crawling API, not just a scraper

Most tools stop at pulling raw HTML. SarvCrawl crawls the site, parses every page and file into clean markdown, then indexes, chunks and embeds it.

  1. step 01

    Crawl

    Follow links from a root URL with a depth, a page limit and path rules, and get back a tree of what it found.

  2. step 02

    Parse

    HTML pages, PDFs and office files convert to clean markdown; scans with no text layer go through OCR.

  3. step 03

    Search

    Query by keyword or by meaning with hybrid search, scoped to one knowledge base, on a paid plan.

One API that crawls and then searches what it found, instead of a scraper stitched to a separate search index.

[ 02 / 04 ]·For agents

Built for AI agents and JavaScript-heavy sites

Pages are rendered in a real browser before they are read, so JavaScript apps that break a static fetch still come back as clean markdown an LLM can use directly.

  • MCP21 MCP tools let an agent run the crawl-then-search pipeline itself, with no glue code.
  • ClientsClaude CodeCursorVS Coden8nNodePythoncURL
  • FormatsMarkdown, HTML, links and screenshots, plus each page’s metadata as JSON.
[ 03 / 04 ]·Plans

Free to start, self-hosted when you need it

1B

1 billion tokens free

on sign-up, no card. At roughly 500 tokens a page, that is on the order of two million pages crawled, parsed, chunked and embedded.

  • Search on paid plansKeyword and semantic search over everything you have ingested, with no separate search service to run.
  • Self-host on EnterpriseThe full stack behind one nginx front door — a Compose file for one host, a Swarm bundle for a cluster — with your data in your own Postgres and object storage.
[ 04 / 04 ]·First ten pages

Getting started

Point SarvCrawl at a small site first (try limit: 10) and read back the markdown and chunks it produced in the console. Each page carries a conversion score, so you can see what changed on the way to markdown. When you move to a paid plan, the same knowledge base is searchable by keyword or by meaning.

crawl.sh
curl -X POST $KB/api/jobs/crawl \
  -H "x-api-key: $KEY" \
  -d '{"kb_id":"kb_3f9c21","url":"https://docs.example.com","limit":10}'

We do not train models on your content. The pipeline is retrieval only — crawl, parse, chunk, embed, search — and your chunks export as JSONL or CSV whenever you want them out.

Real output

See what a page turns into.

Public-domain files run through the production pipeline. The markdown is exactly what the API returned.

Input · Scanned PDFTypewritten 1955 memorandum from President Eisenhower to the Secretary of Defense
Output · markdown
MEMORANDUM FOR

THE SECRETARY OF DEFENSE:

Subject: Automotive and Related Procurement.

This refers to your March fourteenth informal report to me on progress in achieving an effective spread of governmental procurement in the automotive and related fields, so as to include continuing programs of work for the smaller, as well as the larger, companies. As you know, I am interested in this for several reasons:

(a) I intend to do all in my power to see that no action of the government creates a tendency toward a monopoly;

(b) I want to help in the preservation of a mobilization base dispersed among different companies and business leaders, as well as geographically; and

(c) I want to see smaller companies on a comparative footing with others in the conviction that competition between big and little, as well as between the giants themselves, will certainly result to the benefit of our entire economy and of our people.

I am today leaving the city for a week. During my absence, will you please have prepared another brief progress report on what results have been achieved by the Defense Department in spreading this type of work, whether such spread is accomplished on a direct or on a sub-contracting basis.

I do not desire that a staff consume a lot of time in preparing a report in unnecessary detail. I want to know the certainty and magnitude of progress and to have the assurance that this matter is constantly engaging the attention of responsible officials. If such a report could be sent to my office by the twenty-first, that would be early enough.

Typewritten memo, 1955 · unedited API output

Open full example

All examples — with the conversion audit for each.

See what it did

Every answer traces back to a page you can open.

The console is not a wrapper on the API — it is where a person checks the crawl, reads the chunks and finds the bad conversion.

SarvCrawlJobs · Docs KB5 of 128
crawldocs.example.com24completed
uploadsecurity-notes.pdf61completed
monitorchangelog page—watching
crawlhelp.example.com9running
mapexample.com318completed

crawl tree · job_8b41d0

/             200  ·  12 links
├─ /auth      200  ·  security-notes.pdf
├─ /api       200  ·  4 links
└─ /v1/legacy 301  →  /v2

Every job, with the tree it produced

  • Status, inputs and per-job worker logs
  • Crawls render as a tree, so a dead branch is visible
  • Cancel a queued or running job from the row

For developers

Six surfaces, one API key.

REST, Node, Python, a terminal client, 21 MCP tools and an n8n node, all on the same API key.

shell
# crawl a site, then ask it something
curl -X POST $KB/api/jobs/crawl -H "x-api-key: $KEY" \
  -d '{"kb_id":"kb_3f9c21","url":"https://docs.example.com","limit":25}'

curl "$KB/api/kb/kb_3f9c21/search/embed?q=rotate%20api%20key" \
  -H "x-api-key: $KEY"

Asked before every integration

8 things people check first.

What is a web crawling API, and how is SarvCrawl different?

A web crawling API fetches web content programmatically. SarvCrawl also parses what it fetches to markdown, then chunks, embeds and indexes it, so the site you crawled becomes a knowledge base you can search by keyword or by meaning on a paid plan.

Does it handle JavaScript-rendered pages?

Yes. Pages are rendered in a headless browser before they are converted, so client-side apps come back as markdown rather than an empty shell. PDFs and office files are parsed too, and scans with no text layer go through OCR.

Scrape, crawl or map — which one?

Scrape takes one URL. Crawl follows links from a root URL, bounded by depth, a page limit and a path exclude list, and returns a tree of what it found. Map only discovers URLs — no content, no page cost — which is the cheap way to decide what is worth crawling.

What happens to a PDF, and to a scanned one?

It is parsed to markdown, stored beside the original, then indexed, chunked and embedded like any page. With no text layer — a scan or a photo — it goes through OCR instead, and the conversion audit scores the OCR output on its own rather than against a text layer that does not exist.

Do you train models on our content?

No. There is no training or fine-tuning step anywhere in the pipeline. It is retrieval: crawl, parse, chunk, embed, search. The chunks stay in your knowledge base, are readable in the console, and export to JSONL or CSV whenever you want them out.

Can an AI client drive it directly?

Yes — 21 zod-validated MCP tools over stdio or HTTP streamable at /v2/mcp. The tool descriptions tell the model to poll job status until a job completes before it searches, so an agent runs the async pipeline without glue code.

How many tokens do I get for free?

One billion on sign-up, with no card. Tokens are what parsing, chunking and embedding consume, so at roughly 500 tokens to a page of text that is on the order of two million pages through the full pipeline before you owe anything. Searching what you ingested needs a paid plan.

Can we run it ourselves?

On Enterprise, yes. The stack ships as a Compose file for a single host and a Swarm bundle for a cluster, behind one nginx front door, with five dependencies checked at boot: PostgreSQL, Redis, RabbitMQ, MinIO and Elasticsearch.

More in the API docs.

SarvCrawl

Websites are noisy.Your agent deserves the signal.

  • 1 billion tokens free
  • No card to start
  • Exports anytime