Any website or file, searchable in one call.
Point it at a site or drop in a PDF. It comes back as clean markdown, indexed, chunked and embedded — ready to query by keyword or by meaning.
- No card to start
- 1 billion tokens included
- Self-host or cloud
Called from Claude Code · Cursor · VS Code · n8n · Node · Python · cURL
new crawl
Same call from the API: POST /api/jobs/crawl
/authentication → stored
Kept in object storage, with the page row and its job lineage in Postgres — and indexed for keyword search.
authentication.md
# Authentication Every request carries an x-api-key header… ## Rotating a key Issue a second key, move traffic, then revoke…
Stored beside the chunk, so a hit can always be traced back to its page.
24
pages
3
documents
412
chunks
412
vectors
“Issue a second key, move traffic to it, then revoke the first. Keys never expire on their own.”
/authentication — Rotating a key · chunk 02 · 0.91
Paste a site, set how deep to go, and start. That is the whole setup.
See what it did
Every answer traces back to a page you can open.
The console is not a wrapper on the API — it is where a person checks the crawl, reads the chunks and finds the bad conversion.
crawl tree · job_8b41d0
/ 200 · 12 links ├─ /auth 200 · security-notes.pdf ├─ /api 200 · 4 links └─ /v1/legacy 301 → /v2
Every job, with the tree it produced
- Status, inputs and per-job worker logs
- Crawls render as a tree, so a dead branch is visible
- Cancel a queued or running job from the row
parsed markdown
# Authentication Every request carries an x-api-key header. Keys never expire on their own.
Read what your model will be given
- Markdown per page, chunks per page
- Headings and code fences respected
- Export a KB or a job as JSONL or CSV
issue a second key, then revoke the first
pass your key in the x-api-key header
key rotation and storage
Both search modes, scoped to one KB
- Keyword and hybrid semantic, same API key
- Filter by job, type, language and date
- Scores come back with the chunk
91/100
48,210 → 46,802 words
613 chunks
A score for the conversion, not a shrug
- ROUGE-1, Jaccard, vocabulary coverage, length fidelity
- Scanned PDFs judged on their OCR output
- Compare shows source and markdown side by side
jobs per day
Keys, usage and roles you can audit
- Per-key usage charts and daily job counts
- Admin and user roles — enforced at the API
- Orgs with per-org usage for the whole estate
For developers
Six surfaces, one API key.
REST, Node, Python, a terminal client, 21 MCP tools and an n8n node. Nothing is gated behind a higher plan.
# crawl a site, then ask it something
curl -X POST $KB/api/jobs/crawl -H "x-api-key: $KEY" \
-d '{"kb_id":"kb_3f9c21","url":"https://docs.example.com","limit":25}'
curl "$KB/api/kb/kb_3f9c21/search/embed?q=rotate%20api%20key" \
-H "x-api-key: $KEY"import { SarvKB } from 'sarvcrawl'
const kb = new SarvKB({ apiKey: process.env.SARV_KB_KEY })
const job = await kb.crawl({ kbId: 'kb_3f9c21', url: 'https://docs.example.com', limit: 25 })
await kb.waitFor(job.jobId)
const hits = await kb.semanticSearch('kb_3f9c21', 'rotate api key')from sarvcrawl import SarvKB
kb = SarvKB(api_key=os.environ["SARV_KB_KEY"])
job = kb.crawl(kb_id="kb_3f9c21", url="https://docs.example.com", limit=25)
kb.wait_for(job.job_id)
hits = kb.semantic_search("kb_3f9c21", "rotate api key")$ sarv-kb kb create "Docs" $ sarv-kb crawl https://docs.example.com --kb kb_3f9c21 --limit 50 --wait $ sarv-kb find "rotate api key" --kb kb_3f9c21 $ sarv-kb export --kb kb_3f9c21 --format jsonl -o kb.jsonl
{
"mcpServers": {
"sarv-kb": {
"command": "npx",
"args": ["sarvcrawl-mcp"],
"env": { "KB_BASE_URL": "https://crawl.sarv.com", "KB_API_KEY": "sk_live_..." }
}
}
}
# then: "crawl docs.example.com into my Docs KB and tell me how to rotate a key"SarvCrawl → Resource: Job · Operation: Crawl Website
→ URL: {{ $json.domain }} · Max Depth: 2 · Page Limit: 25
SarvCrawl → Resource: Search · Operation: AI Answer
→ Query: {{ $json.question }}Asked before every integration
Six things people check first.
Scrape, crawl or map — which one?
Scrape takes one URL. Crawl follows links from a root URL, bounded by depth, a page limit and a path exclude list, and returns a tree of what it found. Map only discovers URLs — no content, no page cost — which is the cheap way to decide what is worth crawling.
What happens to a PDF, and to a scanned one?
It is parsed to markdown, stored beside the original, then indexed, chunked and embedded like any page. With no text layer — a scan or a photo — it goes through OCR instead, and the conversion audit scores the OCR output on its own rather than against a text layer that does not exist.
Do you train models on our content?
No. There is no training or fine-tuning step anywhere in the pipeline. It is retrieval: crawl, parse, chunk, embed, search. The chunks stay in your knowledge base, are readable in the console, and export to JSONL or CSV whenever you want them out.
Can an AI client drive it directly?
Yes — 21 zod-validated MCP tools over stdio or HTTP streamable at /v2/mcp. The tool descriptions tell the model to poll job status until a job completes before it searches, so an agent runs the async pipeline without glue code.
How many tokens do I get for free?
One billion on sign-up, with no card. Tokens are what parsing, chunking and embedding consume, so at roughly 500 tokens to a page of text that is on the order of two million pages through the full pipeline before you owe anything.
Can we run it ourselves?
Yes. The stack ships as a Compose file for a single host and a Swarm bundle for a cluster, behind one nginx front door, with five dependencies checked at boot: PostgreSQL, Redis, RabbitMQ, MinIO and Elasticsearch.
More in the API docs.
Point it at a site.Ask it something.
- 1 billion tokens free
- No card to start
- Exports anytime