Search, scrape & crawl

What can SarvCrawl do?

Three core building blocks for working with any web content.

Why SarvCrawl?

Search

Search the web and get full page content from results in one call.

curl -X POST https://crawl.sarv.com/api/jobs/search \
  -H "Content-Type: application/json" \
  -H "x-api-key: YOUR_API_KEY" \
  -d '{
    "kb_id": "kb_abc123",
    "query": "knowledge infrastructure",
    "limit": 3,
    "lang": "en",
    "formats": ["markdown"]
  }'

Scrape

Scrape any URL and get its content in markdown, HTML, or other formats.

curl -X POST https://crawl.sarv.com/api/jobs/scrape \
  -H "Content-Type: application/json" \
  -H "x-api-key: YOUR_API_KEY" \
  -d '{
    "kb_id": "kb_abc123",
    "url": "https://example.com",
    "formats": ["markdown", "html"],
    "include_docs": true
  }'

Crawl

Recursively crawl a website from a root URL and store every discovered page.

curl -X POST https://crawl.sarv.com/api/jobs/crawl \
  -H "Content-Type: application/json" \
  -H "x-api-key: YOUR_API_KEY" \
  -d '{
    "kb_id": "kb_abc123",
    "url": "https://docs.example.com",
    "formats": ["markdown", "json"],
    "max_depth": 5,
    "limit": 500,
    "exclude_paths": ["/admin", "/api"],
    "include_docs": true
  }'

Map

Discover all URLs on a website instantly — without scraping content. Returns a full sitemap-style list of every reachable page.

curl -X POST https://crawl.sarv.com/api/jobs/map \
  -H "Content-Type: application/json" \
  -H "x-api-key: YOUR_API_KEY" \
  -d '{
    "kb_id": "kb_abc123",
    "url": "https://docs.example.com",
    "limit": 1000,
    "include_subdomains": false
  }'

Upload

Upload PDF, DOCX, and other documents directly into your knowledge base. Supports OCR for scanned files.

curl -X POST https://crawl.sarv.com/api/jobs/upload \
  -H "x-api-key: YOUR_API_KEY" \
  -F "kb_id=kb_abc123" \
  -F "file=@./report.pdf"

Monitor

Recurring checks that re-scrape a page (or re-crawl a whole site), diff the new content against the last check, and POST a webhook whenever something is new, changed, or removed. Output is always markdown — there's no formats/scrapeOptions field, so diffing stays well-defined.

curl -X POST https://crawl.sarv.com/api/jobs/monitor \
  -H "Content-Type: application/json" \
  -H "x-api-key: YOUR_API_KEY" \
  -d '{
    "kb_id": "kb_abc123",
    "monitor_type": "page",
    "urls": ["https://example.com/pricing"],
    "schedule": "daily",
    "notify_url": "https://your-endpoint.example.com/webhook"
  }'

Page vs. website

Set monitor_type to "page" to watch one or more exact URLs (pass urls), or "website" to crawl a whole site and watch every discovered page (pass url, plus optional limit, include_paths, exclude_paths).schedule accepts natural language ("daily", "every 30 minutes", "hourly") or a raw cron expression — a minimum interval is enforced server-side.

Webhook payload

Every new, changed, removed, or errored page sends a POST to your notify_url:

Stopping a monitor

A monitor job never "completes" — it sits at status: "running" indefinitely by design. Call cancelJob(jobId) to stop the recurring check (this is what actually stops the schedule upstream); deleteJob(jobId) will refuse a monitor that's still running, so cancel first if you also want to remove it.

Knowledge Base Search

Keyword (full-text) search over the pages already stored in a knowledge base — Elasticsearch BM25 with phrase boosts and highlighting. Returns page-level hits immediately, no job required. Use it after a scrape, crawl, or upload job completes.

curl "https://crawl.sarv.com/api/kb/kb_abc123/search?q=knowledge+infrastructure&size=10" \
  -H "x-api-key: YOUR_API_KEY"

GET /api/kb/:id/search — query params: q (required), page, size (max 50), type (scrape | crawl | search | upload), job_id.

Semantic Search

Meaning-based search over chunk embeddings — a dense vector KNN leg and a BM25 lexical leg fused with Reciprocal Rank Fusion (RRF). Returns chunk-level hits with the surrounding text, ideal for RAG and natural-language questions. Prefer this for conceptual queries; use keyword search for exact lookups.

curl "https://crawl.sarv.com/api/kb/kb_abc123/search/embed?q=how+do+refunds+work&size=10" \
  -H "x-api-key: YOUR_API_KEY"

GET /api/kb/:id/search/embed — query params: q (required), size (max 50), type, job_id. Results are ranked, not paginated, so there's no page param.

SDK Reference

Full SDK reference — every method, every endpoint, with response examples.

MCP Servers

Connect SarvCrawl directly to Claude and any MCP-compatible AI client — 20 tools, zero manual API calls.

n8n Workflows

Automate SarvCrawl visually — drop the community node into any n8n workflow to scrape, crawl, search, and manage knowledge bases with no code.

Command Line

Scrape, crawl, search and manage knowledge bases straight from your terminal — a beautiful CLI styled to match SarvCrawl, with live job spinners and clean tables.