Sitemap
Every page, in one list
29 public pages across 6 groups — the product, what people build with it, the full API and SDK reference, and the paperwork. Nothing here is behind a login.
Product01
Use cases03
- MCP for AI editors
/use-cases/ai-mcpsLive web data inside Claude, Cursor and any MCP client, through 21 tools. - AI agents and RAG
/use-cases/ai-chatsTurn docs and help centres into knowledge bases an LLM can actually retrieve from. - Lead enrichment
/use-cases/lead-enrichmentProspect sites turned into structured company and contact signals.
Developers07
- API docs
/docsEvery REST route, with cURL, Node and Python samples for each one. - MCP server
/docs/mcp-server21 tools for Claude Code, Cursor and VS Code, and how to point them at a knowledge base. - Node SDK
/docs/node-sdkThe JavaScript and TypeScript client — install, authenticate, scrape, crawl, search. - Python SDK
/docs/python-sdkThe Python client, with the same surface as the Node one. - CLI
/docs/clisarv-kb, the terminal client — ingest, inspect and search without writing code. - n8n node
/docs/n8nCreate KB, Crawl Website, Get All Jobs and AI Answer as n8n workflow nodes. - Changelog
/changelogWhat shipped, when, and what it changed for anyone calling the API.
Writing11
- Blog
/blogGuides to crawling, parsing and retrieval, honest comparisons of the tools in this market, and how our own pipeline works. - Web scraping for AI in 2026: turning any site into a searchable knowledge base
/blog/web-scraping-for-aiThe complete path from a URL to an answer — crawl, parse, chunk, embed, retrieve — what breaks at each stage, and how the three kinds of tool in this market split the work between you and them. - Cloudflare's AI-crawler wall went up. What actually changed for your pipeline.
/blog/cloudflare-ai-crawler-wallFrom 15 September 2026, mixed-use AI crawlers are blocked by default on ad-supported pages. Here is what the change is, what the block rates actually look like, and what a crawler has to do now to keep working. - Hybrid search in production: fusing BM25 and vector KNN with RRF
/blog/hybrid-search-rrf-in-productionEvery retrieval demo runs one vector index and stops. Production needs two legs and a way to combine them. Here are the constants we actually run — the rank constant, the weights, the floors and the guard that lets the system say it does not know. - How to scrape a website to clean markdown for LLMs
/blog/scrape-a-website-to-markdownHTML costs about five times as many tokens as the same page in markdown, and chunks far worse. Here is why, how to do the conversion four different ways, and the two options that decide whether the output is usable. - 8 best web scraping APIs in 2026, compared honestly
/blog/best-web-scraping-apisEight tools, what each one is actually for, the real entry prices, and the question that decides between them — which half of the work you want to keep. Written by one of the eight, and marked where that matters. - Build a RAG knowledge base from a documentation site
/blog/docs-site-to-rag-knowledge-baseThe tutorial usually stops at the first crawl. The hard parts are the ones after it: deduplicating what you fetched, re-crawling only what changed, and noticing when the source moved without you. - Chunking markdown without an LLM: tables, code fences and an 800-token budget
/blog/chunking-markdown-without-an-llmChunking is where retrieval quality is decided, and the popular answer — ask a model to do it — is slow, expensive and non-deterministic. Here is a router that reads the document shape instead, and the three rules that matter more than the token budget. - Scoring your own PDF conversion: ROUGE, Jaccard, and an OCR path that does not lie
/blog/scoring-pdf-conversion-qualityEvery document parser claims high accuracy and none of them show their working. Here is how to measure a conversion against the original, why that method breaks completely on scanned pages, and what to do instead. - Crawl, scrape or map? Picking the right call — and not paying for pages you do not need
/blog/crawl-vs-scrape-vs-mapThree verbs that sound interchangeable and are not. One fetches a page, one follows links, one only discovers URLs for effectively nothing — and using the third first is the cheapest decision in an ingestion pipeline. - llms.txt, measured: the data says it is not doing what you think
/blog/llms-txt-measuredOut of 500 million AI bot visits in a 90-day window, 408 requested llms.txt. That is the whole story on citations — and it is still worth shipping one, for a completely different reason.
Company04
Legal03
- Terms of service
/terms-of-serviceThe agreement between you and Sarv Webs Pvt. Ltd. for the use of SarvCrawl — what we run, what you are responsible for putting into it, and what each of us owes the other when something goes wrong. - Privacy policy
/privacy-policyWhat SarvCrawl collects, why, who else ever sees it, how long we keep it, and how to make us delete it. Written to be read rather than skimmed past. - Delete your account
/account-deletionnot indexedClose an account and have its content removed. Public on purpose — someone who wants out may already be unable to sign in.