Sitemap
Every page, in one list
144 public pages across 10 groups — the product, what people build with it, the full API and SDK reference, and the paperwork. Nothing here is behind a login.
Product12
- SarvCrawl
/What the product is, the six-step pipeline, the console, and the developer surfaces — the whole story on one page. - Pricing
/pricing1 billion tokens free on sign-up, then pay for what the pipeline consumes. No per-seat charge for people who only read results. - Web scraping API
/web-scraping-apiA web scraping API for AI and RAG: send a URL, get clean markdown, HTML, links or JSON, with JavaScript rendering and PDFs. 1 billion tokens free. - Web crawler API
/web-crawler-apiCrawl whole websites to markdown over an API: depth, page limits and exclude paths, linked PDFs parsed, results searchable. 1 billion tokens free. - Open-source web scraping API
/open-source-web-scraping-apiAn open-source alternative to commercial scraping APIs: Apache-2.0 pipeline, AGPL-3.0 engine, self-host with Docker Compose, or use the hosted API. - Chrome extension
/chrome-extensionCopy any page as Markdown for ChatGPT or Claude, or crawl the site into a knowledge base. - Scrape
/scrapePaste a URL and watch it come back as markdown: rendered in a real browser, stripped of navigation and boilerplate, ready for a prompt or a knowledge base. - Crawl
/crawlStart from one URL. SarvCrawl follows the links, renders every page, parses the PDFs it finds, and turns the lot into markdown that is chunked, embedded and ready to search. - Map
/mapMap reads a site’s sitemaps and links and returns the list of URLs it finds, without rendering a single page. It is the cheap first look before a crawl. - Parse
/parseUpload a PDF, Word file or spreadsheet and get structured markdown back. Text PDFs are parsed directly, scans go through OCR, and every conversion gets a quality score. - Search
/searchTwo kinds of search in one API: a web search that scrapes its results into a knowledge base, and keyword or semantic search over everything you have crawled, scraped or uploaded. - Monitor
/monitorWatch a list of pages or a whole site on a schedule. SarvCrawl re-reads them, compares the content, and posts to your webhook whenever something is added, changed, removed or broken.
Compare21
- Comparisons
/compareSarvCrawl vs Firecrawl, Apify, Bright Data, ScraperAPI, Spider, Jina Reader, Browserbase and more: what each scraping tool is for, its price, and when to pick it. - SarvCrawl vs Firecrawl
/firecrawl-alternativeSarvCrawl vs Firecrawl: pricing, licences and self-hosting compared. SarvCrawl forks the Firecrawl engine and adds document parsing, embeddings and search. - SarvCrawl vs Apify
/apify-alternativeSarvCrawl vs Apify: a marketplace of site-specific Actors billed in compute units, or one API that turns any site into searchable markdown. Pricing and fit compared. - SarvCrawl vs Bright Data
/bright-data-alternativeSarvCrawl vs Bright Data: industrial unblocking and proxies, or a crawl-to-search pipeline for public sites and documents. What each is for, and when to use both. - SarvCrawl vs Oxylabs
/oxylabs-alternativeSarvCrawl vs Oxylabs: enterprise proxies and scraper APIs with an account manager, or self-serve crawling into searchable knowledge bases. Fit and pricing compared. - SarvCrawl vs ScrapingBee
/scrapingbee-alternativeSarvCrawl vs ScrapingBee: a simple fetch-and-unblock endpoint, or crawling plus document parsing, chunking and search behind one key. Features and pricing compared. - SarvCrawl vs ScraperAPI
/scraperapi-alternativeSarvCrawl vs ScraperAPI: proxy rotation and anti-bot bypass for hard targets, or crawling into chunked, searchable knowledge bases. Pricing and fit compared. - SarvCrawl vs Zyte
/zyte-alternativeSarvCrawl vs Zyte: the cheapest per-request scraping at volume, or crawling into chunked, embedded, searchable knowledge bases. Pricing and fit compared. - SarvCrawl vs Crawl4AI
/crawl4ai-alternativeSarvCrawl vs Crawl4AI: an open-source Python crawler you host and operate, or a hosted crawl-to-search API you can also self-host. Features, costs and fit compared. - SarvCrawl vs Scrapy
/scrapy-alternativeSarvCrawl vs Scrapy: write and run your own spiders in Python, or call an API that crawls into chunked, searchable knowledge bases. When each is the right tool. - SarvCrawl vs Spider
/spider-alternativeSarvCrawl vs Spider: a fast Rust crawler billed by bandwidth, or crawling into chunked, embedded, searchable knowledge bases. Speed, pricing and licences compared. - SarvCrawl vs Jina Reader
/jina-reader-alternativeSarvCrawl vs Jina Reader: one URL to markdown per request, or whole-site crawls into chunked, embedded, searchable knowledge bases. Pricing and fit compared. - SarvCrawl vs Browserbase
/browserbase-alternativeSarvCrawl vs Browserbase: hosted browsers you script for agents and logins, or an API that crawls sites into searchable knowledge bases. When to use each, or both. - SarvCrawl vs Scrapfly
/scrapfly-alternativeSarvCrawl vs Scrapfly: a credit-based scraping API built around unblocking, or crawling into chunked, searchable knowledge bases. Pricing and fit compared. - SarvCrawl vs Scrape.do
/scrape-do-alternativeSarvCrawl vs Scrape.do: a low-cost rotating-proxy API for fetching hard targets, or whole-site crawling into searchable knowledge bases. When to use each. - SarvCrawl vs ScrapeGraphAI
/scrapegraphai-alternativeSarvCrawl vs ScrapeGraphAI: prompt-driven JSON extraction from pages, or crawling sites and documents into searchable knowledge bases. Pricing compared. - SarvCrawl vs Tavily
/tavily-alternativeSarvCrawl vs Tavily: live web search and extraction for agents, or a knowledge base of whole sites and documents your agent searches. When to use each, or both. - SarvCrawl vs Exa
/exa-alternativeSarvCrawl vs Exa: a search engine with its own web index for agents, or a knowledge base of the sites and documents you choose. Pricing and fit compared. - SarvCrawl vs Hyperbrowser
/hyperbrowser-alternativeSarvCrawl vs Hyperbrowser: cloud browsers and computer-use agents with stealth and proxies, or crawling sites and documents into searchable knowledge bases. - SarvCrawl vs Browser Use
/browser-use-alternativeSarvCrawl vs Browser Use: an open-source agent that drives a browser through tasks, or an API that crawls sites into searchable knowledge bases. Fit compared. - SarvCrawl vs Context7
/context7-alternativeSarvCrawl vs Context7: a shared catalogue of library docs for coding agents, or any docs site crawled into your own knowledge base and served over MCP.
Playground07
- Scrape playground
/playground/scrapeScrape any URL to clean markdown in your browser and see the real output. No sign-up for a preview; the same call is one line of API code. - Crawl playground
/playground/crawlCrawl up to 20 pages of any site live and read each one as markdown. No sign-up for a preview; sign in to crawl whole sites. - Map playground
/playground/mapList every URL on a website from its sitemaps and links, live in your browser. No sign-up for a preview of the first 25 URLs. - Search playground
/playground/searchRun a live web search and see the results SarvCrawl would scrape into a knowledge base. No sign-up for the top three results. - Parse playground
/playground/parseConvert a PDF, Word or Excel file to markdown in your browser. Nothing is stored; no sign-up for a preview of the output. - Monitor playground
/playground/monitorRun one real page check a day for free: pick a schedule, add your own webhook and we POST the result to it. Sign in to keep watching. - Crawl tree playground
/playground/treeCrawl up to 20 pages and draw the site’s link tree, the same way the SarvCrawl console does. No sign-up needed.
Free tools07
- Free tools
/toolsFree online tools from SarvCrawl: PDF and HTML to Markdown, URL extractor, sitemap generator, website tree and logo extractor. No sign-up for a preview. - PDF to Markdown Converter
/tools/pdf-to-markdownConvert a PDF to clean Markdown online, free. Headings, lists and tables kept; Word and Excel files work too. Nothing is stored. Try it now. - HTML to Markdown Converter
/tools/html-to-markdownPaste HTML and get clean Markdown back, free. Boilerplate stripped, headings, tables, links and code blocks kept. The same converter our crawler uses. - Website URL Extractor
/tools/url-extractorExtract every URL on a website from its sitemaps and links, free. See page titles, copy the list or download it as CSV. No crawler to install. - XML Sitemap Generator
/tools/sitemap-generatorGenerate a valid sitemap.xml for any website, free. We find every page from its links and existing sitemaps and write the XML for you to download. - Website Tree Generator
/tools/website-treeSee how a website is linked together. We crawl it and draw its page tree: each page under the page that first linked to it, with depth and link counts. - Website Logo Extractor
/tools/website-logo-extractorGrab a website’s logo, favicon, app icons and share image from its URL, free. See format and size, open or copy each one. No browser extension needed.
Use cases14
- MCP for AI editors
/use-cases/ai-mcpsLive web data inside Claude, Cursor and any MCP client, through 21 tools. - AI agents and RAG
/use-cases/ai-chatsTurn docs and help centres into knowledge bases an LLM can actually retrieve from. - Lead enrichment
/use-cases/lead-enrichmentProspect sites turned into structured company and contact signals. - Coding agents
/use-cases/coding-agentsCrawl docs sites into knowledge bases coding agents search via MCP. - AI training data
/use-cases/ai-trainingCrawl sites and documents into clean markdown, exported as JSONL. - Enterprise AI data
/use-cases/enterprise-ai-dataTraining-grade datasets from the web and PDFs in any field, audited per file, self-hosted on Enterprise. - Deep research
/use-cases/deep-researchWeb search results stored in a knowledge base, searchable by meaning. - Website migration
/use-cases/migration-teamsCrawl the old site and export every page with its URL tree. - Competitive intelligence
/use-cases/competitive-intelligenceMonitor competitor pricing and changelogs; get a webhook on change. - News monitoring
/use-cases/news-monitoringWatch press rooms on a schedule; new articles crawled into a knowledge base. - Content generation
/use-cases/content-generationGround AI drafts in your own docs and site through search. - Product and support teams
/use-cases/product-teamsHelp centre, docs and release notes in one searchable knowledge base. - AI platforms
/use-cases/ai-platformsPer-customer knowledge bases through the API for chat-with-your-website features. - SEO teams
/use-cases/seo-teamsMap URLs, crawl link trees and export page inventories for audits.
Developers14
- API docs
/docsEvery REST route, with cURL, Node and Python samples for each one. - MCP server
/docs/mcp-server21 tools for Claude Code, Cursor and VS Code, and how to point them at a knowledge base. - Node SDK
/docs/node-sdkThe JavaScript and TypeScript client — install, authenticate, scrape, crawl, search. - Python SDK
/docs/python-sdkThe Python client, with the same surface as the Node one. - CLI
/docs/clisarv-kb, the terminal client — ingest, inspect and search without writing code. - n8n node
/docs/n8nKnowledge bases, scrape and crawl jobs, keyword and semantic search as n8n workflow nodes. - Integrations
/integrationsEvery way to use SarvCrawl: MCP server and hosted endpoint, n8n node, Node and Python SDKs, CLI, Chrome extension and REST API. - Benchmarks
/benchmarksReal conversion-audit scores for a text PDF and a scanned memo, hand-checked OCR misses and the retrieval settings we run. - Performance
/performance15 ms p99 even at peak load, no dropped requests, and pages scraped to markdown in under a second. - OCR and PDF examples
/examplesReal output for a scanned memo, a book page photo and a PDF report, with the conversion audit. - Scanned memo to markdown — OCR example
/examples/scanned-memo-ocrA typewritten 1955 memo, scanned with no text layer, run through SarvCrawl OCR. The original page, the unedited markdown and the conversion audit. - Book page photo to markdown — OCR example
/examples/book-page-ocrA printed page of Alice in Wonderland, uploaded as a JPG and read by SarvCrawl OCR. The original image beside the unedited markdown output. - PDF report to markdown — conversion example
/examples/pdf-report-to-markdownA government report page with footnotes, converted from PDF to markdown by SarvCrawl. The original page, the unedited output and every audit score. - Changelog
/changelogWhat shipped, when, and what it changed for anyone calling the API.
Writing11
- Blog
/blogGuides to crawling, parsing and retrieval, honest comparisons of the tools in this market, and how our own pipeline works. - Web scraping for AI in 2026: turning any site into a searchable knowledge base
/blog/web-scraping-for-aiThe complete path from a URL to an answer — crawl, parse, chunk, embed, retrieve — what breaks at each stage, and how the three kinds of tool in this market split the work between you and them. - Cloudflare's AI-crawler wall went up. What actually changed for your pipeline.
/blog/cloudflare-ai-crawler-wallFrom 15 September 2026, mixed-use AI crawlers are blocked by default on ad-supported pages. Here is what the change is, what the block rates actually look like, and what a crawler has to do now to keep working. - Hybrid search in production: fusing BM25 and vector KNN with RRF
/blog/hybrid-search-rrf-in-productionEvery retrieval demo runs one vector index and stops. Production needs two legs and a way to combine them. Here are the constants we actually run — the rank constant, the weights, the floors and the guard that lets the system say it does not know. - How to scrape a website to clean markdown for LLMs
/blog/scrape-a-website-to-markdownHTML costs about five times as many tokens as the same page in markdown, and chunks far worse. Here is why, how to do the conversion four different ways, and the two options that decide whether the output is usable. - Best web scraping APIs in 2026: 8 compared, with real prices
/blog/best-web-scraping-apisFirecrawl, Apify, Bright Data, Oxylabs, ScrapingBee, Zyte, Crawl4AI and SarvCrawl: what each web scraping API is for, its real 2026 entry price, and who should skip it. - Build a RAG knowledge base from a documentation site
/blog/docs-site-to-rag-knowledge-baseThe tutorial usually stops at the first crawl. The hard parts are the ones after it: deduplicating what you fetched, re-crawling only what changed, and noticing when the source moved without you. - Chunking markdown without an LLM: tables, code fences and an 800-token budget
/blog/chunking-markdown-without-an-llmChunking is where retrieval quality is decided, and the popular answer — ask a model to do it — is slow, expensive and non-deterministic. Here is a router that reads the document shape instead, and the three rules that matter more than the token budget. - Scoring your own PDF conversion: ROUGE, Jaccard, and an OCR path that does not lie
/blog/scoring-pdf-conversion-qualityEvery document parser claims high accuracy and none of them show their working. Here is how to measure a conversion against the original, why that method breaks completely on scanned pages, and what to do instead. - Crawl, scrape or map? Picking the right call — and not paying for pages you do not need
/blog/crawl-vs-scrape-vs-mapThree verbs that sound interchangeable and are not. One fetches a page, one follows links, one only discovers URLs for effectively nothing — and using the third first is the cheapest decision in an ingestion pipeline. - llms.txt, measured: the data says it is not doing what you think
/blog/llms-txt-measuredOut of 500 million AI bot visits in a 90-day window, 408 requested llms.txt. That is the whole story on citations — and it is still worth shipping one, for a completely different reason.
Glossary49
- Glossary
/glossaryWeb crawling, extraction, search and AI-data terms, each explained with how it works and how SarvCrawl handles it. - AI web crawlers
/glossary/ai-web-crawlersAI web crawlers are automated clients run by AI companies to fetch web pages, either to collect training data, to build a search index for AI answers, or to fetch a page a user asked about. Each announces itself with a published user-agent token that sites can allow or block in robots.txt. - Asynchronous scraping
/glossary/asynchronous-scrapingAsynchronous scraping is a pattern in which a scraping request is accepted as a job and answered at once with a job ID, the work runs in the background, and the client collects the result later by polling the job or receiving a callback. - BM25
/glossary/bm25BM25 (Okapi Best Matching 25) is the standard keyword relevance formula used by Elasticsearch, OpenSearch and Lucene. It scores a document by how often the query terms appear in it, how rare those terms are across the corpus, and how long the document is. - Breadth-first vs depth-first crawling
/glossary/breadth-first-vs-depth-first-crawlingBreadth-first crawling fetches every page at one link depth before moving to the next, while depth-first crawling follows one chain of links as far as it goes before backtracking. Most web crawlers are breadth-first because it reaches important, shallow pages early. - Crawl budget
/glossary/crawl-budgetCrawl budget is the number of pages a crawler is able and willing to fetch from a site in a given period. For search engines it is set by how fast the server can respond and how much the engine wants the content; for your own crawls it is the page limit and time you allow. - Crawl depth
/glossary/crawl-depthCrawl depth is the number of link hops between the starting URL and a page. The start page is depth 0, pages it links to are depth 1, and so on; a maximum depth setting stops a crawler from following links beyond that distance. - Crawl scope
/glossary/crawl-scopeCrawl scope is the set of rules that decide which discovered URLs a crawler is allowed to follow, typically by domain, subdomain, path, depth and page count. Links outside the scope are recorded or ignored but not fetched. - CSS selectors and XPath
/glossary/css-selectors-and-xpathCSS selectors and XPath are two query languages for picking elements out of an HTML or XML document tree. CSS selectors are short and familiar from stylesheets; XPath is more verbose but can also select by text content, move up to parent elements and compute values. - Document chunking
/glossary/document-chunkingDocument chunking is splitting a document into smaller passages, called chunks, that are embedded and retrieved individually in a search or RAG system. Where the cuts fall decides what retrieval can return. - Embeddings
/glossary/embeddingsAn embedding is a fixed-length list of numbers that a model produces for a piece of text, placed so that texts with similar meaning end up close together. Embeddings are what make semantic search and most RAG retrieval possible. - Headless browser
/glossary/headless-browserA headless browser is a real web browser, such as Chromium or Firefox, running without a visible window and controlled by code. It loads pages, runs JavaScript and exposes the rendered page to a script for scraping, testing or screenshots. - HTML parsing
/glossary/html-parsingHTML parsing is turning the raw text of an HTML document into a tree of elements, the Document Object Model (DOM), that code can navigate and query. Every scraper, browser and HTML-to-markdown converter parses HTML before it can do anything with a page. - HTML to Markdown
/glossary/html-to-markdownHTML to Markdown conversion turns a web page’s markup into Markdown, a lightweight plain-text format that keeps headings, lists, links, tables and code while dropping tags, attributes and styling. It is a common way to prepare web content for LLMs and search. - HTTP 429 and rate limiting
/glossary/http-429-rate-limitingRate limiting is a server capping how many requests a client may make in a period of time. HTTP 429 Too Many Requests is the status code a server returns when that cap is exceeded, often with a Retry-After header saying when to try again. - HTTP status codes
/glossary/http-status-codesHTTP status codes are the three-digit numbers a server returns with every response to say what happened: 2xx success, 3xx redirect, 4xx a problem with the request, 5xx a problem on the server. For a scraper they decide whether to keep the page, follow it, retry it later or drop it. - Hybrid search
/glossary/hybrid-searchHybrid search runs a keyword search (usually BM25) and a vector search over the same corpus and merges the two ranked lists into one, so results match both exact terms and meaning. - Incremental crawling
/glossary/incremental-crawlingIncremental crawling keeps a previously crawled collection up to date by fetching only new or changed pages instead of recrawling the whole site each time. It relies on change signals such as sitemap dates, HTTP validators and content hashes. - JavaScript rendering
/glossary/javascript-renderingJavaScript rendering is running a page’s scripts, usually in a headless browser, so that content built in the browser exists in the HTML a crawler reads. Without it, crawlers see only the initial HTML the server sent. - Knowledge base (for AI)
/glossary/knowledge-baseA knowledge base for AI is a searchable collection of documents, split into chunks and indexed by keyword and by embedding, that a chatbot, agent or RAG pipeline queries to answer from your own content. - Link extraction
/glossary/link-extractionLink extraction is pulling every hyperlink out of a page and resolving each one to an absolute URL. Crawlers use it to discover new pages, and analysts use it to map a site’s structure or audit its outbound links. - LLM grounding
/glossary/llm-groundingLLM grounding is the practice of making a language model answer from specific source material supplied at query time, such as retrieved documents or web pages, rather than from its training data alone, so each claim can be traced to and checked against a cited source. - LLM tokens
/glossary/llm-tokensTokens are the units a large language model reads and writes: pieces of words, whole words, punctuation or bytes, produced by a tokenizer. Context limits, prices and chunk sizes are all measured in tokens, not characters or words. - LLM-ready content
/glossary/llm-ready-contentLLM-ready content is web or document content converted into clean, structured text, usually markdown, with navigation, scripts and other page furniture removed and the source URL kept, so a language model can read, chunk and cite it without further cleanup. - llms.txt
/glossary/llms-txtllms.txt is a proposed markdown file served at the root of a website that lists its most important pages with one-line descriptions, so a language model or coding agent gets a curated map of the site instead of having to work it out from HTML. - Main content extraction
/glossary/main-content-extractionMain content extraction, also called boilerplate removal, isolates the primary content of a web page, such as an article or documentation body, and discards repeated page furniture like navigation, headers, footers, sidebars, ads and cookie banners. - Model Context Protocol (MCP)
/glossary/model-context-protocolThe Model Context Protocol (MCP) is an open protocol, introduced by Anthropic in 2024, that standardises how AI applications connect to external tools and data. A service exposes an MCP server once, and any MCP client, such as Claude Code, Cursor or VS Code, can use it. - Near-duplicate detection
/glossary/near-duplicate-detectionNear-duplicate detection finds documents that are almost but not exactly identical, such as a page and its print version or two locale variants that differ only in navigation, so they can be removed or merged before indexing. - OCR (optical character recognition)
/glossary/ocrOptical character recognition (OCR) converts images of text, such as scans, photos and screenshots, into machine-readable characters. It is the only way to get text out of a document that has no text layer. - PDF parsing
/glossary/pdf-parsingPDF parsing is extracting text and structure, such as headings, paragraphs, lists and tables, from a PDF file into a usable format like markdown, JSON or plain text. - Polite crawling
/glossary/polite-crawlingPolite crawling is fetching a website in a way that respects its owner’s wishes and does not overload its server: obeying robots.txt, limiting concurrent requests, pausing between fetches, and backing off when the server signals it is busy. - Query expansion
/glossary/query-expansionQuery expansion is the practice of rewriting or adding to a user's search query before retrieval, with synonyms, related terms, rephrasings or several sub-queries, so the search finds relevant documents that do not share the query's exact words. - Reciprocal rank fusion
/glossary/reciprocal-rank-fusionReciprocal rank fusion (RRF) is a method for merging several ranked result lists into one. Each document scores 1 / (k + rank) from every list it appears in, the scores are summed, and the merged list is sorted by that sum. - Redirect handling
/glossary/redirect-handlingRedirect handling is how a crawler responds when a server answers a request with a 3xx status and a new location: following the redirect, limiting chains and loops, deciding which URL to record, and checking that the destination is still in scope. - Reranking
/glossary/rerankingReranking is a second retrieval stage that takes the top candidates from a fast first-stage search and reorders them with a slower, more accurate relevance model, such as a cross-encoder or a language model that reads the query and each passage together. - Retrieval-augmented generation (RAG)
/glossary/retrieval-augmented-generationRetrieval-augmented generation (RAG) is a pattern in which a system first retrieves relevant passages from an external corpus and then gives them to a language model as context, so the answer is grounded in your documents rather than only in what the model memorised during training. - Robots.txt
/glossary/robots-txtRobots.txt is a plain text file at the root of a website that tells crawlers which paths they may and may not fetch. It follows the Robots Exclusion Protocol, standardised as RFC 9309, and is a request rather than an access control. - Scanned vs text PDFs
/glossary/scanned-vs-text-pdfA text PDF contains a text layer, the actual characters, which can be extracted exactly. A scanned PDF contains only images of pages, so its text has to be reconstructed with OCR. The two need different parsing and different quality checks. - Seed URL
/glossary/seed-urlA seed URL is the address a crawler starts from. The crawler fetches it first, extracts its links, and discovers every other page in the crawl by following links outward from it. - Semantic search
/glossary/semantic-searchSemantic search retrieves documents by meaning rather than by matching words: the query and the documents are turned into embeddings, and the results are the documents whose vectors are closest to the query's. - URL frontier
/glossary/url-frontierThe URL frontier is the queue of discovered URLs a crawler has not fetched yet. It decides which page is fetched next and, together with a record of visited URLs, stops the same page being fetched twice. - URL normalization
/glossary/url-normalizationURL normalization (or canonicalization) is the process of rewriting URLs into a consistent form so that different spellings of the same address are recognised as one page. Crawlers use it to avoid fetching and storing duplicates. - Web crawler
/glossary/web-crawlerA web crawler is a program that starts from one or more URLs, downloads each page, extracts the links it finds and follows them to discover more pages. Search engines, archives and AI data pipelines all use crawlers to collect pages at scale. - Web crawling vs web scraping
/glossary/web-crawling-vs-web-scrapingWeb crawling is discovering pages by following links from page to page; web scraping is extracting content or data from a page once you have it. A crawler answers “which URLs exist?”, a scraper answers “what is on this URL?”, and most real jobs do both. - Web scraping
/glossary/web-scrapingWeb scraping is the automated extraction of content or data from web pages: fetching a page and turning its HTML into something a program can use, such as text, markdown, tables or JSON fields. - Web scraping API
/glossary/web-scraping-apiA web scraping API is a hosted service that fetches web pages on your behalf and returns their content in a structured form, such as markdown, HTML or JSON, through an HTTP request. It replaces running your own browsers, queues and parsers. - Web search API
/glossary/web-search-apiA web search API is an HTTP endpoint that takes a search query and returns results from the public web as structured data, typically titles, URLs and snippets, and in some services the full content of each result page. - Website change detection
/glossary/website-change-detectionWebsite change detection is re-fetching chosen pages on a schedule, comparing each copy with the last one, and reporting what was added, removed or changed. It turns a site into a feed of differences, so you react to changes instead of re-reading everything. - XML sitemap
/glossary/sitemap-xmlAn XML sitemap, usually served as sitemap.xml, is a file that lists the URLs on a site a crawler should know about, optionally with the date each page last changed. It helps crawlers find pages that internal links alone might miss.
Company06
- Create an account
/signupSign up and start with 1 billion tokens. No card. - Login
/loginSign in to the console. - Service status
/statusWhether the API, the workers and the queue are up right now, and the recent history. - Sitemap
/sitemapThis page — every public page on the site, grouped. - Press and brand
/press-brandAbout SarvCrawl boilerplate, logo downloads, brand colours, typography, name and usage rules, press contact. - Talk to sales
/talk-to-salesVolume pricing, self-hosting in your own VPC, a security review, or migrating a corpus you already have.
Legal03
- Terms of service
/terms-of-serviceThe agreement between you and Sarv Webs Pvt. Ltd. for the use of SarvCrawl — what we run, what you are responsible for putting into it, and what each of us owes the other when something goes wrong. - Privacy policy
/privacy-policyWhat SarvCrawl collects, why, who else ever sees it, how long we keep it, and how to make us delete it. Written to be read rather than skimmed past. - Delete your account
/account-deletionnot indexedClose an account and have its content removed. Public on purpose — someone who wants out may already be unable to sign in.