Crawl, scrape or map? Picking the right call — and not paying for pages you do not need

01The three verbs

Every API in this space has some version of these three, and the names are consistent enough across vendors that the distinction transfers.

Call	Input	What comes back	Costs you
Scrape	One URL	That page, converted	One page
Crawl	A root URL plus bounds	A tree of pages, converted	Every page it fetched
Map	A root URL	URLs only — no content	Effectively nothing: no page cost

The first two are obvious and the third is the one people skip, which is a shame, because it is the only one that tells you what the other two are about to cost.

Map is the estimate. Running it before a crawl turns "how big is this site?" from a question you answer by paying, into one you answer by asking.

02Scrape: when you know exactly which page

One URL, one document. Use it when the page is the unit of work: a specific article, a single pricing page, a prospect's about page, a PDF you have a direct link to.

curl -X POST https://crawl.sarv.com/api/jobs/scrape \
  -H "content-type: application/json" -H "x-api-key: $KB_API_KEY" \
  -d '{"kb_id":"kb_abc123","url":"https://example.com/pricing",
       "formats":["markdown","links"]}'

Two habits make scrape more useful than it looks. Ask for links alongside the markdown, and you get the page's outbound link graph — which is how one scrape becomes a decision about whether to crawl. And leave include_docs on, so a linked spec sheet or price list comes in with the page rather than being invisible.

Scrape is also the right call for a list of known URLs, which is a more common shape than it sounds — you have a sitemap, or a CSV of company domains, or the output of a previous map. A bounded set of scrapes is more predictable than a crawl and lets you retry individual failures instead of restarting a tree.

03Crawl: when you want a section, and the bounds are the whole point

Crawl follows links from a root. The parameters are not decoration; they are the difference between ingesting a documentation site and ingesting somebody's entire CMS.

max_depth: How many link-hops from the root. Depth 2 on a docs site is usually the docs. Depth 5 is usually the docs, the blog, the careers page and every tag archive.
limit: A hard ceiling on pages. Set it even when you think you know the size — it is the difference between a surprising bill and a truncated crawl you can extend.
exclude_paths: The highest-leverage one. /blog/, /changelog/, /tag/, /author/, /*?page= and locale prefixes are usually most of a site by page count and almost none of it by value.

curl -X POST https://crawl.sarv.com/api/jobs/crawl \
  -H "content-type: application/json" -H "x-api-key: $KB_API_KEY" \
  -d '{
    "kb_id": "kb_abc123",
    "url": "https://example.com/docs",
    "max_depth": 3,
    "limit": 500,
    "exclude_paths": ["/blog/", "/changelog/", "/tag/"]
  }'

A crawl returns a tree, not a list, and the tree is worth looking at. GET /api/jobs/:id/tree shows what was found from where, which is how you discover that four hundred of your five hundred pages came from a paginated archive nobody wanted.

04Map: the call to make first

Map discovers URLs and returns them. No content, no conversion, no chunking, no embedding — and therefore no page cost. It exists so you can answer three questions before committing to anything.

How big is this actually? A site that felt like two hundred pages is often four thousand, and you would rather find that out now.
What is the URL shape? Seeing the paths is what tells you which exclude_paths to set. You cannot guess /en-gb/ and /print/ and /?utm_source= from the homepage.
Is there a sitemap worth seeding from? Crawling from a sitemap is one request per page you want, rather than the full link graph. Cheaper for you, much cheaper for the site — which, in a year where blocking has become the default, is not only a courtesy.

curl -X POST https://crawl.sarv.com/api/jobs/map \
  -H "content-type: application/json" -H "x-api-key: $KB_API_KEY" \
  -d '{"kb_id":"kb_abc123","url":"https://example.com",
       "limit":1000,"include_subdomains":false}'

That last pipeline — group the discovered URLs by directory and count them — takes ten seconds and is usually the moment somebody says "why are there nine hundred pages under /tag/".

include_subdomains deserves a moment of thought, because it is the switch that turns "the docs" into "the docs, the status page, the community forum and the marketing microsite from 2019". Leave it off unless you have a reason.

05The sequence that costs least

Put together, the cheap order is always the same:

Map the site. Free, and it tells you the shape.
Group the URLs by path and decide what you actually want.
Crawl with exclude_paths set from what you just learned, and a limit that matches the count you just saw.
Look at the job tree and ten pages by eye before trusting the rest.
Set up a monitor so the corpus stays current instead of decaying.

Steps 1 and 2 take five minutes. Skipping them is how a 500-page crawl becomes a 6,000-page one, and — worse than the cost — how nine hundred tag-archive pages end up in a vector index, where they are semantically indistinguishable from each other and drag on every query. Junk in the corpus is a retrieval problem, not just a billing one; there is a whole section on what that does to search quality in hybrid search in production.

Everything you ingest, you pay for three times: once to fetch it, once to embed it, and then continuously, in the retrieval quality of every query that has to step around it.

For the full ingest-and-refresh version of this with deduplication and incremental re-crawls, see building a RAG knowledge base from a documentation site. For where these three calls sit in the wider pipeline, web scraping for AI in 2026.