Map

Every URL on a site, in seconds

Map reads a site’s sitemaps and links and returns the list of URLs it finds, without rendering a single page. It is the cheap first look before a crawl.

Scrape Crawl Map Search Parse Monitor Crawl tree

5,000
URLs per map on the API, from sitemaps and links
0
Pages rendered: map discovers, it does not fetch content
Seconds
For most sites, so you can plan a crawl before you pay for it

Why map before you crawl

Fast and cheap

No browser, no parsing: map is the quickest way to see how big a site is.

Plan exclusions

See the archives, tag pages and duplicates you will want in exclude_paths before the crawl starts.

Titles where known

URLs come back with the page title when the sitemap or link gives one.

Export the list

Download the URL list as CSV, or as a sitemap.xml with the free sitemap generator.

Feed a scrape list

Pick exactly the URLs you need and scrape just those, instead of crawling everything.

Find the right section

Map a docs site to find the API reference or changelog before pointing an agent at it.

How map works

01 / 03
Send the site
POST /api/jobs/map with the URL and a limit.
02 / 03
Sitemaps and links are read
Map reads the site’s sitemaps and the links it can see, and removes duplicates.
03 / 03
Use the list
Crawl it, scrape a subset, generate a sitemap.xml, or keep it for an audit.

Map from your code

A map is a short job; its URL list is attached to the job when it completes.

curl -X POST https://crawl.sarv.com/api/jobs/map \
  -H "x-api-key: YOUR_API_KEY" -H "Content-Type: application/json" \
  -d '{"kb_id":"kb_abc123","url":"https://docs.example.com","limit":1000}'

Map API reference

Questions

How do I find all the pages on a website?

Run a map: it reads the site’s sitemaps and links and returns every URL it finds. The demo above shows the first 25; a free account shows the full list.

Is map the same as a sitemap?

Map uses the sitemap when the site has one, and adds URLs found in links, so it also works for sites with a missing or stale sitemap.

Does map fetch page content?

No. It only discovers URLs, which is why it is fast. Use scrape or crawl to read the pages.

Can I get the result as a sitemap.xml?

Yes, the free sitemap generator turns a map into a valid sitemap.xml file.

See every page before you crawl

1 billion tokens free on sign-up, no card. Roughly 500 tokens a page.

Start with 1 billion free tokens Read the API docs