XML sitemaps in more depth
The sitemap protocol, published at sitemaps.org and supported by the major search engines, defines an XML file of <url> entries. Each has a required <loc> and optional <lastmod>, <changefreq> and <priority>. Sitemaps are discovered through a Sitemap: line in robots.txt, by submitting them to a search console, or at the conventional /sitemap.xml path.
A sitemap is a hint, not a command. It says “these URLs exist”, not “index these”. It is most useful for large sites, new sites with few inbound links, and pages that are hard to reach by following links, such as deep pagination.
Format and limits
- One sitemap file may hold at most 50,000 URLs and be at most 50 MB uncompressed. It may be gzipped.
- Larger sites use a sitemap index: a file of
<sitemap>entries that each point to a child sitemap. - URLs must be absolute and on the same host as the sitemap, unless cross-host submission is verified.
- Google says it uses
<lastmod>when it is consistently accurate and ignores<changefreq>and<priority>.
Sitemaps for your own crawls
Sitemaps are not only for search engines. For anyone collecting a site, the sitemap is the fastest inventory available. It lists pages without fetching them, often includes pages that navigation does not link to, and its lastmod values tell an incremental crawl which pages may have changed. Combining sitemap URLs with link extraction from the pages themselves gives the most complete picture, because each catches pages the other misses.
Sitemaps also reveal how a site is organised. Large sites often split their sitemap index by section or content type, such as sitemap-docs.xml and sitemap-blog.xml, which tells you at a glance how much content each section holds and lets you crawl one section without touching the rest. Comparing the sitemap’s URL count with what a link-following crawl finds is a quick health check in both directions: pages only in the sitemap are often orphaned, and pages only found by links may have been left out of the sitemap by mistake.
Common pitfalls
- Stale or fake `lastmod`. Setting every date to today trains crawlers to ignore the field.
- Listing non-canonical URLs, redirects, 404s or
noindexpages. - Forgotten sitemaps that a CMS stopped updating years ago, so new pages never appear.
- Assuming completeness. Many sites omit whole sections; always check the sitemap against what the site links to.
How SarvCrawl handles it
The map endpoint lists a site’s URLs from its sitemaps and from links, without fetching page content, so you can see what a site contains before deciding what to crawl. To build a sitemap for a site that lacks one, the free sitemap generator crawls it and outputs the XML.
Frequently asked questions
Do I need a sitemap?
Small, well-linked sites can be crawled without one. Sitemaps help most on large sites, new sites, and sites with pages that are hard to reach through links.
How many URLs can a sitemap contain?
Up to 50,000 URLs and 50 MB uncompressed per file. Larger sites split URLs across several sitemaps and list them in a sitemap index.
Does adding a URL to a sitemap guarantee indexing?
No. It helps discovery, but search engines still decide whether to crawl and index each URL.