Redirects in more depth
A redirect is a server saying “not here, try there”. The response carries a 3xx status code and a Location header with the new URL. Browsers follow it silently; a crawler has to decide what to do and what to remember.
Sites redirect for many reasons: moving from HTTP to HTTPS, adding or removing www, renaming pages, sending visitors to a language version, or forcing a login. Each one costs the crawler an extra request, and some lead somewhere the crawl was never meant to go.
Redirect status codes
| Code | Meaning | Method kept? |
|---|---|---|
| 301 | Moved permanently | Not guaranteed; clients often switch POST to GET |
| 302 | Found (temporary) | Not guaranteed; clients often switch POST to GET |
| 303 | See other | No; always GET |
| 307 | Temporary redirect | Yes |
| 308 | Permanent redirect | Yes |
For a crawler doing GET requests, the method distinction rarely matters. The permanent/temporary distinction does: search engines treat a permanent redirect as a signal to replace the old URL with the new one, and a temporary one as a reason to keep the old URL. Pages can also redirect without a 3xx, using a <meta http-equiv="refresh"> tag or JavaScript, which only a crawler that renders the page will see.
Example: a redirect chain
http://example.com/docsreturns 301 tohttps://example.com/docs.https://example.com/docsreturns 301 tohttps://www.example.com/docs/.https://www.example.com/docs/returns 200 with the page.
Three requests for one page. Crawlers cap chains at a fixed number of hops and treat loops, where a URL eventually redirects back to itself, as errors. Seeding at the final URL avoids the chain altogether.
Common pitfalls
- Redirects that leave the scope. An in-scope link can redirect to another domain, a login service or a different section. Check scope after the redirect, not only before.
- Duplicate content via redirects. Two URLs that redirect to the same page are one page. Without tracking the destination, a crawler may store it twice.
- Confusing redirects with canonicals. A
rel="canonical"tag is a hint in the HTML, not a redirect; the server still returns 200. - Redirect-to-homepage for missing pages. Some sites redirect every dead link to
/, which looks like success but is really a soft 404.
How SarvCrawl handles it
Crawls follow redirects. If a redirect takes a page outside the crawl’s scope, for example to another domain, the page is rejected with an error explaining why rather than being stored. The HTTP status code of every stored page is kept and returned by GET /api/kb/:id/pages. SarvCrawl does not read rel="canonical" tags, and the final URL after a redirect is not guaranteed to be the one recorded, so seed crawls at the address a site actually serves.
Frequently asked questions
Do crawlers follow 301 and 302 redirects?
Yes. Almost all crawlers follow both. Search engines treat 301 and 308 as permanent and 302 and 307 as temporary when deciding which URL to keep in their index.
How many redirects will a crawler follow?
It depends on the crawler. Most cap a chain at a small number of hops; Google says it follows up to ten. Shorter chains are faster and safer.
Do crawlers follow JavaScript redirects?
Only crawlers that render pages in a browser will see them. A crawler that only reads the raw HTML sees the original page.
Is a canonical tag the same as a redirect?
No. A redirect sends the client to a different URL. A canonical tag serves the page normally and suggests which URL should be treated as the main version.