Status codes in more depth
Every HTTP response starts with a status line, such as HTTP/1.1 200 OK. The first digit gives the class; the rest narrows it down. Browsers mostly hide the code, but a crawler has to act on it for every URL, and storing it is the cheapest way to see what went wrong in a large crawl.
The codes that matter, and what to do
| Code | Meaning | What a scraper should do |
|---|---|---|
| 200 OK | The page was returned. | Keep it, but check it is the real content and not an error or login page. |
| 301, 308 | Moved permanently. | Follow, and record the new URL in place of the old one. |
| 302, 303, 307 | Moved temporarily. | Follow, but keep the original URL as the one to revisit. |
| 304 Not Modified | Unchanged since your last copy (conditional request). | Reuse the stored copy. |
| 401, 403 | Authentication required, or refused. | Often a block or login wall. Retrying unchanged rarely helps. |
| 404 Not Found | Nothing at this URL. | Drop it; recheck later if it used to exist. |
| 410 Gone | Deliberately and permanently removed. | Drop it and stop revisiting. |
| 429 Too Many Requests | You hit a rate limit. | Slow down and retry after Retry-After. |
| 500, 502, 503, 504 | Server or gateway error. | Retry later with backoff; 503 often means overload. |
429 has its own entry: see HTTP 429 and rate limiting for backoff with jitter.
Redirects and soft errors
Redirects matter to a crawler because they change what a URL is. A permanent redirect means the old address should be replaced; a chain of several hops, or a loop, should be cut off after a few steps. A redirect can also lead off the site entirely, which a scoped crawl has to refuse. See redirect handling.
The status code is also not always honest. A soft 404 is a “page not found” message served with 200, and many bot challenges and login pages return 200 too. Checking for very short content or known error text catches most of them.
Common pitfalls
- Treating every non-200 as permanent. 429 and 5xx are temporary by nature; record them and come back.
- Retrying 404s in a loop. They will not fix themselves on the next request.
- Indexing soft 404s. Error pages served with 200 end up in a knowledge base as junk.
- Not storing the code. Without it, a crawl that quietly returned 403 for half the site looks like a short site.
How SarvCrawl handles it
SarvCrawl stores the HTTP status code of every page it fetches, and GET /api/kb/:id/pages returns it with each page, so you can find failures after a crawl. Responses of 401, 403 and 429 are retried through a different proxy path. Redirects are followed; if the target falls outside the crawl’s scope, the page is rejected with an error saying why. When a monitored page stops loading, the monitor endpoint reports it with status error rather than skipping it.
Frequently asked questions
Should a scraper retry a 403?
Once or twice at most. A 403 usually means the server has refused the client, and repeating the same request rarely changes that. Slowing down, or checking whether the site allows automated access, is more useful.
What is the difference between 404 and 410?
Both mean nothing is at the URL. 410 Gone says the removal is deliberate and permanent, so a crawler can stop checking; a 404 may come back.
What is a soft 404?
An error page returned with status 200 instead of 404. It looks like success to a scraper, so detect it by content: very short pages or known “not found” text.
Which status codes are safe to retry?
429 and 5xx codes, with backoff. For 429, wait for the Retry-After time if the server gives one.