What asynchronous scraping is
In a synchronous call the client sends a URL and the connection stays open until the content comes back. That is simple, and it works for a fast static page. It breaks when the page needs a browser to render, when the site is slow, or when the request is a crawl of hundreds of pages: the work outlasts any sensible HTTP timeout.
In an asynchronous call the service validates the request, queues it and replies immediately, typically with 202 Accepted and a job ID. The client then asks about that job until it is finished, or the service calls the client back. The request and the result are two separate exchanges.
Sync versus async
| Synchronous | Asynchronous | |
|---|---|---|
| Response | The content | A job ID |
| Typical status | 200 OK | 202 Accepted |
| Long work | Times out | Runs as long as it needs |
| Progress | None until done | Status, logs, partial results |
| Cancelling | Close the connection and hope | An explicit cancel call |
| Client code | One request | Submit, then poll or receive a callback |
Crawls are asynchronous almost everywhere, because a crawl is many fetches whose count is unknown at the start. Single-page scrapes are offered both ways by different services.
Job IDs, polling and callbacks
- Submit the job. The response carries a job ID; store it, since it is the only handle on the work.
- Poll the job by ID at an interval. Each response reports a status such as queued, running, completed or failed.
- When the status is final, read the result, or the error if it failed.
The alternative to polling is a webhook: the client registers a URL and the service POSTs to it when the job finishes. That saves requests but needs a public endpoint and handling for missed or repeated deliveries, so many clients poll anyway.
Poll politely. Start with a short interval and back off as the job runs longer, set an overall deadline, and treat a polling 429 as a reason to slow down, not to retry immediately. See HTTP 429 and rate limiting.
Common pitfalls
- Losing the job ID. If the submit response is not stored, the work runs and nobody can collect it.
- Polling in a tight loop. It burns rate limit and does not make the job faster.
- No deadline. A job stuck in a non-final state should eventually be cancelled and reported, not polled forever.
- Treating 202 as success. Accepted means queued. The job can still fail on the target site.
- Resubmitting on timeout. A slow poll response does not mean the job is gone; resubmitting can start duplicate work.
How SarvCrawl handles it
Every SarvCrawl job endpoint is asynchronous: scrape, crawl, web search, map, monitor and upload all return 202 with a job_id. Poll GET /api/jobs/:id for status, read /logs and /tree for progress, and cancel with DELETE /api/jobs/:id/cancel.
Each user can run one active crawl and one active scrape at a time; another returns 429 with CONCURRENCY_LIMIT. There is no batch-scrape endpoint and no webhook on crawl completion, so completion is read by polling. Monitors are the exception on the callback side: they POST page changes to the notify_url you give them.
Frequently asked questions
Why do scraping APIs return a job ID instead of the page?
Rendering a page in a browser can take seconds and a crawl can take minutes, longer than an HTTP request should stay open. A job ID lets the work run in the background while the client checks back.
How often should I poll a scraping job?
Every few seconds at first, backing off for longer jobs, with an overall deadline. Faster polling uses rate limit without making the job finish sooner.
Is a webhook better than polling?
It uses fewer requests and reports completion sooner, but it needs a publicly reachable endpoint and must cope with missed or duplicate deliveries. Polling is simpler to run and works from anywhere.
What does HTTP 202 mean for a scrape?
That the request was valid and queued, not that it succeeded. The job can still fail later, which you learn by checking its status.