What are HTTP 429 and rate limiting?

Fixed window

Count requests per clock minute; reset at the boundary. Simple, but allows bursts at the edges.

Sliding window

Count requests in the last N seconds at every moment. Smoother.

Token bucket

A bucket refills at a steady rate and each request spends a token, allowing short bursts up to the bucket size.

Two headers matter on a 429 response. Retry-After gives either a number of seconds or an HTTP date. Many APIs also send rate limit headers, such as RateLimit-Remaining or the older X-RateLimit-* family, so clients can slow down before they hit the cap.

HTTP/1.1 429 Too Many Requests
Retry-After: 30
Content-Type: application/json

{"error": "rate limit exceeded"}

Honour Retry-After when it is present; otherwise back off exponentially with random jitter so many clients do not retry in lockstep.

import random, time, requests

def get_with_backoff(url, retries=5):
    for attempt in range(retries):
        r = requests.get(url, timeout=30)
        if r.status_code not in (429, 503):
            return r
        wait = r.headers.get("Retry-After")
        delay = float(wait) if wait and wait.isdigit() else min(60, 2 ** attempt)
        time.sleep(delay + random.uniform(0, 1))
    r.raise_for_status()

• Retrying immediately. Tight retry loops turn a rate limit into a block.
• Ignoring HTTP-date values in Retry-After; parse both forms.
• Per-host concurrency. A crawler that respects a delay per worker but runs 50 workers against one host is still hammering it. Limit per host.
• Retrying non-idempotent requests such as POSTs that create resources, which can duplicate them.
• Treating 429 as permanent. It is temporary by definition; record it and come back later rather than dropping the URL.

The surest way to avoid rate limits is to send fewer requests. Crawl jobs take a limit and max_depth and exclude_paths, and run with ignoreQueryParameters so URL variants are not fetched twice. The map endpoint lists URLs from sitemaps and links without fetching page content, and the monitor endpoint re-checks just the pages you choose on a schedule instead of recrawling a whole site to spot changes.