What Is JavaScript Rendering in Web Crawling?

JavaScript rendering in more depth

Modern sites deliver pages in two broad ways. With server-side rendering (SSR) or static generation, the server sends HTML that already contains the content. With client-side rendering (CSR), the server sends a near-empty shell plus JavaScript, and the browser fetches data and builds the page. Many sites mix the two.

A plain HTTP request, the kind curl or requests makes, never runs JavaScript. On a CSR page it returns something like <div id="root"></div> and nothing else. Rendering closes that gap by loading the page in a real browser engine and reading the DOM after scripts have run.

How rendering works

A headless browser loads the URL like a normal visitor.
It downloads and executes scripts, which call APIs and insert content into the DOM.
The crawler waits until the page is “ready”, then serialises the rendered DOM to HTML.
The rendered HTML goes through the normal extraction steps.

Deciding when a page is ready is the hard part. Common signals are:

load event: Fires when the initial resources finish. Often too early for CSR apps, which fetch data afterwards.
Network idle: No network requests for a short window. A good general-purpose signal, but needs a cap for pages that poll forever.
Selector wait: Wait for a specific element. Precise, but requires knowing each site’s markup.
Fixed delay: Sleep for N seconds. Simple, slow and unreliable.

Example: rendering with Playwright

from playwright.sync_api import sync_playwright

with sync_playwright() as p:
    browser = p.chromium.launch()
    page = browser.new_page()
    page.goto("https://app.example.com/pricing", wait_until="networkidle", timeout=30_000)
    html = page.content()  # the DOM after scripts ran
    browser.close()

A quick check for whether a page needs rendering: view the page source (not the inspector). If the text you want is missing from the source, it is built by JavaScript.

Common pitfalls

Cost. Rendering uses far more CPU, memory and time than a plain fetch. Render only what needs it if you run your own fleet.
Waiting forever. Pages with analytics beacons, chat widgets or long polling never go fully idle; always cap the wait.
Lazy content. Infinite scroll and “load more” buttons need interaction, not just waiting.
Content behind consent banners may not render until a button is clicked.

How SarvCrawl handles it

Every page SarvCrawl scrapes or crawls is rendered in a real browser before it is read, so client-rendered content is captured without per-site configuration. The rendered page is then converted to markdown, and you can also request the rendered HTML or a screenshot to check what the browser saw. You can try a URL in the playground.

Frequently asked questions

Do search engines render JavaScript?

Google renders JavaScript in a later pass after the initial crawl. Many other crawlers, including several AI crawlers, read only the initial HTML, so server-rendered content is the safest bet.

How can I tell if a site needs JavaScript rendering?

Compare the raw page source with what you see in the browser. If the content is missing from the source, it is rendered client-side.

Is JavaScript rendering slow?

It is slower than a plain HTTP fetch because the browser must download and run scripts and wait for data. Capped waits keep it predictable.

What is JavaScript rendering?