What Is a Headless Browser? Uses, Tools and Limits

Headless browsers in more depth

A headless browser has the same engine as the browser on your desktop: the HTML parser, CSS layout, JavaScript engine and network stack. It just has no window. Instead of a person clicking, a program sends commands over a protocol such as the Chrome DevTools Protocol or WebDriver BiDi.

That makes it the standard tool when a plain HTTP request is not enough: pages that need JavaScript rendering, flows that need clicks or form input, and screenshots or PDFs of a page as a person would see it.

Tools and how they work

Playwright: Drives Chromium, Firefox and WebKit from Node, Python, Java and .NET. Built-in auto-waiting.
Puppeteer: Node library for Chrome and Chromium over the DevTools Protocol; Firefox support via WebDriver BiDi.
Selenium: The long-established WebDriver-based option, common in test suites.

A typical scraping session launches a browser, opens a page (a tab), navigates to a URL, waits for a readiness signal, reads the DOM or takes a screenshot, then closes the page. Reusing one browser process across many pages saves startup time; isolating each job in a fresh context keeps cookies and storage from leaking between jobs.

Example: screenshot and HTML with Puppeteer

import puppeteer from 'puppeteer';

const browser = await puppeteer.launch();
const page = await browser.newPage();
await page.goto('https://example.com', { waitUntil: 'networkidle0', timeout: 30000 });
const html = await page.content();
await page.screenshot({ path: 'example.png', fullPage: true });
await browser.close();

Common pitfalls

Resource use. Each browser takes hundreds of megabytes of memory. Running dozens in parallel needs real capacity planning and limits on concurrency.
Leaked processes. A crash that skips browser.close() leaves orphaned processes that slowly exhaust a server.
Unbounded waits. Pages that never go idle hang a worker unless every wait has a timeout.
Detection. Many sites can tell an automated browser from a person and may block or challenge it. Respect site terms and rate limits.
Using it when you do not need it. Server-rendered pages are faster and cheaper to fetch with plain HTTP.

How SarvCrawl handles it

SarvCrawl runs the browsers for you. Every scrape and crawl renders pages in a real browser, and you choose the output: markdown, HTML, links, screenshots or JSON metadata. There is no browser fleet to size or patch on your side; you send a request with an x-api-key to the REST API or use the sarvcrawl SDK. For pages behind a login in your own browser, the Chrome extension sends the page from your tab instead.

Frequently asked questions

What is the difference between headless and headful?

A headful browser shows a window; a headless one does not. The engine is the same, so pages render the same way, though some sites detect headless mode.

Is Playwright or Puppeteer better for scraping?

Both work well. Playwright supports more browsers and languages; Puppeteer is focused on Chrome from Node. Pick the one that fits your stack.

Can a headless browser take screenshots?

Yes. Screenshots and PDF rendering are standard features, including full-page captures.

What is a headless browser?