What Is Web Scraping? Definition, Methods and Pitfalls

Web scraping in more depth

A browser renders HTML for people to read. A scraper reads the same HTML for a program. The output can be narrow, such as a price and a product name pulled into a spreadsheet, or broad, such as the full readable text of every article on a site for a search index or a language model.

Scraping is usually paired with crawling. The crawler decides which pages to visit; the scraper decides what to keep from each one. A single-page job, such as converting one article to markdown, is scraping without crawling.

How web scraping works

Fetch the page over HTTP, or load it in a headless browser when content is built by JavaScript.
Parse the HTML into a tree (the DOM).
Select the parts you want with CSS selectors or XPath, or use main content extraction to keep the article and drop navigation.
Transform the result into the target format: cleaned text, markdown, a table or JSON.
Store it, with the source URL and fetch time so the data can be refreshed and traced.

There are two broad styles. Selector-based scraping targets exact elements and is precise but breaks when the site changes its markup. Content-based scraping converts the whole readable page and is robust to layout changes, which is why it suits AI and search use cases.

Example: selectors versus whole-page conversion

from bs4 import BeautifulSoup
import requests

html = requests.get("https://example.com/post", timeout=10).text
soup = BeautifulSoup(html, "html.parser")

# Selector-based: exact fields, brittle if the markup changes
title = soup.select_one("h1").get_text(strip=True)

# Content-based: keep the readable body as text
for tag in soup(["nav", "footer", "aside", "script", "style"]):
    tag.decompose()
body = soup.get_text("\n", strip=True)

For one-off conversions you can paste a URL or HTML into a tool such as the HTML to markdown converter.

Common pitfalls

Scraping the shell. A plain HTTP request to a single-page app returns an empty template; the content only appears after JavaScript rendering.
Keeping the chrome. Menus, cookie banners and footers repeated on every page pollute search results and waste model context.
Brittle selectors. A class-name change silently returns nothing. Validate output and alert on empty fields.
Ignoring the rules. Respect robots.txt, terms of service and rate limits.

How SarvCrawl handles it

The scrape endpoint renders the page in a real browser and returns the formats you ask for: markdown, HTML, links, screenshots and JSON metadata. Main-content extraction (onlyMainContent) strips nav, footer, aside, script and style elements. Results land in a knowledge base where they are chunked and embedded for search, and you can call it from the REST API, the sarvcrawl Node and Python SDKs, the sarv-kb CLI or the MCP server.

Frequently asked questions

What is the difference between web scraping and web crawling?

Crawling discovers pages by following links. Scraping extracts content or data from a page. A crawl usually scrapes every page it discovers.

What format should scraped content be in for an LLM?

Markdown is a common choice: it keeps headings, lists, tables and links while dropping the markup noise of raw HTML, which saves tokens and helps chunking.

Do I need a headless browser to scrape?

Only when the content is rendered by JavaScript. Server-rendered pages can be scraped with a plain HTTP request, which is faster and cheaper.

What is web scraping?