What Is Main Content Extraction? Boilerplate Removal

Main content extraction in more depth

On a typical site most of the HTML on any page is identical from page to page: the same header, menu, footer and sidebar. If you index or embed whole pages, that repeated text dominates. Searches for “pricing” match every page whose footer links to pricing, and model context fills up with menus.

Main content extraction keeps only the part that makes the page unique. It is what browser “reader mode” does, and it is usually the step between fetching a page and converting it to Markdown.

How it works

Semantic rules: Drop elements that HTML5 marks as page furniture: <nav>, <header>, <footer>, <aside>, plus <script> and <style>. Prefer <main> or <article> if present. Fast and predictable, but only as good as the site’s markup.
Text-density heuristics: Score blocks by text length, link density and punctuation; long blocks with few links are likely content. Used by readability-style libraries.
Cross-page comparison: Text that appears on many pages of the same site is boilerplate. Requires several pages from the site.
Learned models: Classifiers trained to label blocks as content or boilerplate. More robust, more expensive.

Example: a rule-based pass

from bs4 import BeautifulSoup

def main_content(html):
    soup = BeautifulSoup(html, "html.parser")
    for tag in soup(["nav", "header", "footer", "aside", "script", "style"]):
        tag.decompose()
    root = soup.find("main") or soup.find("article") or soup.body
    return str(root)

Common pitfalls

Quiet truncation. The worst failure returns a short, clean, plausible page that is missing the part you wanted. Spot-check the first pages of any new crawl by eye.
Listing and index pages. On a directory, pricing table or search results page, the “chrome” is the content.
Docs sidebars. On some documentation sites the sidebar is the only place the hierarchy exists.
Div soup. Sites that never use semantic elements defeat rule-based extraction, and everything is kept.
Content in `<aside>`. Callouts and warnings are sometimes marked as asides and get dropped.

How SarvCrawl handles it

Scrapes and crawls support onlyMainContent, which strips nav, footer, aside, script and style elements before the page is converted to markdown. Turn it off for listing pages or docs sites where the navigation carries the structure. Pages are rendered in a real browser first, so the extraction runs on the content a visitor actually sees. Test a URL both ways in the playground.

Frequently asked questions

Is main content extraction the same as reader mode?

It is the same idea. Browser reader modes use readability-style heuristics to show only the article; extraction pipelines do the same for programs.

When should I turn main content extraction off?

For listing pages, pricing tables, directories and documentation sites whose sidebar is the only record of structure. In those cases the parts it removes are the content.

Does it remove ads?

Rule-based extraction removes ads only when they sit in elements it strips. Heuristic and learned methods catch more, but none are perfect.

What is main content extraction?