HTML parsing in more depth
An HTML file is a string. To find a heading, a table or every link, code needs structure: which elements exist, which contain which, and what attributes they carry. A parser reads the string, splits it into tokens such as start tags, end tags and text, and builds the DOM tree from them.
Unlike XML, HTML is forgiving by design. Unclosed tags, misnested elements and stray text are all common, and the HTML standard defines exactly how a parser must recover from each of them. That is why a browser shows the same page whatever state the markup is in, and why a standards-compliant parser gives the same tree as a browser.
From markup to tree
A compliant parser closes the first <li> when the second opens, closes <b> and the list at </ul>, and wraps everything in the <html>, <head> and <body> elements the source never wrote. The result is a clean tree you can query.
| Language | Common parsers |
|---|---|
| Python | html.parser (standard library), lxml, html5lib; BeautifulSoup wraps any of them |
| JavaScript | parse5 (spec-compliant), cheerio (jQuery-style API), the browser’s own DOMParser |
| Go | golang.org/x/net/html |
Raw HTML versus rendered HTML
Parsing only sees what it is given. On a page that builds its content with JavaScript, the HTML from a plain HTTP request may be an empty shell. JavaScript rendering runs the page in a headless browser first, and the rendered DOM is what gets parsed.
Once a page is parsed, you query it with CSS selectors or XPath, walk it to collect links, or convert it to markdown for a language model.
Common pitfalls
- Parsing HTML with regular expressions. Nesting, attributes in any order and broken markup defeat them quickly. Use a parser.
- Different parsers, different trees. Lenient parsers do not all repair broken markup the same way; a spec-compliant parser matches the browser.
- Parsing the unrendered page. If the content is missing, check whether it is added by JavaScript.
- Ignoring encoding. Decoding bytes with the wrong character set garbles text before the parser even starts.
How SarvCrawl handles it
SarvCrawl renders pages in headless Chromium, falling back to a plain HTTP fetch, and waits for the network to go idle before reading the DOM. It then strips navigation, footers, asides, scripts and styles and keeps the main content, which is returned as markdown, HTML, a list of links or a JSON page object with metadata and the status code. There is no option to choose which elements to keep; if you need that, request html and parse it yourself. The free HTML to markdown tool shows the conversion on any snippet.
Frequently asked questions
What is the DOM?
The Document Object Model is the tree of elements a parser builds from HTML. Browsers render it, scripts change it, and scrapers query it.
Why not parse HTML with regex?
HTML is nested and often malformed, and regular expressions cannot track nesting reliably. A parser handles both and gives you a tree to query.
Which HTML parser should I use?
For scraping, a spec-compliant parser such as html5lib or parse5 gives the same tree as a browser. lxml is faster and fine for well-formed pages.
Why is the content I see in the browser missing from the parsed HTML?
It is probably inserted by JavaScript after load. Render the page in a headless browser and parse the rendered DOM.