Why PDF parsing is hard
A PDF describes how a page looks, not what it is. Internally it is a list of drawing instructions: place these glyphs at these coordinates in this font. There is often no notion of a paragraph, a heading, a table cell or even a space between words. A parser has to infer all of that from positions and font sizes.
For RAG and search, structure is as important as text. Headings are what a chunker splits on, and a table is only useful if its rows keep their headers. A conversion that keeps every word and loses every heading has transcribed the document, not parsed it.
Quality varies more at this stage than anywhere else in a document pipeline. A born-digital PDF with a clean text layer converts almost perfectly; a two-column paper may come out with its columns interleaved; a scanned invoice converts to whatever OCR believed it saw. These are different processes with different failure modes, not grades of one.
How it works
- Classify each page. If there is a text layer, read it; if the page is an image, run OCR.
- Group glyphs into words, lines and blocks by position.
- Find reading order, including multiple columns, sidebars and footnotes.
- Infer structure: font size and weight suggest headings, indentation and markers suggest lists, aligned runs suggest tables.
- Remove page furniture: running headers, footers and page numbers.
- Emit markdown or JSON.
Common failure modes
- Interleaved columns. A two-column paper read straight across produces sentences that alternate between two arguments. Every word survives, so a word-overlap score looks excellent.
- Flattened tables. Cells come out as a stream of values with no row or column.
- Repeated furniture. The same header and footer forty times in one document.
- Broken lines. Splitting on line endings instead of sentences, which shows up as many short sentences of uniform length.
- Ligatures and hyphenation.
figlyphs and words hyphenated across lines break keyword search.
A bad conversion does not throw. It produces confident, well-formatted, plausible markdown that is missing a column.
Measuring it
For text PDFs, compare output with the text layer. ROUGE-1 catches dropped content; ROUGE-2 drops when word order is scrambled, so a high ROUGE-1 with a low ROUGE-2 is the signature of interleaved columns. Jaccard catches substitution, length fidelity catches truncation and duplication. Also count structure: headings, tables, lists.
Use the scores as a gate, not a dashboard: audit a sample before ingesting a corpus, route failures to a different parser or OCR instead of dropping them, and watch the distribution rather than the average, because a long tail of bad conversions produces a few confidently wrong answers.
How SarvCrawl handles it
SarvCrawl parses uploaded PDFs, Office files and EPUB into markdown, and finds and parses linked documents during a crawl when you ask it to. Scanned PDFs with no text layer go through OCR. The markdown is then chunked on its headings and embedded into your knowledge base.
Each completed PDF upload can be checked with a conversion audit that scores ROUGE-1, Jaccard, vocabulary coverage and length fidelity, plus structure counts and a token ratio. It is described in scoring your own PDF conversion. For single files, use PDF to Markdown or the parse API.
Frequently asked questions
What is the best format to convert a PDF to for AI?
Markdown is a strong default. It keeps headings, lists and tables in a compact form that models read well and that chunkers can split on.
Why does text copied from a PDF come out jumbled?
PDFs store glyph positions, not reading order. Multi-column layouts, sidebars and tables are reassembled by guessing, and naive extraction reads straight across the page.
How do I check if a PDF parsed correctly?
Compare the output with the PDF's own text layer using measures such as ROUGE and Jaccard, and count headings and tables. For scanned PDFs, judge the OCR output on its own readability, because there is no text layer to compare against.