01The one stage nobody measures
Ask anyone running a retrieval system how good their search is and you will get a number. Ask how good their PDF parsing is and you will get a shrug, or a vendor's marketing figure, or — most often — the assumption that it is fine because the output looked fine in the one document somebody opened.
This is a strange gap, because document conversion is the stage with the widest quality variance in the whole pipeline. A born-digital PDF with a clean text layer converts almost perfectly. A two-column academic paper converts with the columns interleaved. A scanned invoice converts to whatever the OCR model believed it saw. These are not different grades of the same thing; they are different processes with different failure modes, and a single "accuracy" number covers all three.
A bad conversion does not throw. It produces confident, well-formatted, plausible markdown that is missing a column.
And it propagates. A mangled table gets chunked, embedded and retrieved exactly as enthusiastically as a good one, and the model at the end has no way to know. The only place to catch it is at conversion.
02What "did the conversion work" actually means
It decomposes into three questions, which need different measurements because they can fail independently.
- Content — is the text still there?
- Compare the markdown against the PDF's own text layer. This is a text-similarity problem and there are standard tools for it.
- Structure — is the shape still there?
- A conversion can preserve every word and lose every heading, table and list. The text score will be near perfect and the document will be unchunkable.
- Readiness — is the output fit to embed?
- Repetition, noise, broken sentences, duplicated lines. A document can pass the first two and still be a bad thing to put in an index.
Most people measure the first, if anything. The second is what determines whether your chunker has anything to work with, and it is cheap to count.
03The content metrics, and what each one catches
Four numbers, each blind to something the others see.
| Metric | Measures | Catches |
|---|---|---|
| ROUGE-1 | Unigram overlap with the source text | Dropped content — a missing column, a skipped page |
| ROUGE-2 / ROUGE-3 | Bigram and trigram overlap | Scrambled word order. Interleaved columns score well on ROUGE-1 and badly here. |
| Jaccard | Vocabulary set overlap, order-free | Substitution — OCR turning "rn" into "m" across a document |
| Length fidelity | Output length against input length | Truncation and duplication, the two failures that are obvious once measured and invisible otherwise |
The ROUGE-1 versus ROUGE-2 gap is the most diagnostic single signal in the set. A two-column paper converted by a parser that reads straight across the page keeps every word — ROUGE-1 near 1.0 — while producing sentences that alternate between two unrelated arguments. ROUGE-2 collapses. One number looks excellent and the document is unreadable.
To that we add vocabulary coverage: how much of the source's distinct vocabulary survived. It is the metric that catches selective loss — a conversion that dropped exactly the table cells, which are a small fraction of the word count and a large fraction of the meaning.
04Structure and readiness
Structure is a counting problem
Headings, bullets, code blocks, links, tables. Count them in the output. A 40-page specification that converted to zero headings and zero tables did not convert; it transcribed. Every downstream stage — chunking especially — depends on structure existing, so this number is a direct predictor of retrieval quality even when the text score is perfect.
Readiness is about what embedding will do with it
A cluster of signals about whether the output is a good thing to put in an index:
- Lexical diversity — a low score usually means repeated headers and footers, once per page, now forty times in one document.
- Named-entity coverage — proper nouns are the highest-value tokens for retrieval and the first casualty of bad OCR.
- Noise level — the ligature artefacts, stray glyphs and hyphenation debris that mark a rough extraction.
- Duplicate line ratio — page furniture, repeated verbatim.
- Average sentence length and its variance — uniform, short sentences are a signature of a parser that broke on line endings rather than on punctuation.
That last one is a good example of why a composite score is not enough. A document whose sentences are all exactly one line long has been split on newlines, and no amount of good ROUGE will make its chunks meaningful.
05The problem that breaks all of the above
Every metric in this post compares the output to the PDF's text layer.
A scanned PDF has no text layer.
This is obvious once stated and it is a genuine trap, because the comparison does not error — it succeeds, against an empty string, and returns zero. In the first version of our audit, every scanned document failed, which is exactly as useful as no audit at all: the population you most need to check is the population the check cannot see.
Scoring OCR output against a non-existent text layer does not measure OCR quality. It measures whether the document was scanned.
So scanned documents take a separate path and are judged intrinsically — on properties of the output alone, with no reference to compare against. Does it read like language? Is the vocabulary plausible? Are there runs of characters no word contains? Is the line structure consistent? None of that can tell you a name was misread. All of it can tell you a page came out as noise, and the difference between "this page is garbage" and "this page is fine" is most of the value.
The report says which mode it ran in, because a content score means two different things depending on the answer, and a dashboard that shows them in one column is lying.
06Running the audit
It is an endpoint on the job. Upload a PDF, and when the job completes ask for its audit:
Some deliberate limits, worth stating plainly:
- Upload jobs only, and only completed ones. A crawled HTML page has no original to compare against — this is a conversion audit, not a content one.
- PDF only, up to 200 MB.
- It runs as a native Rust addon in a worker thread with a 120-second timeout. Text-similarity metrics over a 200-page document are genuinely expensive, and the thread boundary is there so a pathological document stalls its own audit rather than the API.
The report also carries a token ratio — PDF tokens in, markdown tokens out. That is the cost line for everything downstream, and an unexpected ratio is often the first sign that something went wrong: a number far below 1 means content was dropped, far above it means something was duplicated.
07What to do with a bad score
The point is not a dashboard. It is a gate.
- Audit a sample before ingesting a corpus. Twenty documents out of two thousand will tell you whether the other 1,980 are worth the tokens.
- Route the failures, do not drop them. A low structure score often means the document wants a different parser or an OCR pass, not exclusion.
- Keep the score on the document. When a retrieval answer turns out to be wrong six months from now, the first useful question is whether the source was well converted, and that is only answerable if somebody wrote it down at ingest.
- Watch the distribution, not the mean. A corpus averaging 0.85 with a long tail at 0.3 is a corpus with a few hundred documents that are actively harmful in an index.
That last one is the real argument for measuring at all. Bad conversions do not lower the average quality of your answers by a little. They produce a small number of confidently wrong ones, and those are the answers people remember.
What happens to a well-converted document next is chunking markdown without an LLM; the wider pipeline is in web scraping for AI in 2026.