What the audit measures
Every completed PDF upload can be scored against its own text layer by GET /api/jobs/:id/audit. The report has three parts — content, structure and ML readiness — plus a token count in and out, an overall score and a verdict. The full reasoning is in scoring PDF conversion quality.
| Metric | Measures | Catches |
|---|---|---|
| ROUGE-1 | Unigram overlap with the source text | Dropped content: a missing column, a skipped page |
| ROUGE-2 / ROUGE-3 | Bigram and trigram overlap | Scrambled word order, such as interleaved columns |
| Jaccard | Vocabulary set overlap, order-free | Substitution, such as OCR reading "rn" as "m" |
| Vocabulary coverage | Share of the source's distinct words that survived | Selective loss, such as dropped table cells |
| Length fidelity | Output length against input length | Truncation and duplication |
Text PDF: a government report page
Page 2 of Congressional Research Service report R43419, a page of prose with two footnotes. Output fetched from the production API on 2026-10-06 and checked by hand against the source. Original, output and audit.
| Score | Value |
|---|---|
| Tokens in → out | 570 → 570 (ratio 1) |
| ROUGE-1 F1 | 100 |
| ROUGE-2 F1 | 99.5 |
| ROUGE-3 F1 | 99.1 |
| Jaccard | 100 |
| Vocabulary coverage | 100 |
| Length fidelity | 100 |
| Content score | 99.8 |
| Structure score | 0 (no headings, lists or tables in the output) |
| ML readiness | 83 (lexical diversity 39.8, named-entity coverage 90.4, noise 0) |
| Overall / verdict | 74 / ready |
Every word of the text layer is present and the reading order holds, with footnotes at the end. The overall score is held down by structure: the page has no headings or tables, so there was nothing for the structure metric to count.
- Miss: the drop cap splits off its letter — "C ongressional".
- Miss: superscripts flatten, so footnote markers read "years.1".
Scanned PDF: a typewritten memo from 1955
A memorandum from President Eisenhower to the Secretary of Defense, scanned with no text layer and read by OCR. Original, output and audit.
| Score | Value |
|---|---|
| Mode | OCR (no text layer to compare against) |
| Tokens in → out | 0 → 270 |
| Content metrics | 0 across the board, by design in OCR mode |
| Lexical diversity | 55.2 |
| Noise level / duplicate lines | 0 / 0 |
| Average sentence length | 33.5 words |
| ML readiness | 84.3 |
| Overall / verdict | 75.9 / ready |
Checked by hand: the body text matches the scan word for word, the lettered list (a)–(c) stays a list, line-end hyphens are rejoined and the library stamp is ignored.
- Miss: the date line (April 12, 1955) and the D.D.E. sign-off were dropped.
Intrinsic scoring can tell a page came out as noise. It cannot tell a line is missing: this scan lost its date line and still read "ready".
Image upload: a photographed book page
Page 15 of Alice in Wonderland, uploaded as a JPG. The audit runs on PDFs only, so this one was checked by hand. Original and output.
- Body text matches the page word for word; curly and nested quotes are kept.
- The running header and the asterisk divider are left out.
- Miss: one stray comma — "feeling!", said — that the page does not have.
- Miss: italics come through as plain text.
The numbers chunking and search run on
These are operating settings and the measurements behind them, not a leaderboard. Each is explained in chunking markdown without an LLM or hybrid search in production.
| Setting | Value | Why |
|---|---|---|
| Chunk size | 800 tokens (cl100k_base) | A complete idea, roughly 600 words |
| Chunk overlap | 100 tokens | Answers that straddle a boundary; about 12% duplication |
| Minimum chunk | 50 characters | Below it, a fragment rather than a passage |
| Retrieval floor | 100 tokens | Crawled error pages measured 66 to 70 tokens and passed the old 50-token floor |
| Embeddings | 1024-dim FP32, cosine | Concurrency 8, 30-second timeout per call |
| RRF constant | k = 60 | Rank 0 vs rank 9 differs by about 15% |
| Low-confidence flag | best cosine < 0.25 | Set only when nothing matched on the keyword leg either |
| Audit limits | PDF up to 200 MB, 120 s | Runs in a worker thread so one document cannot stall the API |
Published figures we cite
Third-party results our posts rely on. We did not measure these; they are here so the claims on this site can be traced.
| Result | Figure | Discussed in |
|---|---|---|
| WANDS e-commerce, NDCG | RRF 0.7068 · BM25 0.6983 · KNN 0.6953 · tuned hybrid 0.7497 | Hybrid search in production |
| Contextual retrieval (Anthropic) | 49% fewer failed retrievals; 67% with reranking | Hybrid search in production |
| llms.txt requests | 408 of ~500,000,000 AI bot visits in 90 days | llms.txt, measured |
Run the audit on your own files
Upload a PDF to any knowledge base and, once the job completes, ask for its audit.
Before ingesting a corpus, audit a sample: twenty documents out of two thousand will tell you whether the rest are worth the tokens. Watch the distribution rather than the mean.
Questions
Which files can the conversion audit score?
Completed PDF upload jobs, up to 200 MB. Crawled HTML pages and images have no original text layer to compare against, so they are not audited.
Why do scanned PDFs score zero on ROUGE and Jaccard?
A scan has no text layer, so content metrics have nothing to compare with. The audit switches to OCR mode and judges the output on readiness signals such as lexical diversity, noise and duplicate lines.
Are these results cherry-picked?
They are three public-domain files, published with the unedited output and every miss we found by hand, including a dropped date line that the audit did not catch.