Enterprise · for AI companies
Training-grade data from the web and every PDF you own.
Crawl any site and parse documents from any field, scans included, into clean markdown with a quality score on every file. Then export it as a dataset your models can train on.
corpus.jsonlml verdict: ready- text PDFGovernment report, two footnotes99.8 content
"markdown": "NASA Appropriations and Authorizations: A Fact Sheet…"
"source": "NASA Appropriations and Authorizations: A Fact Sheet, page 2 — Congressional Research Service report R43419, U.S. government work, public domain"
- scan · OCRTypewritten memo, 195584.3 ml readiness
"markdown": "MEMORANDUM FOR…"
"source": "Memorandum from President Eisenhower for the Secretary of Defense, 12 April 1955 — U.S. National Archives (NARA 186497), public domain"
From source to dataset
Collected, parsed, verified, delivered.
One pipeline takes raw pages and documents to records you can train on, and checks every one on the way.
- [ 01 / 04 ]·Collect
Every page of every source
Crawl whole sites with depth, page limits and path rules, rendered in a real browser so JavaScript-heavy pages come back complete. Upload document archives in bulk alongside them.
- [ 02 / 04 ]·Parse
Any document, any layout
PDF, Office, EPUB, images and audio turn into clean markdown. Scans with no text layer go through OCR, with reading order, headings and footnotes kept.
- [ 03 / 04 ]·Verify
A quality score on every file
Each conversion is audited against its source for content fidelity, structure and ML readiness, with noise and duplicate lines measured. Weak files are flagged before they reach your dataset.
- [ 04 / 04 ]·Deliver
Ready to train on
Stream the corpus as JSONL through the API, with the markdown, source URL and metadata on every record. Or download the whole knowledge base, or its chunks, as a ZIP from the console.
Any field
Built for the documents your domain runs on.
Dense tables, footnotes, two-column layouts and decades-old scans. The documents that matter in your domain are the hard ones.
Legal
Contracts, case law, filings and regulations
Finance
Annual reports, filings, statements and research notes
Healthcare and life sciences
Papers, clinical guidelines and drug labels
Government and policy
Reports, scanned archives and public records
Research and education
Journals, theses, textbooks and course material
Engineering and manufacturing
Manuals, spec sheets, standards and datasheets
15 ms
p99 API response at peak
0
dropped requests at peak
< 1 s
per page, fetched to markdown
The Enterprise plan
Your infrastructure, your terms.
Everything in our plans, plus the controls a data team needs.
Runs in your VPC
Self-host the full stack with Compose on one host or Swarm across a cluster. Your data stays in your own Postgres and object storage.
Your concurrency, your limits
Custom concurrency and rate limits sized to your corpus, from a pilot to a full crawl of the web you need.
Tokens on your terms
Committed volume at a fixed rate, or bring your own GPUs for parsing and embedding.
SSO and data residency
Single sign-on for your team and org-level control over where data lives.
SLA and a named account manager
A service level in writing and one person who knows your pipeline.
Migration help
We help move an existing corpus in, so you start from what you already have.
Before you talk to us
What data teams ask first.
Do you train models on our data?
No. There is no training step anywhere in the pipeline. What you collect is yours: it stays in your knowledge bases, and on Enterprise it can stay inside your own infrastructure.
Which file types can you process?
Web pages, PDF, Office documents, EPUB, images and audio. Scanned PDFs and photos with no text layer go through OCR, and you can set the OCR language per upload.
How do we know the data is clean?
Every converted file gets an audit: ROUGE, Jaccard and vocabulary coverage against the source, a structure score, and an ML-readiness score that measures noise and duplicate lines. You can filter out anything below your bar before export.
What does the output look like?
One JSON record per page or document, with the markdown, the source URL, the title and the rest of the metadata. Stream it as JSONL from the API, the CLI or the SDKs, or download the knowledge base and its chunks as ZIP files from the console.
Can it run in our own cloud?
Yes. Enterprise ships the full stack as a Compose file for a single host or a Swarm bundle for a cluster, behind one nginx front door.
Tell us what your models need to read.
Send us a sample of your hardest documents and we’ll show you the dataset that comes back.