Writing
Guides to crawling, parsing and retrieval, honest comparisons of the tools in this market, and write-ups of how our own pipeline works — with the constants we actually run in production rather than a diagram of one.
The complete path from a URL to an answer — crawl, parse, chunk, embed, retrieve — what breaks at each stage, and how the three kinds of tool in this market split the work between you and them.
Read itHTML costs about five times as many tokens as the same page in markdown, and chunks far worse. Here is why, how to do the conversion four different ways, and the two options that decide whether the output is usable.
Read itEight tools, what each one is actually for, the real entry prices, and the question that decides between them — which half of the work you want to keep. Written by one of the eight, and marked where that matters.
Read itThree verbs that sound interchangeable and are not. One fetches a page, one follows links, one only discovers URLs for effectively nothing — and using the third first is the cheapest decision in an ingestion pipeline.
Read itThe tutorial usually stops at the first crawl. The hard parts are the ones after it: deduplicating what you fetched, re-crawling only what changed, and noticing when the source moved without you.
Read itEvery retrieval demo runs one vector index and stops. Production needs two legs and a way to combine them. Here are the constants we actually run — the rank constant, the weights, the floors and the guard that lets the system say it does not know.
Read itChunking is where retrieval quality is decided, and the popular answer — ask a model to do it — is slow, expensive and non-deterministic. Here is a router that reads the document shape instead, and the three rules that matter more than the token budget.
Read itEvery document parser claims high accuracy and none of them show their working. Here is how to measure a conversion against the original, why that method breaks completely on scanned pages, and what to do instead.
Read itFrom 15 September 2026, mixed-use AI crawlers are blocked by default on ad-supported pages. Here is what the change is, what the block rates actually look like, and what a crawler has to do now to keep working.
Read itOut of 500 million AI bot visits in a 90-day window, 408 requested llms.txt. That is the whole story on citations — and it is still worth shipping one, for a completely different reason.
Read it10 posts. If you are starting cold, Web scraping for AI in 2026: turning any site into a searchable knowledge base is the one that frames the rest. Everything they describe is reachable from the API reference.