Chunking markdown without an LLM: tables, code fences and an 800-token budget

Semantic chunking with an LLM is the fashionable answer and it is genuinely appealing: a model reads the document and decides where the seams are, which is exactly the judgement you want.

Four objections, in the order they bite:

1. 01Cost scales with the corpus, not the queries. An LLM call per document — or worse, per window — across a hundred thousand pages is a real line item, and it recurs on every re-crawl.
2. 02It is the slowest stage in the pipeline. Chunking sits between parsing and embedding, in the hot path of every ingest job. A model call there sets the throughput of the whole system.
3. 03It is non-deterministic. Re-ingest the same document and get different chunk boundaries, which means different embeddings, which means a corpus that quietly changes shape under you and a retrieval regression you cannot reproduce.
4. 04It fails in the interesting cases. The places a naive splitter goes wrong — mid-table, mid-code-block — are structural. You do not need a language model to see a pipe table; you need to look for one.

So our chunking worker has no LLM dependency at all. Token counting is tiktoken with the cl100k_base encoder — the same encoder the downstream model uses, which is the point of counting tokens rather than characters — and every decision after that is structural.

The insight that made this tractable: documents are not all the same kind of thing, and one splitter cannot be right for all of them. A specification, an API reference and a CSV export of support tickets want completely different treatment.

So the first step is not splitting. It is classification.

The record-list branch is the one that surprises people. A document that is a list of two hundred support tickets is not prose with an unusual shape — it is two hundred documents that happen to share a file. Recursive splitting on such a file produces chunks containing the tail of ticket 47 and the head of ticket 48, which is a passage that is not true of anything.

# The whole configuration surface. Deliberately small.
chunk_provider   = "markdown_router"
chunk_size       = 800    # tiktoken cl100k_base, not characters
chunk_overlap    = 100
min_chunk_chars  = 50     # below this it is a fragment, not a passage

1. A table is never split mid-row

This is the most consequential rule in the file. Half a table is not half as useful as a table; it is useless, and it is worse than useless because it looks fine. A chunk containing rows 12 to 19 of a pricing table, with no header row, is eight lines of numbers with no idea what any column means.

So tables are protected as atomic units. If a table exceeds the budget on its own, the budget loses — an oversized chunk that contains a complete table beats two chunks that each contain half of one.

There is a second half to this, which happens at index time rather than chunk time: a table's column headers are hoisted into a separate, boosted lexical field alongside the page title and section headings. A table body is a grid of numbers that keyword search can do nothing with; its headers say "plan", "tokens", "concurrency", and those are the words someone will actually search for.

2. A fenced code block is never split

Same argument, sharper. Half a function is not code. A chunk that starts mid-block is also syntactically broken in a way that confuses a model reading it: it opens inside a construct that never began.

Fenced blocks are kept atomic. In the one case where a single block genuinely exceeds the budget — a long configuration file, a large JSON sample — it is split, and the fence is reopened on the continuation so that each piece is still valid markdown and still declares its language. A model reading chunk two can see it is looking at YAML.

3. Split on the document's own headings, not on a character count

Markdown headings are a real outline, which is the main reason to convert to markdown at ingest in the first place. Splitting on them means a chunk corresponds to a section somebody deliberately wrote as a unit, and inherits a heading that says what it is about.

Character-count splitting produces chunks that begin mid-sentence and end mid-sentence and describe nothing in particular. They embed near the middle of the space — semantically central, specifically about nothing — which makes them plausible neighbours for queries they cannot answer.

Every failure mode here has the same shape: a chunk that looks fine and is not true on its own.

The numbers themselves are the least interesting decisions in the file, and they get the most debate.

800 tokens is roughly 600 words: a section of documentation, three or four paragraphs, a complete idea. Big enough to carry its own context, small enough that a retrieved chunk is mostly signal. Anywhere from 500 to 1,000 is defensible, and the choice matters far less than whether rules 1 to 3 hold.

100 tokens of overlap exists for one reason: an answer that straddles a boundary. Without overlap, a sentence split across two chunks is in neither. With it, the boundary region appears in both and one of them will be retrievable. The cost is about 12% duplication in the index, which is cheap insurance.

The min_chunk_chars floor of 50 catches the tail end of a split — a heading with nothing under it, a stray line. Below that it is not a passage.

Chunking and embedding are one pass rather than two. Each chunk is vectorised as it is produced — 1024 dimensions, FP32, from a model served on Triton — and stored beside the text it came from, with a concurrency of 8 and a 30-second timeout per call.

The interesting part is what happens when that call fails. The chunk is still indexed, marked embedding_status: "failed", with the vector field omitted rather than set to null — an Elasticsearch dense_vector cannot hold a null, so writing one would fail the whole document.

The result is that a chunk with a failed embedding is still findable by keyword and is simply invisible to vector search. A corpus with a bad afternoon degrades; it does not lose documents. That is worth designing for, because the alternative — dropping the chunk — is a silent hole in the index that nothing will ever tell you about.

Four checks, none of which need an evaluation harness:

1. 01Read twenty chunks at random. Can you tell what each one is about without the document around it? If not, that is the bug, and it is upstream of everything else.
2. 02Grep for orphaned table rows. A chunk starting with | and no header row is a split table. There should be none.
3. 03Look at the size distribution. A spike at exactly your budget means the splitter is hitting the limit rather than finding seams — the document structure is not being used.
4. 04Check the short tail. A pile of 60-token chunks is usually boilerplate, nav fragments or error pages that survived parsing, and they will show up in search results for queries they have nothing to do with.

In our console the chunks a job produced are readable directly, which makes check 1 a five-minute job rather than a script. The equivalent over the API is GET /api/kb/:id/pages/:page_id/chunks.

What happens to these chunks next — the two-legged retrieval that queries them, and the filters it applies — is in hybrid search in production. What produced the markdown they were cut from is scraping a website to clean markdown.