What Is Document Chunking? Splitting Text for RAG

What chunking is for

Embedding models and prompts both have limits, and a whole 40-page manual is not a useful search result. So documents are cut into passages of a few hundred tokens. Each chunk gets its own vector and its own keyword entry, and search returns chunks, not documents.

That makes chunking the stage that sets the ceiling on retrieval. If the answer to a question is split across two chunks, neither of which is convincing on its own, no ranking function can fix it.

A chunker's job is not to cut text into equal pieces. It is to produce passages that are individually true.

Chunk size is a trade-off. Small chunks are precise but lose the context that made them meaningful, and evidence gets spread across units that are each unconvincing. Large chunks keep context but dilute it: one relevant sentence is outvoted by hundreds of tokens of neighbouring prose, both in the embedding and in the prompt.

How chunking works

Most chunkers combine three ideas:

Split points: Where cuts are allowed: headings first, then paragraphs, then sentences. Fixed character counts are the fallback, not the plan.
Budget: A maximum size in tokens, counted with the same tokenizer the downstream model uses, not in characters.
Overlap: A small window repeated at each boundary, so a sentence that straddles a cut appears whole in at least one chunk.

chunk_provider   = "markdown_router"
chunk_size       = 800    # tiktoken cl100k_base tokens, not characters
chunk_overlap    = 100
min_chunk_chars  = 50     # below this it is a fragment, not a passage

800 tokens is roughly 600 words, about a section of documentation. Anything from 500 to 1,000 is defensible; the split rules matter more than the number. 100 tokens of overlap costs about 12% duplication in the index.

Common pitfalls

Splitting a table mid-row. Rows without their header row are numbers with no meaning. Keep tables whole even if that exceeds the budget.
Splitting a code block. Half a function is not code. If one block must be split, reopen the fence on the continuation so each piece still declares its language.
Character-count splitting. Chunks that start and end mid-sentence are about nothing in particular, and they sit near the middle of embedding space, close to every query.
Treating a record list as prose. A file of 200 support tickets is 200 documents. Recursive splitting produces chunks holding the end of one ticket and the start of the next.
Tiny fragments. A heading with nothing under it, or a 60-token nav fragment, pollutes results.

Quick checks: read twenty chunks at random and ask whether each is understandable alone; search for chunks that begin with a pipe and have no header row; look for a spike in the size distribution at exactly the budget.

How SarvCrawl handles it

Every crawled page and uploaded file is converted to markdown and chunked automatically. The chunker routes on document shape first (record lists get one chunk per record), then splits on the document's own headings with an 800-token budget and 100 tokens of overlap, and never splits a table or a fenced code block. It has no LLM in the loop, so re-ingesting the same document gives the same chunks.

Table column headers are also copied into a boosted keyword field, so a pricing table is findable by the words people search with. Chunks are embedded with contextual chunk embeddings as they are produced. The reasoning, rule by rule, is in chunking markdown without an LLM.

Frequently asked questions

What is the best chunk size for RAG?

There is no single best size. A few hundred to about a thousand tokens works for most text; we use 800 with 100 of overlap. Splitting on headings and keeping tables and code whole matters more than the exact number.

Should chunks overlap?

A small overlap helps when an answer straddles a boundary, because the boundary text then appears in both neighbouring chunks. Around 10 to 15 percent of the chunk size is a common choice.

Should I use an LLM to chunk documents?

It can find good seams, but it is slow, costs a model call per document on every re-ingest, and is non-deterministic. Most chunking failures are structural, such as split tables, and can be avoided by reading the markdown structure directly.

What is document chunking?