What Are LLM Tokens? Tokenization, Limits and Cost

What a token is

Models do not see letters or words. A tokenizer turns text into a sequence of integer ids from a fixed vocabulary, and the model works on those. Common words are usually one token; rarer words split into pieces; spaces and punctuation are often folded in. tokenization might become token + ization.

Everything practical is measured in tokens: the context window, the price per million input and output tokens, rate limits, and the chunk budget in a RAG system.

The context window is a hard limit on input plus output tokens for one request. Anything that does not fit is cut or rejected, which is why RAG systems send a few retrieved chunks rather than whole documents, and why the size of those chunks is set in tokens.

How tokenization works

Most modern tokenizers use byte-pair encoding (BPE) or a close relative. Training starts from bytes and repeatedly merges the most frequent adjacent pair into a new token until the vocabulary reaches its target size. Frequent strings end up as single tokens; unusual ones stay as several.

Each model family has its own tokenizer, so the same text has different counts on different models. For English prose a rough rule is about four characters, or three-quarters of a word, per token. Code, non-English text, numbers and identifiers usually take more.

import tiktoken

enc = tiktoken.get_encoding("cl100k_base")
ids = enc.encode("Retrieval-augmented generation grounds answers.")
print(len(ids))                       # token count
print([enc.decode([i]) for i in ids]) # the pieces

Why it matters, and common mistakes

Feeding models HTML. Navigation, scripts and markup are all tokens. One commonly cited blog post measures 16,180 tokens as HTML and 3,150 as markdown, more than five to one.
Counting characters for chunk sizes. A 2,000-character chunk can be very different token counts in prose and in code. Count with the tokenizer your model uses.
Assuming counts transfer between models. Switching model can change cost and whether text fits.
Filling the context window. Fitting is not the same as helping. More retrieved text costs more and can dilute the passage that matters.
Forgetting output tokens. They are usually priced higher than input tokens.
Ignoring the prompt around the content. System instructions, tool descriptions and conversation history all count against the same window and the same bill.

Markdown is not a nicer format for humans. It is a five-times-cheaper format for models.

How SarvCrawl handles it

SarvCrawl converts crawled pages and uploaded documents to markdown before anything else happens, which removes most of the token overhead of HTML. Its chunker counts with tiktoken cl100k_base and uses an 800-token budget with 100 tokens of overlap; search excludes chunks under 100 tokens.

Usage is metered in tokens, at roughly 500 tokens per page of text, and new accounts get 1 billion tokens free with no card. The PDF conversion audit reports a token ratio, PDF tokens in against markdown tokens out, where an unexpected ratio is often the first sign that content was dropped or duplicated. Try a page at HTML to Markdown.

Frequently asked questions

How many tokens is a word?

For English prose, roughly 1.3 tokens per word on average, or about four characters per token. Code, numbers and other languages often use more. Count with your model's tokenizer for exact figures.

How many tokens is a page of text?

Around 500 tokens for a typical page of prose. Dense text, tables and code can be considerably more.

Why do different models give different token counts?

Each model family trains its own tokenizer with its own vocabulary, so the same text splits into different pieces.

How do I reduce token usage when feeding web pages to an LLM?

Convert HTML to markdown and extract the main content first, so navigation, scripts and markup are not sent. Then retrieve only the relevant chunks rather than whole pages.

What are LLM tokens?