What Is BM25? The Keyword Ranking Formula Explained

What BM25 is

BM25 is a ranking function from the probabilistic retrieval work of the 1990s, and it is still the default similarity in Lucene-based engines. It improves on plain TF-IDF in two ways: repeated occurrences of a term give diminishing returns instead of growing without limit, and long documents are not rewarded just for containing more words.

In a vector-first world it remains essential, because it is the part of a search system that matches exact strings: error codes, product names, ids and versions.

It is also cheap. A BM25 query runs against an inverted index, a map from each term to the documents containing it, so it needs no model call at query time and its results are fully explainable: you can see which terms matched and how much each one contributed.

How the formula works

For each query term q, BM25 multiplies two factors and sums them over all terms:

IDF (rarity): Rare terms count more. Lucene uses ln(1 + (N - n + 0.5) / (n + 0.5)), where N is the number of documents and n the number containing the term.
Saturated term frequency: f · (k1 + 1) / (f + k1 · (1 - b + b · len / avglen)), where f is how often the term appears in the document.
k1: How quickly repeats saturate. Elasticsearch defaults to 1.2.
b: How much document length is penalised, from 0 (none) to 1 (full). The default is 0.75.

{
  "query": {
    "multi_match": {
      "query": "rotate api key",
      "fields": ["title^2", "section_heading", "text"],
      "minimum_should_match": "50%"
    }
  }
}

In practice the defaults are rarely the problem, and per-field boosts matter more than k1 and b. A match in a title or heading is usually stronger evidence than a match deep in body text, so search titles and headings as separate, boosted fields.

Common pitfalls

Problems with BM25 in production are almost never in the formula. They are in which documents are allowed to match.

Over-strict matching. If every term must match, a ten-word question with one word missing from the corpus returns zero hits. Require all terms for one- or two-word queries and around half for longer ones.
Fuzziness on identifiers. Typo tolerance helps recieve find receive, and makes v2.11.162 match v2.11.163. Disable it for tokens with digits or separators.
No synonyms. BM25 cannot know that throttled means rate limit. That is what the vector leg of hybrid search is for.
Unsearchable tables. A table body is a grid of numbers. Index the column headers somewhere searchable.

How SarvCrawl handles it

SarvCrawl's keyword search is a BM25 query over four fields: the chunk text, the page title, the section heading, and a boosted search_terms field that holds the page title, headings and the column headers of any table in the chunk. minimum_should_match is 100% for queries of one or two terms and 50% otherwise, and fuzziness is off for any term containing a digit or separator.

GET /api/kb/:id/search is the keyword leg alone; /search/embed fuses it with vector search through reciprocal rank fusion. See the KB search docs and hybrid search in production.

Frequently asked questions

What does BM25 stand for?

Best Matching 25. BM is the name of the family of ranking functions and 25 is the variant number; it is often called Okapi BM25 after the Okapi system where it was first used.

What is the difference between BM25 and TF-IDF?

Both reward frequent and rare terms. BM25 adds saturation, so the tenth occurrence of a word adds little, and length normalisation, so long documents are not favoured just for being long.

Is BM25 still useful with embeddings?

Yes. Embeddings handle meaning but miss exact strings like error codes, ids and version numbers. Production systems usually run BM25 and vector search together.

What is BM25?