Hybrid search in production: fusing BM25 and vector KNN with RRF

The tempting way to combine two result lists is to normalise both scores and add them. Do not. A BM25 score and a cosine similarity are not the same kind of number, they do not share a range, and the range BM25 produces shifts with the corpus and the query length. Any normalisation you pick is a hyperparameter you now have to tune per corpus, and it will drift.

Reciprocal Rank Fusion throws the scores away and keeps only the ranks:

// Rank-based fusion: 60 is the constant from the original RRF paper,
// and the value Elasticsearch uses as its own default.
const RRF_RANK_CONSTANT = 60

function fuse(legs) {
  const scores = new Map()

  for (const { name, hits, weight } of legs) {
    hits.forEach((hit, rank) => {
      const prev = scores.get(hit.id) ?? { hit, rrf: 0, via: [] }
      prev.rrf += weight / (RRF_RANK_CONSTANT + rank + 1)
      prev.via.push(name)
      scores.set(hit.id, prev)
    })
  }

  return [...scores.values()].sort((a, b) => b.rrf - a.rrf)
}

That is the whole algorithm. A document at rank 0 in one leg contributes 1/61; at rank 9 it contributes 1/70. A document that appears near the top of both legs accumulates from both and beats a document that only one leg loved — which is exactly the behaviour you want, and it is why fusion suppresses the semantic noise that plagues a pure vector index.

The constant is a flattener, not a tuning knob

People reach for k first when hybrid results look wrong. It is almost always the wrong knob. k controls how quickly the contribution decays down the list: a small k makes rank 1 enormously more valuable than rank 5, a large k flattens everything toward equality. At 60, the gap between rank 0 and rank 9 is about 15% — deliberately gentle, because rank positions from two incomparable systems are themselves noisy and over-trusting position 1 from a leg that happened to get lucky is how you get a confidently wrong answer.

Elasticsearch has an rrf retriever that will do all of the above server-side. We do not use it, for a boring commercial reason: it is not in the Basic licence tier. Anyone self-hosting the stack — and self-hosting is a first-class path here, not a formality — would hit a licence wall on the single most important query in the product.

So the two legs go out as one msearch and the fusion happens in Node:

{}
{ "size": 200, "query": { "bool": { "must": [ { "multi_match": {
      "query": "how do I cancel a running crawl",
      "fields": ["search_terms^4", "text^3", "title^2", "section"],
      "minimum_should_match": "50%",
      "fuzziness": "AUTO"
} } ], "filter": [ { "range": { "token_count": { "gte": 100 } } } ] } } }
{}
{ "size": 200, "knn": { "field": "embedding", "k": 200,
    "num_candidates": 1600, "query_vector": [ /* 1024 floats */ ] } }

One round trip, two result sets, fusion in memory. The cost is a few milliseconds of CPU on a list of at most 400 hits; the benefit is that the retrieval path behaves identically whether the cluster is ours or yours.

Each leg is allowed to fail on its own

A hybrid search that returns nothing because one leg threw is worse than a single-leg search. If the KNN leg errors — a malformed vector, an embedding service timeout, a shard that is briefly unavailable — the error is logged and the BM25 leg still answers, and vice versa. Degrading to keyword-only is a bad day; returning a 500 to somebody's chatbot is an incident.

The same principle runs one layer down. Embedding happens inline during chunking, and a chunk whose embedding call fails is still indexed — written with embedding_status: "failed" and the vector field omitted rather than null, because an Elasticsearch dense_vector cannot hold null. That chunk stays findable by keyword and is simply invisible to KNN. A corpus with 3% failed embeddings degrades; it does not break.

Fusing two legs at equal weight treats "ECONNRESET" and "why do long crawls stall overnight" as the same kind of question. They are not. The first is a literal — the user is holding a string and wants the document containing it. The second is a description, where the words are a gesture at a meaning.

So before fusing, the query is classified. The heuristic is deliberately crude, because a clever classifier is one more thing to be wrong in a way nobody can debug:

// Three cheap signals. Any one of them means "the user is holding a
// literal", which is the case the vector leg is worst at.
const terms = query.trim().split(/\s+/)

const isLexical =
  terms.length <= 3 ||                 // short queries are lookups
  /["']/.test(query) ||                // quoting is an explicit ask
  /\b\w*\d\w*\b/.test(query)          // ids, versions, dates, error codes

const [wKnn, wBm25] = isLexical
  ? [0.6, 1.0]   // trust the literal
  : [1.0, 0.7]   // trust the meaning

Note that neither weight ever goes to zero. A lexical query still gets vector results, ranked lower — because sarv_sk_ appearing in a page about key rotation is genuinely useful even when the exact-match page exists. Switching a leg off entirely is how you build a search that is excellent on the queries you imagined and useless on the rest.

Nobody has ever had a production problem with the BM25 formula. The problems are all in which documents are allowed to be candidates in the first place.

Partial matching, or one stopword kills the query

The default behaviour of a multi_match is more conjunctive than people expect. A ten-word question in which one word does not appear anywhere in the corpus returns zero BM25 hits. Not few — zero. And because the vector leg still returns its 200, fusion happily produces results and nobody notices that half the system silently dropped out. This is the single most common hybrid-search bug and it is invisible from the outside.

The fix is minimum_should_match, scaled to the query:

// A two-word query means both words. A ten-word question means
// "most of it" — the user is describing, not enumerating.
const minimumShouldMatch = terms.length <= 2 ? '100%' : '50%'

// Fuzziness helps a typo in a word and destroys an identifier.
// Anything carrying a digit or a separator is matched exactly.
const fuzziness = /[\d_\-./]/.test(term) ? 0 : 'AUTO'

The fuzziness gate matters more than it looks. AUTO on a word of four or more characters is a good deal — it forgives recieve for receive. AUTO on v2.11.162, kb_x28mQeny3v or 2026-09-07 is a disaster: it matches neighbouring versions, neighbouring ids and neighbouring dates, and those are precisely the queries where being off by one is worse than returning nothing.

Field boosts, and one field that is not the text

The lexical leg searches four fields at search_terms^4, text^3, title^2, section. Three of those are obvious. search_terms is not: it is a curated field the chunking worker writes, holding the page title, the section heading, any headings inside the chunk, and — the part that earns its keep — the column headers of any table in the chunk.

A pricing table's body is a grid of numbers that BM25 can do nothing with. Its headers say "plan", "tokens", "concurrency". Hoisting those into a boosted field is what makes a table retrievable by the words a person would use to look for it.

The mistake with the vector leg is asking it for the number of results you want. If the caller wants 10 and you run KNN with k: 10, fusion has ten candidates to work with and no room to promote anything. The leg should return a pool, and the pool should be much larger than the answer.

// Both legs fetch a pool, not a page. Fusion, the filters and the
// diversity pass all need slack to work with.
const poolSize = Math.min(200, Math.max(50, perPage * 5))

const knn = {
  field: 'embedding',
  k: poolSize,
  // HNSW explores num_candidates per shard and keeps the best k.
  // Under-setting this is where recall quietly disappears.
  num_candidates: Math.max(500, poolSize * 8),
  query_vector: vector,
}

num_candidates is the one to watch. HNSW is an approximate index: it walks a graph and returns the best k of however many nodes it bothered to visit. Set the candidate count too low and recall drops in a way that never appears as an error — you simply stop seeing documents that are in the index, and no log line tells you.

The vectors themselves are 1024-dimensional FP32, produced by a model served on Triton and stored on the chunk beside the text it came from, with cosine similarity. The dimension is worth stating because it is a commitment: changing it means reindexing the corpus, so it is the number to be sure about before the first million chunks go in.

This one is specific to retrieval over crawled content, and it took a while to find.

Crawl a large site and you will collect error pages. Not many as a fraction, but hundreds in absolute terms: soft 404s, maintenance notices, rate-limit interstitials, empty templates. Each one is short, generic, and — this is the problem — semantically central. A page that says nothing in particular embeds near the middle of the space, which makes it a plausible neighbour for almost any query.

A page that says nothing embeds near everything. Under a 50-token floor, crawled error pages landed at 66 to 70 tokens, passed the filter, and turned up as a mediocre-but-present result for queries they had nothing to do with.

The floor is now 100 tokens, plus an explicit exclusion for chunks whose text is an Internal Server Error phrase. Both are blunt. Both are correct: a 90-token chunk of real prose is a sentence and a half, and losing it costs less than letting a wall of soft-404s sit in the candidate pool of every query in the corpus.

"filter": [
  { "range": { "token_count": { "gte": 100 } } }
],
"must_not": [
  { "match_phrase": { "text": "Internal Server Error" } }
]

The general lesson is that retrieval quality over crawled data is mostly an ingestion problem wearing a search costume. Before tuning weights, look at what is actually in the index.

Chunks from one document are similar to each other by construction — same vocabulary, same subject, often overlapping text. So the top of a fused list has a strong tendency to be five chunks of one page, which is the least useful possible answer: it gives a model one source and the illusion of five.

After fusing and before truncating, the list is passed through a per-page cap. At most two chunks from any one page when the caller asked for five or fewer, and three otherwise. Demoted chunks are not deleted; they backfill from below, so a query that genuinely only has one relevant document still returns something.

This is the cheapest quality win in the whole pipeline. It costs one pass over a list of a few hundred items and it changes what a downstream model can do, because the difference between one source and four is the difference between paraphrasing and corroborating.

A nearest-neighbour search always returns neighbours. Ask a corpus of crawl documentation about the offside rule and it will hand back the ten least-distant chunks it has, with no indication anywhere in the response that all ten are irrelevant. If an agent is on the other end of that call, it will now answer a football question from your API docs.

So the response carries a flag. If nothing in the fused list matched on the lexical leg at all, and the best cosine similarity is below 0.25, the result sets low_confidence: true.

curl -s -H "x-api-key: $SARV_KEY" \
  "https://crawl.sarv.com/api/kb/$KB_ID/search/embed?q=what+is+the+offside+rule&size=5"

The response also reports each leg separately — knn_score, bm25_score, match_via — rather than a single fused number. That is deliberate. A caller debugging a bad answer needs to know which leg produced it, and "the hybrid score was 0.016" tells them nothing. A result that matched only on KNN with a mediocre cosine looks completely different from one both legs agreed on, and the response should say so.

The flag does not make the system smarter. It makes it able to decline — which, for anything feeding a language model, is the more valuable property.

Two things are conspicuously absent from the above, and it is worth being straight about why.

A reranker

The standard next move is a three-stage pipeline: retrieve ~100 candidates cheaply, pass them to a cross-encoder, keep the top 5–10. The published numbers are good — Anthropic reports contextual retrieval cutting failed retrievals by 49%, and 67% when combined with reranking. The pool is already sized for it; poolSize of 200 is a reranker's input in all but name.

What stops it today is latency and cost. A cross-encoder pass over 100 chunks is a second model in the hot path of every search, and search here is synchronous — the SDKs return results rather than a job id. That is a deliberate product property and a reranker has to fit inside it, not break it.

Contextual chunk embeddings

The other half of Anthropic's result is prepending a short LLM-written summary of the document to each chunk before embedding it, so a chunk that says "this dropped to 12%" carries the knowledge of what "this" was. It is the most promising accuracy work available and it has an obvious cost: an LLM call per chunk at ingest, against a pipeline whose chunking stage currently has no LLM dependency at all and is the better for it.

The honest summary is that the fusion described here is table stakes, and the interesting work is upstream of it. If you want that half of the story, the companion post on chunking markdown without an LLM covers what the chunks look like before any of this runs, and the pillar guide puts both in the context of the full pipeline.