What reranking is
First-stage search has to be fast over millions of chunks, so it compares cheap representations: term statistics in BM25, precomputed vectors in semantic search. Neither reads the query and the passage together, so the order of the top results is approximate.
Reranking accepts that and fixes it on a small set. Take the top few dozen candidates, score each against the query with a model that actually reads both, and sort by that score. The expensive model only ever sees a short list.
Two-stage retrieval
- Retrieve a candidate pool with keyword, vector or hybrid search. Optimise for recall: the right passage must be somewhere in the pool.
- Rerank the pool with a relevance model. Optimise for precision: the right passage should be first.
- Cut to the handful of results the application uses.
A reranker can only reorder what it is given. If first-stage retrieval missed the passage, reranking cannot recover it, which is why the pool should be several times larger than the final result count.
Kinds of reranker
| Model | How it scores | Trade-off |
|---|---|---|
| Bi-encoder (first stage) | Embeds query and passage separately, compares vectors | Fast, precomputable, less precise |
| Cross-encoder | Reads query and passage as one input, outputs a relevance score | More accurate, one model call per pair |
| LLM reranker | A language model judges relevance, often on a small scale | Handles nuanced questions, slowest and costliest |
Cross-encoders are the classic choice: small models trained on query-passage relevance pairs. LLM rerankers trade cost for flexibility, since a general model can judge relevance for questions no training set anticipated.
Common pitfalls
- Reranking too few candidates. Reordering a top 5 changes little. The gain comes from promoting something ranked 20th.
- Reranking too many. Cost and latency grow with every candidate. A few dozen is a common range.
- No fallback. If the reranker times out, return the first-stage order rather than an error.
- Expecting it to fix recall. It cannot surface a passage the first stage did not retrieve.
- Paying for it on every query. Short exact lookups often rank correctly already. Make reranking optional.
How SarvCrawl handles it
Knowledge-base smart search, POST /api/kb/:id/search/smart, is two-stage. The first stage runs vector search and BM25 for each sub-query and fuses them with reciprocal rank fusion. The second is an optional LLM rerank, on by default, that scores the top 40 results from 0 to 3 for relevance to the input. Pass "rerank": false to skip it and get the fused order.
Hybrid search (GET /api/kb/:id/search/embed) has no rerank stage; it returns the RRF-fused order with per-leg scores. Knowledge-base search is on paid plans. Details are in the KB search docs and hybrid search in production.
Frequently asked questions
What is the difference between a bi-encoder and a cross-encoder?
A bi-encoder embeds the query and the passage separately and compares the vectors, so passage vectors can be computed in advance. A cross-encoder reads both together and outputs a score, which is more accurate but has to run for every pair at query time.
How many results should be reranked?
Enough that the right passage is likely in the pool, typically a few dozen. Too few gives the reranker nothing to promote; too many adds latency and cost for little gain.
Is reranking worth the latency?
For RAG and question answering, where the top few passages decide the answer, it often is. For short exact lookups that already rank correctly, it adds cost without changing the result, so make it optional.
Can reranking replace hybrid search?
No. It reorders candidates but cannot find passages the first stage missed. Hybrid search improves recall; reranking improves the order of what was recalled.