Reranking is a second-stage retrieval step that takes an initial set of candidate documents and reorders them by semantic fit to the query using a more powerful but slower cross-encoder model.
Reranking follows a first-stage retrieval pass that quickly returns a broad candidate set from a vector index. A cross-encoder model then scores each candidate against the original query, and the top results replace the original ranking. This two-stage design balances speed and accuracy.
That second pass is where most quality gains come from in production search. First-stage retrieval favors speed; it returns plausible matches using fast approximate nearest-neighbor search. Reranking tolerates fewer candidates because it processes them with a model that reads the query and document together, capturing nuance that single-vector similarity misses.
Think of it like this. Think of a librarian who first pulls every book mentioning travel, then reads each candidate against your actual question and shelves the best matches at the front.
First-stage retrieval fetches a candidate set, typically twenty to a hundred documents, using fast vector search or BM25. Each query-document pair is fed to a cross-encoder that outputs a relevance score. Documents are reordered by score and the top results are returned.
"Reranking replaces retrieval." It is a second stage that depends on solid first-stage recall. "Any cross-encoder works." Domain-specific rerankers outperform general ones on specialized content. "More candidates is better." Reranking cost scales linearly with candidate count.
Improves ranking quality significantly, but adds latency and compute cost per query. Best used when first-stage recall is already acceptable but precision needs a lift, especially with diverse or ambiguous queries.