Rerankers.
A reranker reorders what your retriever already found, which means it raises precision and cannot raise recall — if the passage that answers the question never made the candidate list, no reranker will put it there. So the number that decides whether a reranker is worth its latency is not nDCG, it is the gap between your recall at 5 and your recall at 50; that gap is the entire available gain, and you can measure it in an afternoon before you buy anything.
What a reranker computes that a retriever structurally cannot.
The two stages look like "search, then better search". They are different computations, and the difference is where the query meets the document.
- A dense retriever never shows the document the query. Each document is embedded once at index time; the query is embedded at request time; relevance is the similarity of two vectors that were produced independently. That independence is exactly what makes the index possible — you can precompute millions of document vectors because none of them depends on a question nobody has asked yet.
- A cross-encoder reranker concatenates the query and one document into a single forward pass and reads out a relevance score. Attention runs over both texts together, so the model can see that this clause answers that qualifier. Nothing about it can be precomputed, because the input does not exist until the query arrives.
- The cost shapes follow from that. Retrieval cost is roughly flat in corpus size — an approximate-nearest-neighbour lookup over a prebuilt index. Reranking cost is linear in the number of candidates you score, paid at request time, every time.
This is the only stage of a retrieval pipeline whose bill is denominated in candidates rather than queries. It is also the only stage that gets to read the query and the document at the same time. Those two facts are the same fact, and the two-stage retrieve-then-rerank shape exists to buy the second while rationing the first.
Precision is for sale; recall is not.
A reranker's input is a list. Whatever is missing from that list is missing from the answer, so the first stage's recall@k is a hard ceiling on the whole system. This makes the buying decision unusually measurable — you need one labelled set of queries and three numbers.
- recall@5 vs recall@50 is your headroom. If retrieving 50 candidates finds the right passage 88% of the time but your top 5 only contain it 64% of the time, a perfect reranker is worth 24 points and a good one most of that. If the two numbers are within a point or two, your retriever already ranks well and a reranker buys latency for nothing.
- If recall@50 itself is low, a reranker is the wrong purchase. A ceiling of 0.6 is a parsing, chunking, or query problem — the fixes live in document parsing and query understanding, and no amount of reordering substitutes for either.
- Hybrid retrieval and reranking solve different halves. Fusing BM25 with dense search raises recall by catching the exact-term matches embeddings miss; the reranker then sorts the enlarged pile. They compose, and the order matters: widen first, sort second — see hybrid search and reranking.
- Watch the metric you report. nDCG@10 rewards reordering and will move when a reranker lands even if the answer rate does not. Keep an end-to-end answer-correctness number next to it, the way evaluating RAG insists.
Three implementations, three different bills.
"Reranker" names a job, not an architecture. The three families differ by how much work they are allowed to do ahead of time, and that single axis predicts their latency, their index size, and their price.
- Cross-encoder — the default. One forward pass per (query, document) pair, usually a small encoder. A MiniLM-class model scores a batch of ~128 passages in tens of milliseconds on a datacentre GPU; a larger BGE-class model on the same batch takes several hundred. Hosted endpoints price it by depth rather than by query — Cohere's rerank API bills a search unit per query of up to 100 documents at roughly $2 per thousand — which is the clearest possible signal that depth is a budget line. These models are small enough to self-host, and reranking is one of the jobs where a local model is genuinely competitive; see small and local models.
- Late interaction — precompute most of it. ColBERT-style models embed every token at index time and compute a cheap per-token maximum-similarity at query time. You get much of the cross-encoder's sensitivity at a fraction of the request-time cost, and you pay for it in index size, which can run an order of magnitude above a single-vector index. The trade-off is worked through in late interaction retrieval.
- LLM listwise reranking — highest ceiling, worst economics. Put the candidates in a prompt and ask the model for an ordering. It can outscore a cross-encoder on queries that need reasoning across passages, and it costs seconds of latency plus a token bill that scales with candidates times their length. Published comparisons put cross-encoders ahead on cost-effectiveness while conceding listwise a slightly higher peak. Reserve it for the queries where a human would also have had to read all ten passages.
Swapping reranker models is cheap in a way swapping embedding models is not: a reranker holds no state, so changing it is a deploy, while changing an embedding model invalidates the whole index and triggers a reindex. That asymmetry is a good reason to put your experimentation budget on the reranker and leave the embedder alone.
Depth is the knob, and almost nobody sets it deliberately.
Having bought a reranker, the one parameter that moves quality is how many candidates you hand it. Ablations across reranker sizes and retrievers show nDCG@10 still climbing as depth goes 10 → 20 → 50, with 20 chosen as the usual cost compromise rather than the quality optimum.
- Reranking 5 to keep 5 is a no-op wearing an upgrade's clothes. Retrieve wide, keep narrow: fetch 50, rerank, pass the top 5 to the model. The gain lives entirely in the candidates the first stage ranked 6th through 50th.
- Set depth from the recall curve, not from a blog post. Pick the smallest k where recall@k flattens. Below that you are capping your own ceiling; above it you are paying per candidate for nothing.
- Depth trades directly against p95. Both latency and cost are linear in k, and the reranker sits on the critical path of every query — which is why this is a cost, quality, latency decision rather than a retrieval one. Reranking 50 candidates with a mid-sized cross-encoder is a second-scale operation on modest hardware.
- Do not plan on caching it. Embeddings cache beautifully because a document's vector is query-independent; rerank scores are keyed on the pair, so a hit needs the same query and the same candidate set. Semantic caching at the answer layer is the better place to recover this cost.
- Re-measure after any index change. New chunk sizes, a new parser, or a refreshed corpus all move the recall curve, and your depth setting was derived from the old one.
Do this before you add a reranker, not after: take 100 labelled queries and compute recall@5, recall@20 and recall@50 against your current index. The spread between recall@5 and recall@50 is the maximum a reranker can return to you. Under about two points, skip it and spend the effort on parsing and query rewriting. Above ten, add a small cross-encoder at depth 50, keep 5, and expect most of the gap. Then read chunking and vector search for the stage that sets the ceiling, and evaluating RAG for the measurement that stops you shipping a metric win as a product win.