On retrieval, part 2 of 3

Retrieve, Then Rerank

Keyword search and vector search miss different things. The systems that work run both, then spend real compute deciding which results matter.

In May 2026, a team of researchers ran a simple head-to-head inside the coding agents most of us now use every day. They gave Claude Code, Codex CLI and Gemini CLI, along with a harness of their own, the same 116 questions over the same long conversation histories, and let each agent search with either grep or vector retrieval. With results returned straight into the conversation, grep won in every one of the ten harness and model pairings they tested, by as much as 93.1% to 75.9% for Codex with GPT-5.4 (Sen et al.).

Then they changed one thing. Instead of returning results inline, they wrote them to files the agent had to read. For Codex, grep’s accuracy fell from 93.1% to 55.2%, and vector retrieval came out ahead at 67.2%. Across the ten pairings, vector retrieval now beat grep in five. The authors’ conclusion is the part worth keeping: the way the harness delivers results can “invert or erase the lexical advantage” without any change to the data.

Notice what grep is. It’s keyword search, the oldest kind there is, run in a loop by a model that reads what comes back and decides what to look for next. The agents that dropped their vector databases didn’t stop doing retrieval. They rebuilt a cruder version of the two techniques this essay is about: searching by words as well as meaning, and then spending real intelligence on deciding which results matter.

This is the second of three essays on retrieval. The first ended on the problem with similarity: an embedding measures how much a question resembles a document, not whether the document answers it. This one covers the two fixes.

Two Ways to Miss

Keyword search and vector search fail in opposite directions. A keyword index, usually scored with an algorithm called BM25, finds documents that share the question’s words, weighted by how rare those words are. It’s excellent at the things embeddings blur together, like part numbers, error codes, statute citations, drug names and anything else where the exact string is the point. It’s poor at paraphrase. Ask about “ending a lease early” and it won’t find the clause titled “early termination.”

Embeddings are the reverse. They’re good at paraphrase and poor at exactness, because a vector doesn’t store the string, it stores a position in a space of meanings, and two error codes that differ by one digit sit almost on top of each other.

Which matters more? It depends on the data, and the common assumption that vectors always win doesn’t hold up. In an April 2026 study of 23,088 questions over financial reports, contracts and tables, plain BM25 found the right document in its top five results 64.4% of the time. OpenAI’s text-embedding-3-large managed 58.7% (Akarsu, Karaman and Mierbach). Financial documents are full of exact figures, line items and named entities, and that’s where keywords are strong.

Hybrid Search

The fix for two retrievers that miss different things is to run both and merge the results. The standard way to merge them is reciprocal rank fusion, which ignores the retrievers’ raw scores, since BM25 scores and cosine similarities aren’t comparable, and combines only their rankings. Each document gets a score from every list it appears in:

\text{RRF}(d) = \sum_{r \in R} \frac{1}{k + \text{rank}_r(d)}

where rank_r(d) is the document’s position in retriever r’s results, and k, conventionally 60, keeps any single list from dominating. A document ranked well by both retrievers rises. A document ranked first by one and absent from the other still has a chance.

It’s a few lines of code, and it works. In the same April 2026 study, fusing BM25 with dense retrieval raised top-five recall to 69.5%, better than either alone, and the authors were direct about the implication: “We recommend hybrid retrieval as the minimum viable baseline for any RAG deployment.”

Reranking

Hybrid search improves what comes back. It doesn’t do much about the order, and the order matters, because whatever lands in the top few results is what the model actually reads.

Both first-stage retrievers have the same structural limit. They score the question and each document separately: the document’s vector or keyword index is built in advance, with no knowledge of the question it will eventually be matched against. That’s what makes them fast enough to search millions of documents. It’s also why they can’t tell a passage that mentions the question’s terms from one that answers it.

A reranker reads the question and each candidate together. The classic form is a cross-encoder, a model that takes the pair as one input and outputs a relevance score, so every word of the question can be weighed against every word of the passage. That’s far too slow to run against a whole corpus, which is why it runs second, on the top few dozen or hundred candidates the first stage already found. The first stage is cheap and broad, and the second spends its more expensive attention only on what the first stage found.

Is it worth the extra step? It’s usually where the biggest gain is. Adding a reranker on top of hybrid search in the April 2026 study raised top-five recall from 69.5% to 81.6%, the largest single jump in the pipeline.

Share of 23,088 financial-document questions where the right document appeared in the top five results. Dense retrieval used OpenAI text-embedding-3-large; the reranker was Cohere Rerank v4.0 Pro. Source: Akarsu, Karaman and Mierbach, “From BM25 to Corrective RAG,” April 2026.

Show as a table

Method	Recall@5
Dense (embeddings)	58.7%
BM25 (keywords)	64.4%
Hybrid (RRF)	69.5%
Hybrid + rerank	81.6%

The same pattern shows up on harder retrieval. On BRIGHT, the reasoning-intensive benchmark from the first essay, researchers trained a retriever specifically for reasoning tasks, ReasonIR, which reached 29.9 nDCG@10 on its own and 36.9 with a reranker on top (Shao et al.). Even a retriever built for reasoning got about 23% better from a second stage.

When the Reranker Thinks

The newest rerankers aren’t small cross-encoders. They’re language models, and some of them reason before they rank. Rank1, presented at COLM 2025, was trained on 635,000 examples of a reasoning model working out whether a passage answers a question, and it performed at the state of the art on reasoning-heavy and instruction-following retrieval, worked well on data unlike its training set, and produced a reasoning chain explaining each judgment (Weller et al.).

That last property matters more than the benchmark, because a reranker that explains why it ranked a contract clause first gives a compliance reviewer something to check, which a bare relevance score doesn’t.

The fair caution is generalization and cost. An EMNLP 2025 study compared 22 reranking methods in 40 variants and found that LLM-based rerankers did best on familiar queries but generalized less well to queries newer than their training data, which it tested with a dataset built from questions written after the models were trained (Abdallah et al.). A reranker that reasons over every candidate also costs real inference per question, which at a hundred candidates adds up fast. The practical pattern is to reserve heavy reranking for the questions where the order really matters, and to measure it on your own recent data rather than trust a leaderboard built on older questions.

Where the Agents Fit

So what are coding agents doing when they grep in a loop? They run a cheap keyword search, read the results, judge which ones matter, and search again with better terms. That’s a first stage and a reranker, with the model playing both the query rewriter and the judge. It works because the model is a capable reranker and because code is full of exact identifiers that keyword search handles well. The May 2026 study is a warning about how fragile that is: change how results reach the model and the advantage can disappear.

We built the explicit version of this into our BI agent. When someone asked about hot dogs, a search_entities tool pulled candidates from a vector database, a reranker sorted out which items the question actually meant, and the model saw the result, decided whether the Polish sausage counted, and searched again if it didn’t have what it needed. Retrieval found the candidates, reranking put them in order, and the model made the final call.

Takeaways

If I were building retrieval today, I’d start with hybrid search rather than embeddings alone, because keyword and vector search miss different things and fusing them is nearly free. I’d put a reranker on top before tuning anything else, since in the research and in my own work that second stage is where the biggest gain is. I’d use a reasoning reranker where the order of results decides an outcome someone has to defend, and keep it off the questions where a fast cross-encoder is good enough. And I’d measure all of it on my own recent questions, because every number in this essay comes from someone else’s data.

The last essay in this series is about what neither stage can retrieve: the relationships between things, which no single document contains. That’s The Data Between the Data.

Finding candidates is cheap. Deciding which ones matter is where the intelligence goes.