A reranker (rerank) in RAG is a second retrieval model that takes already found passages together with the question and reorders them by relevance. It does not find new documents; it only refines the ranking of candidates from the first stage. A recent open test on a corpus of 64 notes showed that plain semantic search already put the right note first in 23 out of 24 cases, and the reranker fixed the top result only once. A reranker pays off where the first stage returns many similar candidates and their order determines which context makes it into the model’s answer — and that needs to be tested on your own data, not turned on by default.
Prepared by AIrassvet October 14, 2026 based on the Habr article "I Added Rerank to Obsidian. What the Tests on 64 and 500 Notes Showed" (October 13, 2026) and the documentation for Sentence Transformers and Cohere. It is intended for project leaders, CIOs, and teams building enterprise search, a knowledge base, or a RAG assistant and deciding whether to add a reranking stage. We did not repeat the experiment ourselves: the figures belong to the author of the original article and apply to their test repositories. The article does not compare reranker vendors and does not cover vector database tuning. AIrassvet is not affiliated with the author of the test or the developers of the models mentioned; the section at the end of the article describes the publisher’s services.
Contents
- At a Glance
- What a Reranker Is and How Two-Stage Retrieval Works
- How Search Quality Is Measured: nDCG, Hit@1, and MRR
- What the Test on 64 Notes Showed
- What a Reranker Costs and How Much Latency It Adds
- The 500-Note Test: Document or Chunk
- When You Need a Reranker — and When You Don’t
- How to Test a Reranker on Your Own Knowledge Base
- Privacy: What Goes to the External Provider
- Automatic or On Demand
- Frequently Asked Questions
- Primary Sources and the Validation Method
At a Glance
- A reranker refines the order, but does not search again. If the first stage did not find the right document among the candidates, the reranker will not bring it back.
- The benefit is small on a small corpus. In the test on 64 notes, nDCG@10 rose from 0.94235 to 0.97502, but in six of the eleven "improvements" only weak supporting materials moved up.
- Sometimes it gets worse. In two of 24 queries, the useful note moved lower, even though the top result remained correct.
- Money is not the main constraint; latency is. One rerank call cost about $0.00008 on average, but added about half a second at the median and up to 4 seconds in the worst case.
- A metric may not show the benefit. On 500 notes, document-level metrics did not change, while the number of chunks with a complete answer in the top ten increased from 29 to 39–40 out of 40, depending on the model.
- The test author’s decision: the reranker is off by default and runs when the "Refine Results" button is clicked.
What a Reranker Is and How Two-Stage Retrieval Works
Semantic search is search by meaning, not by exact word match. Documents are split into chunks, each chunk is converted into a numeric vector — an embedding — and when a query comes in, the system looks for chunks with the nearest vectors. That is how a note about "resuming a task from a checkpoint" is found for the query "how to continue processing after a failure."
A bi-encoder is a model that encodes the question and the document separately. This is fast and works for collections of millions of records, but, as the Sentence Transformers documentation notes, this kind of search can return irrelevant candidates (Sentence Transformers, Retrieve & Re-Rank).
A cross-encoder is a model that takes the question and the document together and produces a single relevance score. It is more accurate because it compares the texts in context with each other, but scoring thousands or millions of pairs that way is too slow. So a fast search first selects, for example, 100 candidates, and then a cross-encoder reranks them.
A reranker is exactly that second stage: a model that orders the found documents from most to least relevant to the query (Cohere, Rerank Overview). The exact architecture depends on the model, but the goal is the same — to refine the order of already found candidates.
| First stage: embeddings | Second stage: reranker | |
|---|---|---|
| What it does | Finds candidates that are semantically close | Reorders candidates by relevance |
| How it compares | The question vector and document vectors are computed independently | The question and document are fed to the model together |
| Speed | Suitable for the whole corpus | Suitable only for dozens to hundreds of candidates |
| Can it find something that was missed? | Yes, that is its job | No — it only works with what the first stage found |
| What the provider gets | Texts when building the index and the question during search | Readable passages on each rerank call |
We covered where the reranker fits in the overall architecture of enterprise AI search in the article about a business RAG system.
How Search Quality Is Measured: nDCG, Hit@1, and MRR
To compare search "before" and "after," you need labels: for each question, you decide in advance which documents are useful. The test author used a scale from 0 to 3: 0 — the document is not related to the question, 1 — weakly related, 2 — useful but incomplete answer, 3 — direct and complete answer.
- Hit@1 shows whether a relevant document ended up in first place.
- MRR@10 averages the reciprocal rank of the first relevant document: first place gives 1, second — 0.5, third — about 0.33, and not appearing in the top ten — 0.
- nDCG@10 (Normalized Discounted Cumulative Gain) takes into account both the relevance of each document and its position in the top ten: the lower a useful document is ranked, the smaller its contribution. The value ranges from 0 to 1, where 1 means the ideal ranking according to the labels.
Important caveat: if all documents are assigned the same score, any reorderings preserve the perfect metric. That is exactly what happened in the author’s second experiment, discussed below.
What the Test on 64 Notes Showed
The author develops the open-source Veynrel plugin for Obsidian with semantic search and tested whether it made sense to run the reranker automatically (Habr, 10/13/2026). For the test, they built a synthetic repository that should be difficult for the first stage: 64 notes, 80 chunks, and similar topics with varying usefulness. There were 28 questions: for 24, an answer existed in the corpus, and four more were designed so that no answer would be available. The positive questions were split into six groups: paraphrases, negations, similar operations, different languages, multiple sources, and long notes.
The first stage used text-embedding-3-small embeddings, with up to 30 candidates per question; the second stage used Voyage Rerank 3 Lite through OpenRouter. The model was not given the correct answers, and the documents were not manually added.
| Metric (24 questions with an answer) | Embeddings only | After the reranker |
|---|---|---|
| Relevant document in first place | 23 out of 24 | 24 out of 24 |
| Relevant document in the top three | 24 out of 24 | 24 out of 24 |
| MRR@10 | 0.97917 | 1.00000 |
| nDCG@10 | 0.94235 | 0.97502 |
By nDCG@10, there were 11 improvements, 11 cases with no change, and 2 regressions; the average score increased by about 0.0327. But the author looked at which documents moved:
- In six of the eleven improvements only weak materials scored 1 changed positions, while the useful documents stayed put.
- In five cases relevant notes moved up, and only once did the reranker fix the top result — it moved the needed note from second place to first.
Where the reranker helped. The question was how to continue a paused data conversion without repeating completed operations. The database contained two similar notes: one about a planned service shutdown and one about checkpoints after an abnormal termination. The first stage put the planned shutdown first, while the reranker put the checkpoint note first. nDCG@10 for this query rose from 0.5788 to 0.9640.
Where it got worse. In the query about exercises for active speech, the main answer stayed first, but a useful note about comprehensive practice dropped from fourth place to eighth; nDCG@10 fell from 0.9149 to 0.8872. The second regression was in the query about checking restore from backup. The author verified that the needed text was included in the model queries, so the issue was not data loss.
Two more observations. On four multilingual queries, the metric did not change, and one of them took more than four seconds. For unanswered questions, the reranker still returned an ordered list: a high reranker score by itself does not prove that the answer exists in the database.
How much the reranker costs and what latency it adds
First, the author compared three models on four simple questions — all of them put the needed document first (Habr, 10/13/2026):
| Model | Correct first result | Average latency | Cost of 4 queries |
|---|---|---|---|
| Cohere Rerank 3.5 | 4 out of 4 | 490 ms | $0.004 |
| Voyage Rerank 3 Lite | 4 out of 4 | 521 ms | $0.0000102 |
| Qwen3 Reranker 8B | 4 out of 4 | 662 ms | $0.0002844 |
On this sample, Voyage was about 392 times cheaper than Cohere, but the models use different pricing schemes, and the ratio may differ at other volumes.
In the main test on 64 notes, 28 reranker calls cost $0.00223506, and the full run including embeddings cost $0.00243830; one clarification — about $0.00008 on average. That is roughly 31 times more expensive than in the short test: up to 30 candidates were used per refinement, and the cost grows with their number and length.
| Reranker latency (test on 64 notes) | Time |
|---|---|
| Average | 675 ms |
| Median | 516 ms |
| 95th percentile | 1042 ms |
| Longest query | 4020 ms |
The author’s conclusion: cost turned out to be the least noticeable limitation, while latency was the most noticeable. If the user clicked a button themselves, half a second is tolerable; waiting for a second model on every search gets annoying fast. For a RAG assistant, where an answer is generated after search, this latency adds to the total response time.
Test on 500 notes: document or fragment
The third run was conducted on a demo repository of 500 notes (1,600 fragments) — the methodology and aggregated results were published together with the article (Habr, 10/13/2026). The repository turned out to be template-based: technology names and generic descriptions without specific answers. In the end, there were only 4 substantive questions and 20 unanswered diagnostic ones. An AI agent labeled 12,000 question-document pairs, and a second agent without access to the model results checked them; there was no independent human review.
At the document level, all 500 notes received a “full answer” rating for all four questions, so Hit@1, MRR@10, and nDCG@10 were all 1 both before and after the reranker. But the plugin sends only one fragment per note to the reranker, and the answer could be in a different section. When the fragments themselves were evaluated separately, the picture changed:
| Fragments with a full answer in the top ten (4 questions, 40 positions total) | Result |
|---|---|
| Original search | 29 out of 40 |
| Voyage Rerank 3 Lite | 40 out of 40 |
| Cohere Rerank 3.5 | 39 out of 40 |
| Qwen3 Reranker 8B | 40 out of 40 |
The change affected two of the four questions. The reranker moved up candidates whose selected fragment already contained the answer. For RAG, this is an important observation: the model window receives fragments, not documents, so search quality should also be measured at the fragment level. A document-level metric may show “no change” where the context for generation actually improved. The author himself emphasizes that he is not drawing conclusions about large technical knowledge bases from this run: there were too few substantive questions.
When you need a reranker — and when you don’t
Below is an AIrassvet editorial decision table based on the test findings and the documentation’s description of two-stage search. These are guidelines for checking, not measured thresholds.
| Situation | Is a reranker likely needed? | Why |
|---|---|---|
| Hundreds of heterogeneous documents, specific questions | Probably not | The first stage often already puts the right document first |
| Thousands of similar documents: policies, instructions, incident resolutions with the same wording | Probably yes | The first stage needs to find a pool of candidates, and the second needs to tell them apart |
| One topic is covered across several documents with different levels of detail | Probably yes | Order determines whether the user sees an example or a definition first |
| RAG with a limited context window | Must test | The order of chunks determines which context reaches the model |
| The first stage often does not find the right document at all | No, fix the first stage first | A reranker will not recover what was missed: improve chunking, hybrid search, and metadata |
| A strict response-latency requirement | Only with measurement | Median around 0.5 s, tail up to seconds |
| Data cannot be sent to an external provider | Only a local model | The cloud reranker receives readable text for every refinement |
If your database has many exact identifiers — SKUs, contract numbers, error codes — first test hybrid search by keywords and semantics; an example of this approach is covered in the article on smart search for an online store.
How to Test a Reranker on Your Knowledge Base
The method below follows the logic of the test described in the Habr article and is suitable for a corporate knowledge base. This is an editorial plan, not the result of our own implementation.
- Collect questions from real work. Use real employee or customer queries, including paraphrases, negations (“without sending to the cloud”), similar tasks, different languages, and questions with no answer in the knowledge base.
- Label relevance in advance. Use a 0–3 scale for documents, and separately note whether the selected chunk contains the full answer. It is better to label before running the models and have people review a sample manually.
- Fix the candidate pool. Save the first-stage candidates (for example, 30) and pass exactly those to the reranker so the comparison is fair.
- Calculate several metrics. Hit@1, MRR@10, nDCG@10 at the document level, and the share of chunks with a full answer in the top 10.
- Review each change by eye. How many improvements affect only weak materials, how many times the top result was fixed, and where things got worse.
- Measure cost and latency. Median, 95th percentile, longest query, and billing cost per refinement.
Final report template to fill out before making a decision:
| Metric | Without reranker | With reranker | Decision |
|---|---|---|---|
| Hit@1 | |||
| nDCG@10 | |||
| Chunks with a full answer in the top 10 | |||
| Fixed top results / regressions | |||
| Median and p95 latency | |||
| Cost per refinement |
If you are choosing between several rerankers or embedding models, the same set of questions works as your own benchmark; we explained why public rankings are not enough for this choice in the article about your own benchmark for choosing an LLM. For how to evaluate search, answers, and RAG hallucinations separately, see the article on RAG accuracy.
Privacy: what goes to the external provider
The reranker needs readable chunk text. With a cloud model, the selected chunks are sent to the provider with every refinement — in addition to the texts sent when the index was built, if the embeddings are cloud-based too.
The test author found a non-obvious leak (Habr, 10/13/2026): the note-splitting module used the file name as a fallback title, and that name went directly into the request body sent to the provider, even though path fields were removed from the request. Fixing it required a separate representation of the chunk for external transmission.
Practical takeaways for a corporate knowledge base:
- check the contents of outgoing HTTP requests, not just the list of fields;
- remember that system data — file names, paths, metadata — can end up in the chunk text;
- if your documents contain personal or business-sensitive data, consider a local reranker or a model within a Russian infrastructure boundary.
Automatically or by button
The test author did not have data that would make it possible to know in advance which queries would benefit from the second stage. So in plugin version 2.1.0, they chose manual mode (Habr, 10/13/2026): the user gets standard results and, if they seem inaccurate, clicks “Refine results.” The plugin stores up to 30 first-stage candidates, shows 10, and does not repeat search during refinement. Automatic mode is still available in settings.
For a corporate system, there are three options:
| Mode | When it fits | Risk |
|---|---|---|
| Always on | Testing on your data showed a consistent gain, and the latency is acceptable | You pay latency on every request, even when there is no benefit |
| By button | A search scenario where a person reviews results manually | The user may not notice the button |
| By rule | Enabled for certain query types or sections of the knowledge base | The rule needs to be justified by measurement; otherwise it is just a guess |
Frequently Asked Questions
What Is a Reranker in RAG?
It is a second search model that takes the retrieved passages together with the question and reorders them by relevance. It does not search for new documents; it only refines the ranking of the candidates found in the first stage.
How Is a Cross-Encoder Different from Regular Embeddings?
Embeddings encode the question and the document separately and compare vectors, which is fast. A cross-encoder takes the question and the document together and evaluates them in context, which is more accurate but too slow for an entire database, so it is used on dozens or hundreds of candidates.
Does a Reranker Always Improve Search?
No. In an open test on 64 notes, it fixed the top result only once out of 24, and in two queries it made the order of the next documents worse. The benefit depends on how many similar documents are in the database and how well the first stage works.
How Much Does a Reranker Cost?
In a test on Habr, one reranking pass with Voyage Rerank 3 Lite cost about $0.00008 on average with 30 candidates. The cost increases with the number and length of candidates and depends on the model pricing, so it is worth measuring on your own data.
Can a Reranker Find a Document That Is Not in the Results?
No. It works only with first-stage candidates. If the needed document did not make it into the candidate pool, you need to improve the first stage: chunking, embeddings, hybrid search, or metadata.
How Do We Know Whether a Reranker Is Needed for Our Knowledge Base?
Test it on your own questions: compare search with and without a reranker using Hit@1, nDCG@10, and the share of passages with a complete answer, review each change, and measure latency and cost. It is worth enabling if the gain is consistent and the latency is acceptable for the use case.
Primary Sources and Verification Method
Method: we read the original article in full on October 14, 2026, listed all metrics, prices, and latencies along with the author’s caveats, and cross-checked the description of two-stage search against the documentation. AIrassvet did not run its own reranker tests; the decision table and verification plan are the editorial team’s methodology.
- Habr, 10/13/2026: “I Added Rerank to Obsidian. What the Tests on 64 and 500 Notes Showed” — test setup, metrics, costs, latency, privacy, and the decision to keep it manual.
- Sentence Transformers: Retrieve & Re-Rank — bi-encoder and cross-encoder in two-stage search.
- Cohere: Rerank Overview — what a reranker is for.
Limitations: all of the author’s tests were run on synthetic or template note stores, not on a corporate knowledge base; the 500-note labeling was done by AI agents without human review; provider prices and plans change.
How AIrassvet Helps Tune Search in a RAG System
If your AI assistant answers questions from internal documents, response quality starts with which passages it retrieves. The decision to add a reranker should be based on measurement, not defaults. On the AIrassvet website you can find services directly related to this task:
- RAG systems and enterprise knowledge bases: document chunking, first-stage retrieval, and reranking validation on your own questions.
- Deployment and fine-tuning of language models on the customer’s own servers—if document passages cannot be sent to an external provider.
- Process audits, data preparation, and question labeling for search evaluation.
- MVP development, integration, and testing before rollout across the full database.
The first step is to choose one section of the knowledge base and one user group, collect their real questions, describe the document sources and transfer constraints, and agree on a benchmark: what share of correct first results and what latency will be sufficient for you.
Bottom Line: Measure First, Then Add the Second Stage
A reranker is a useful tool, but not a required layer in every RAG system. The open test showed that on a small database with a strong first stage, it almost does not change the top result, sometimes worsens the next positions, and adds about half a second of latency. But where there are many similar documents and the order of passages matters for generation, the second stage can move passages with complete answers to the top.
That is why the decision should be data-driven: your own questions, labeled data, document- and passage-level metrics, and measurements of cost and latency. If the gain is stable, enable the reranker for the needed scenarios; if not, improve the first-stage search.
