Rank1: Test-Time Compute for Reranking in Information Retrieval
Orion Weller, Kathryn Ricci, Eugene Yang, Andrew Yates, Benjamin Van Durme · 2025
Read Rank1: Test-Time Compute for Reranking in Information Retrieval by Orion Weller, Kathryn Ricci, Eugene Yang, Andrew Yates, Benjamin Van Durme (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2502-18418.txt.
Introduction
Orion Weller and co-authors build Rank1, a search reranker that thinks before it ranks. Rerankers reorder search results, and the new reasoning models — the ones that mull before answering — had never been harnessed for the job. Their 2025 paper distills a big reasoning model's thought process into a smaller reranker.
The training set, released openly, holds over six hundred thousand reasoning traces over web search queries and passages. Models raised on it top reasoning and instruction-following benchmarks. They also travel well to new domains, because prompts can steer them, and their reasoning chains can be shown to users or fed to retrieval systems. Even quantized, shrunken versions keep most of the strength. Search engineers get a reranker that explains itself.
Catalog record
| CREATOR | Orion Weller, Kathryn Ricci, Eugene Yang, Andrew Yates, Benjamin Van Durme |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 39 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2502-18418
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Rank1: Test-Time Compute for Reranking in Information Retrieval Orion Weller, Kathryn Ricci, Eugene Yang, Andrew Yates, Benjamin Van Durme Abstract We introduce Rank1 , the first reranking model trained to take advantage of test-time compute. Rank1 demonstrates the applicability within retrieval of using a reasoning language model (i.e. OpenAI’s o1, Deepseek’s R1, etc.) for distillation in order to rapidly improve the performance of a smaller model. We gather and open-source a dataset of more than 600,000 examples of R1 reasoning traces from queries and passages in MS MARCO. Models trained on this dataset show: (1) state-of-the-art performance on advanced reasoning and instruction following datasets; (2) work remarkably well out of distribution due to the ability to respond to user-input prompts; and (3) have explainable reasoning chains that can be given to users or RAG-based systems. Further, we demonstrate that quantized versions of these models retain strong performance while using less compute/memory. Overall, Rank1 shows that test-time compute allows for a fundamentally new type of explainable and performant reranker model for search. 1 1 1 Models, code, and data are available at https://github.com/orionw/rank1 1 Introduction Reasoning language models (LMs) like OpenAI’s o1, Deepseek’s R1, and Gemini’s Flash-Thinking have shown improved reasoning abilities through the use of test-time compute, i.e. generating a reasoning chain of tokens that allow the model to “think” before giving the final answer. Another large benefit to these style of models is that the reasoning chain can easily be distilled into smaller models. As shown by Deepseek’s R1 ( Guo et al. 2025 ) smaller models learn incredibly well from simple supervised fine-tuning on the larger model’s reasoning chains. The benefits that reasoning models bring to general text generation would also be valuable in an information retrieval (IR) context: allowing models additional time to reason why a passage could be relevant, while also allowing an auditable reasoning process to give to the user or RAG system. For this approach to be maximally effective, the model must be able to reason over both query and passage; if applied solely to the query, the reasoning model would not know the passage context and would have to try to infer it. This would be a form of query-expansion ( Nogueira et al. 2019 ) and limits the model’s ability to be precise. Thus, our work focuses on bringing test-time compute to IR in a reranking setting, where the model needs to compute the relevance of an initial top-k candidates. To accomplish this goal, we sample 635,000 examples of R1’s thought process on the MS MARCO dataset ( Nguyen et al. 2016 ) . We then fine-tune a suite of LMs on these reasoning chains and find that they show remarkable reasoning capabilities. Surprisingly, they also exhibit an ability to be prompted despite training from the base LMs only (without instruction fine-tuning) and while having no instruction-based IR training data (only MS MARCO). This includes state-of-the-art performance on the BRIGHT benchmark for reasoning ( Su et al. 2024 ) , the NevIR benchmark on complex negation understanding, and the mFollowIR dataset on multilingual instruction-following in IR despite having no non-English reasoning training data ( Weller et al. 2025 ) . Figure 1: Example reasoning traces from R1, used to train Rank1 . Note the self-inquisitory reasoning ( in blue ) where the model questions is if it has the correct answer. We also conduct a detailed analysis of performance on traditional IR benchmarks, such as TREC DL19 ( Craswell et al. 2020 ) and BEIR ( Thakur et al. 2021 ) . We find that these datasets are likely model-saturated, as Rank1 surfaces an extremely large number of unjudged documents (364% more than RankLLaMA-14B). We argue these benchmarks are no longer helpful for distinguishing between the best performing rerankers and that focus should be put on benchmarks that examine advanced reasoning, instruction-following, and have more modern (e.g. post-ChatGPT) annotations. Overall, Rank1 shows the many benefits that test-time compute can bring to the field of IR: explainable reasoning chains that can be audited by users or used by agentic RAG systems, significantly improved reasoning performance, and adaptability from user-given prompts . 2 Model Training 2.1 Data Preparation In order to distil from R1, we first need to gather data to use to prompt it. We use the MS MARCO collection due to its diversity in topics and common use in previous work. We use kluster.ai as the API service to access R1 using their batch mode. We generate data from an equal number (25% of the data) from each of the (1) positive examples in MS MARCO, (2) sampled negatives from Tevatron 2 2 2 From https://huggingface.co/datasets/Tevatron/msmarco-passage-aug (gathered from BM25 and CoCondenser), (3) rank 1-5 hard negatives from mT5-13B, and (4) rank 5-10 hard negatives from mT5-13B. However, we found that R1 classified roughly 80% of the mT5 hard negatives as positives. Thus we did another round of generation using only hard negatives from rank 5-10 and easy negatives from Tevatron. As the mined hard negatives do not have an official label, it is likely that many of them were false negatives and that R1 classified them correctly. After all generation, our dataset has 635,264 examples of R1 generations, where R1 labeled 62.9% as relevant and 37.1% as non-relevant. We show a plot of these generation lengths in Figure 2 where we see a fairly normal distribution. Although we thought there may be length differences between these four subsets of data, we found that they all had the same rough distribution. 2.2 Data Mix and Quality Filtering Since we had a surplus of documented judged relevant, we tried various methods to come to our final data mix. We initially tried using all the data, after balancing for the labels. We found that this performed significantly worse than filtering based on the labels we were most sure about (i.e. the positives from MS MARCO and the negatives from Tevatron). However, even on those subsets, 15% of R1’s final prediction disagreed with the implied labels – thus we filtered out these instances. Beyond being labeled as positives, we found that a large number of the mT5 mined hard negative samples were noisy. To alleviate this, we used a model trained on the first mix to self-filter the data. 3 3 3 We used the Mistral 24B version as it was the largest that fit on 1 GPU, filtering all instances where the model’s prediction didn’t agree with R1’s prediction. This filtered approximately another 10% of the data, mostly false negatives. Since we still had a surplus of ”relevant” labelled instances, we took all positives from the official MS MARCO positives and all negatives from the self-filtered set. This left us with a training set of 386,336 high quality training samples: 136k from the original MS MARCO positives, 154k from the Tevatron negatives, and 96k from the mT5 negatives. We note that having more negatives is standard: RankLLaMA ( Ma et al. 2024 ) trained on a 15:1 ratio of negatives to positives for 7 million examples. Figure 2: Distribution of word lengths of the reasoning chains generated from R1. It has a slightly rightward skew but is generally normal shaped. Note that there is no