LimRank : Less is More for Reasoning-Intensive Information Reranking
Tingyu Song, Yilun Zhao Siyue ZhangChen ZhaoArman CohanYale NLP Lab · 2025
Read LimRank : Less is More for Reasoning-Intensive Information Reranking by Tingyu Song, Yilun Zhao Siyue ZhangChen ZhaoArman CohanYale NLP Lab (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2510-23544.txt.
Introduction
Tingyu Song and four co-authors show that search reranking does not need huge amounts of training data. Reranking means taking a first list of search results and reordering it so the best matches rise to the top. Standard practice fine-tunes large language models on massive example sets, which costs a lot of computing time and money.
The authors built an open-source pipeline, LimRank-Synthesizer, that creates varied and hard training examples on its own. Fine-tuning on that synthetic data produced their model, LimRank, trained on under five percent of the data earlier work used. Even so, it scored competitively on two tough benchmarks, one for reasoning-heavy retrieval and one for following search instructions. Extra tests showed it also helps with science literature search and question answering that draws on retrieved text.
Catalog record
| CREATOR | Tingyu Song, Yilun Zhao Siyue ZhangChen ZhaoArman CohanYale NLP Lab |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 51 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2510-23544
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
LimRank : Less is More for Reasoning-Intensive Information Reranking Tingyu Song, Yilun Zhao Siyue ZhangChen ZhaoArman CohanYale NLP Lab Abstract Existing approaches typically rely on large-scale fine-tuning to adapt LLMs for information reranking tasks, which is computationally expensive. In this work, we demonstrate that modern LLMs can be effectively adapted using only minimal, high-quality supervision. To enable this, we design LimRank-Synthesizer , a reusable and open-source pipeline for generating diverse, challenging, and realistic reranking examples. Using this synthetic data, we fine-tune our reranker model, LimRank . We evaluate LimRank on two challenging benchmarks, i.e., Bright for reasoning-intensive retrieval and FollowIR for instruction-following retrieval. Our experiments demonstrate that LimRank achieves competitive performance, while being trained on less than 5% of the data typically used in prior work. Further ablation studies demonstrate the effectiveness of LimRank-Synthesizer and the strong generalization capabilities of LimRank across downstream tasks, including scientific literature search and retrieval-augmented generation for knowledge-intensive problem solving. SighingSnow/LimRank 1 Introduction Figure 1: (Top) An illustration of reasoning-intensive reranking scenarios that demand more than surface-level semantic matching. These tasks require multi-step inference, contextual reasoning, and recognition of implicit relationships between queries and documents. (Bottom) An overview of LimRank-Synthesizer , which generates high-quality training data for reranking tasks. Recent studies have increasingly leveraged LLMs for reranking tasks in information retrieval Peng et al. 2025 ; Zhuang et al. 2025 . While LLMs have shown effectiveness in general-purpose reranking scenarios, emerging research reveals notable limitations when these models are applied to reasoning-intensive retrieval settings Zhang et al. 2025b ; Weller et al. 2025b ; Shao et al. 2025 ; Song et al. 2025b ; Zhang et al. 2025a . These limitations extend beyond performance metrics. They arise from the inherent challenges LLMs encounter when relevance depends on more than surface-level keyword matching or shallow semantic similarity. In such scenarios, effective retrieval requires multi-step inference, contextual reasoning, and the ability to recognize implicit relationships between queries and documents Su et al. 2024 . Inspired by the growing success of reasoning LLMs Jaech et al. 2024 ; Guo et al. 2025 , several recent studies Weller et al. 2025b ; Zhuang et al. 2025 ; Yan et al. 2025 have begun exploring the training of LLMs that can leverage test-time computation to improve performance in reasoning-intensive retrieval tasks. However, contemporary approaches typically rely on large-scale supervised fine-tuning. For instance, Rank1 Weller et al. 2025b is fine-tuned on over nearly 700K examples of DeepSeek-R1’s reasoning traces, which is expensive in terms of compute and data. We hypothesize that frontier LLMs already possess considerable latent reasoning capabilities for reranking, and that these capabilities can be activated and steered using a small number of carefully curated, high-quality examples that encourage extended deliberation. This “less is more” approach has shown promise in other domains: for example, LIMA Zhou et al. 2023 and LIMO Ye et al. 2025 achieve strong performance in instruction following and complex reasoning with minimal but strategically selected examples, demonstrating that carefully curated demonstrations can effectively steer pretrained models without the need for massive fine-tuning. To our knowledge, we are the first to investigate this paradigm in reranking tasks. We introduce LimRank-Synthesizer , a modular and open-source pipeline for generating high-quality reranking training data through several novel design choices. LimRank-Synthesizer is guided by three core principles: domain diversity, alignment with real-world use cases, and difficulty diversity. It generates retrieval queries paired with corresponding positive and negative passages, and employs frontier reasoning models ( i.e., DeepSeek-R1) to produce multi-step reasoning traces. We then apply LLM-based filtering to discard low-quality traces. Using LimRank-Synthesizer , we generate a compact yet effective dataset of 20K examples—only 2.85% of the data used in Rank1. We fine-tune Qwen2.5-7B on this dataset to produce our reranker model, LimRank . We evaluate LimRank on two challenging reranking tasks: (1) reasoning-intensive retrieval using Bright Su et al. 2024 , and (2) instruction-following retrieval using FollowIR Weller et al. 2025a . LimRank achieves the nDCG@10 score of 28.0% on Bright and p -MRR score of 1.2 on FollowIR , representing the best performance among models with 7B-level parameters. To better understand the strengths and limitations of LimRank , we conduct an in-depth human evaluation, revealing that LimRank excels particularly in settings that require multi-hop reasoning, subtle instruction disambiguation, and context-sensitive reranking. Additionally, we conduct extensive ablation studies on LimRank-Synthesizer and show that each component of our guidelines is essential. We also train different model variants using synthetic IR datasets of the same size ( i.e., 20K). Models trained with our data consistently outperform those trained with other synthetic IR datasets used by Rank1 , Promptriever, and ReasonIR. To assess LimRank ’s practical utility, we further deploy and evaluate it in two real-world-inspired tasks: (1) Scientific Literature Search on the LitSearch dataset Ajith et al. 2024 , and (2) Retrieval-Augmented Generation (RAG) on the GPQA benchmark Rein et al. 2024 . Compared to the previous state-of-the-art, Rank1-7B, LimRank demonstrates strong generalization, achieving 30.3% accuracy on GPQA (vs. 28.3%) and a competitive 60.1% Recall@5 on LitSearch (vs. 60.8%), demonstrating its effectiveness as a plug-in reranker in real-world systems. 2 Related Works Reasoning-intensive Information Retrieval Su et al. 2024 has gained increasing attention due to its practical relevance in real-world scenarios. Some works, such as Rank1 Weller et al. 2025b and ReasonIR Shao et al. 2025 , address this challenge by fine-tuning models on large-scale, reasoning-focused retrieval datasets to enhance the performance of rerankers and retrievers, respectively. Recently, other works Ji et al. 2025 ; Abdallah et al. 2025 ; Liu et al. 2025 have proposed complex training strategies using synthetic data. However, the large data volume required by these methods leads to high training costs, and the training strategies themselves introduces extra training cost. Some other works Jin et al. 2025 ; Song et al. 2025a ; Zhuang et al. 2025 employs reinforcement learning to equip retrievers and rerankers with reasoning capabilities. While effective, these methods often entail significant computational, data, and engineering overhead. Recent studies Zhou et al. 2023 ; Ye et al. 2025 suggest that modern LLMs possess substantial latent reasoning abilities across a range of tasks, which can be efficiently activated and enhanced using small amounts of task-specific training data. To support this, various data selection methods Xia et al. 2024 ; Liu et al. 2024 have been proposed and shown to be effective. To our best knowledge, our work is the first to apply such “less is more” exploration to reranking tasks. Element Data Generation Guideline Query Domain Diversity: Queries should not limited to everyday contexts but also span specialized domains such as finance, law, and healthcare. Alignment with Daily: Life Queries vary in complexity to mirror real-world use cases. Simple queries reflect straightforward needs, while complex ones mimic intricate scenarios. Difficulty Diversity: The dataset should contain simple queries with challenging ones. Hard queries incorporate patterns like instruction-following, multi-step reasoning, and so on. Reasoning Chain Well-Structured Reasoning: Reasoning chains should be organized logically to enhance inference efficiency. Clear step-by-step processes help the model navigate solutions systematically during search operations. Human Verification: Reasoning chain undergoes human review to validate accuracy, coherence, and alignment with the query’s intent. This ensures robust, error-resistant logic. Adaptive Analysis: For simple queries,