Rearank : Reasoning Re-ranking Agent via Reinforcement Learning

Le Zhang, Bo Wang, Xipeng Qiu, Siva Reddy, Aishwarya Agrawal · 2025

Read Rearank : Reasoning Re-ranking Agent via Reinforcement Learning by Le Zhang, Bo Wang, Xipeng Qiu, Siva Reddy, Aishwarya Agrawal (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2505-20046.txt.

Introduction

Le Zhang and four co-authors build REARANK, a language model agent that thinks out loud before reordering search results. Reranking matters because first-pass retrieval returns candidates in rough order. A model that reasons explicitly about the list should rank better and explain itself.

Training used reinforcement learning plus data augmentation, and needed only one hundred seventy-nine annotated examples. Built on a seven-billion-parameter open model, REARANK matched GPT-4 on standard retrieval benchmarks, in and out of domain. On reasoning-heavy BRIGHT benchmarks it surpassed GPT-4. The authors highlight the result as evidence that reinforcement learning can sharpen model reasoning for ranking tasks without massive labeling efforts.

Catalog record

CREATORLe Zhang, Bo Wang, Xipeng Qiu, Siva Reddy, Aishwarya Agrawal
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE38 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2505-20046

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Rearank : Reasoning Re-ranking Agent via Reinforcement Learning Le Zhang, Bo Wang, Xipeng Qiu, Siva Reddy, Aishwarya Agrawal Abstract We present Rearank , a large language model (LLM)-based listwise reasoning reranking agent. Rearank explicitly reasons before reranking, significantly improving both performance and interpretability. Leveraging reinforcement learning and data augmentation, Rearank achieves substantial improvements over baseline models across popular information retrieval benchmarks, notably requiring only 179 annotated samples. Built on top of Qwen2.5-7B, our Rearank -7B demonstrates performance comparable to GPT-4 on both in-domain and out-of-domain benchmarks and even surpasses GPT-4 on reasoning-intensive BRIGHT benchmarks. These results underscore the effectiveness of our approach and highlight how reinforcement learning can enhance LLM reasoning capabilities in reranking. The code is available https://github.com/lezhang7/Rearank . Figure 1: (Top) Average rerank results on popular benchmarks (over BM25 top 100), the performance improves with RL training; (Bottom) Rearank inference example. The agent provides the reasoning and final ranking of all passages, unlike current agents Sun et al. (2023) ; Pradeep et al. (2023b) that only output the final answer. 1 Introduction Information retrieval (IR) is a core building block of intelligent systems, providing the foundation for accessing, organizing, and reasoning over information. Modern IR systems Reimers and Gurevych (2019) ; Reimers and Gurevych (2020) ; Wang et al. (2021) ; Formal et al. (2021) typically follow a two-stage approach: initial retrieval (e.g., fast lexical methods Robertson et al. (2009) ) to gather candidates, followed by reranking for fine-grained prioritization of relevant results. This two-stage process is particularly crucial for Retrieval-Augmented Generation systems Lewis et al. (2020) ; Borgeaud et al. (2022) ; Zhang et al. (2023) , where accurate retrieval and effective reranking of context passages significantly impact generated output quality. Recent advances in large language models (LLMs) have shown strong promise for this reranking phase Sun et al. (2023) , particularly through their use as reranking agents that rely on direct outputs rather than internal states ( e.g. logits ) . This paradigm, exemplified by zero-shot prompting methods Sun et al. (2023) ; Ma et al. (2023) ; Zhuang et al. (2024) , offers significant deployment flexibility, especially in model-as-a-service scenarios. However, effectively adapting LLMs specifically as reranking agents presents several key challenges: (i) LLMs are not inherently optimized for ranking objectives, and crucially, current zero-shot methods do not learn from the interaction signals generated during the reranking process; (ii) Achieving competitive performance often necessitates supervised fine-tuning, a process severely constrained by the scarcity and high cost of acquiring high-quality labeled ranking data Sun et al. (2023) ; (iii) The decision-making processes within these models frequently lack transparent and interpretable reasoning, which limits explainability and fails to leverage test-time scaling properties of LLMs; (iv) State-of-the-art reranking agents frequently depend on large, often proprietary models (e.g., GPT-4) or face significant challenges in local deployment. This dependency on large models incurs substantial computational costs and significant inference latency (e.g., reranking 20 passages with reasoning process using DeepSeek-R1 Guo et al. (2025) via API takes around 90-120 seconds). In this work, we propose Rearank , the first reasoning listwise reranking agent. Rearank ’s optimization for listwise reranking with explicit reasoning is incentivized using reinforcement learning (RL). This approach effectively leverages the rich, order-based signals inherent in listwise reranking. To address the significant data scarcity challenge typically associated with training listwise models, we introduce a data augmentation pipeline capable of generating extensive listwise ranking sets from a remarkably small seed of only 179 annotated queries. At inference, Rearank generates explicit, interpretable reasoning for each ranking step, as shown in fig. 1 , enabling test-time scaling. Built upon the principle of injecting strong reasoning capabilities into a compact model, Rearank operates with low operational cost. The combination of its practical model size and listwise reranking strategy enhances inference efficiency by minimizing LLM calls, thereby facilitating local deployment. Our experimental results demonstrate Rearank ’s effectiveness: it significantly surpasses baseline models and achieves performance comparable to strong models like GPT-4 and the recent powerful reasoning model Qwen3-32B on standard and out-of-domain benchmarks. Notably, Rearank even surpasses strong GPT-4 performance on reasoning-intensive tasks, highlighting its advantages in combining reasoning with practical efficiency, as summarized in fig. 1 . Our main contributions are threefold: (i) We introduce Rearank , a novel reasoning-based reranking agent based on the listwise reranking strategy that effectively integrates explicit reasoning capabilities into the reranking process. We formularize the reranking problem in RL framework, and propose a data synthesis method requiring only 179 annotated queries, and a new reward model leveraging ranking information for training, enabling efficient RL training of Rearank ; and (ii) Rearank achieves significant performance improvements over strong baselines and matches or surpasses results from competitive models like GPT-4 and strong reasoning model Qwen3, particularly on reasoning-intensive tasks, while offering substantially improved inference efficiency due to its compact model size. (iii) We provide a comprehensive analysis on reasoning transferability and examining the relationship between reasoning length and final ranking performance to better understand the role of reasoning in reranking. 2 Related Work Large Reasoning Models Recent advancements in LLMs have yielded increasingly sophisticated reasoning capabilities, often emerging with scale Wei et al. (2022a) ; Kojima et al. (2022) . Techniques like Chain-of-Thought prompting Wei et al. (2022b) and its variants Kojima et al. (2022) ; Wang et al. (2022) further enhance these skills by enabling explicit reasoning processes. Beyond prompting, training methods like RL are used to incentivize long0 CoT reasoning; models such as Deepseek-R1 Guo et al. (2025) , OpenAI o1, and o3 leverage RL for enhanced reasoning, showing general task improvements. These developments enable LLM application in complex domains like math problems and planning. Our work applies these advanced reasoning capabilities to reranking. LLMs for Re-ranking LLMs are increasingly being used for reranking, moving beyond traditional feature-based models Zhang et al. (2024b) ; Lee et al. (2024) ; Ma et al. (2024a) ; BehnamGhader et al. (2024) ; Sachan et al. (2022) . Approaches range from pointwise Liang et al. (2022) , pairwise Qin et al. (2023) , setwise Zhuang et al. (2024) to listwise methods Ma et al. (2023) ; Sun et al. (2023) ; Pradeep et al. (2023b) . While few-shot/zero-shot prompting was explored, supervised fine-tuning on ranking data shows further gains Pradeep et al. (2023b) ; Pradeep et al. (2023a) . Building on this, our work proposes a novel RL approach without cold start for training a listwise LLM reranker, focusing on robust performance and reasoning capabilities, especially in out-of-domain and reasoning-intensive scenarios. A concurrent work Zhuang et al. (2025) also trains an LLM as a setwise reranker using RL. It simplifies the task to finding the single most relevant passage index , relying on a sparse text-matching binary reward signal. This signal lacks the rich, order-based information present in listwise ranking, which consequently necessitates extensive training data. Furthermore, setwise inference is highly inefficient as it ranks only one passage at a time , unlike our listwise method which reranks an entire passages simultaneously as shown in fig. 2 . Therefore, we adopt the listwise reranking strategy. Figure 2: Listwise vs. Setwise Reranking. Setwise reranking yields binary scores (0 or 1); listwise reranking offers richer, continuous scores between 0 and 1. 3 Method

Read the complete text