Gumbel Reranking: Differentiable End-to-End Reranker Optimization
Siyuan Huang, Zhiyuan Ma, Jintao Du, Changhua Meng, Weiqiang Wang, Jingwen Leng, Minyi Guo, Zhouhan Lin · 2025
Read Gumbel Reranking: Differentiable End-to-End Reranker Optimization by Siyuan Huang, Zhiyuan Ma, Jintao Du, Changhua Meng, Weiqiang Wang, Jingwen Leng, Minyi Guo, Zhouhan Lin (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2502-11116.txt.
Introduction
Siyuan Huang and co-authors present a new way to train search rerankers in a 2025 computing paper. Search systems first pull many pages, then a reranker picks the best few to feed an answer engine. Training rerankers is hard because labeled page pairs are rare. Older methods train on fake examples that differ from real use, and they score each page alone.
The new method treats ranking as a mask that picks a top set of pages all at once. A noise trick makes that pick smooth enough to train end to end against the final answer loss. Tests showed clear gains, with one benchmark's recall up by a tenth. Better rerankers mean answer engines read the right pages first.
Catalog record
| CREATOR | Siyuan Huang, Zhiyuan Ma, Jintao Du, Changhua Meng, Weiqiang Wang, Jingwen Leng, Minyi Guo, Zhouhan Lin |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 59 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2502-11116
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Gumbel Reranking: Differentiable End-to-End Reranker Optimization Siyuan Huang, Zhiyuan Ma, Jintao Du, Changhua Meng, Weiqiang Wang, Jingwen Leng, Minyi Guo, Zhouhan Lin Abstract RAG systems rely on rerankers to identify relevant documents. However, fine-tuning these models remains challenging due to the scarcity of annotated query-document pairs. Existing distillation-based approaches suffer from training-inference misalignment and fail to capture interdependencies among candidate documents. To overcome these limitations, we reframe the reranking process as an attention-mask problem and propose Gumbel Reranking, an end-to-end training framework for rerankers aimed at minimizing the training-inference gap. In our approach, reranker optimization is reformulated as learning a stochastic, document-wise Top- k k attention mask using the Gumbel trick and relaxed top- k k sampling . This formulation enables end-to-end optimization by minimizing the overall language loss. Experiments across various settings consistently demonstrate performance gains, including a 10.4% improvement in recall on HotpotQA for distinguishing indirectly relevant documents. 3 3 footnotetext: Code is available at this url . 1 1 footnotetext: Work done during an internship at Ant Group. 2 2 footnotetext: Corresponding Author. 1 Introduction Figure 1: Vanilla reranker training methods for RAG systems typically rely on supervised learning of query-document pairs, which is limited by the scarcity of labeled data. To address this issue, existing methods leverage various LLM-supervised losses. However, this can lead to potential gaps between training and inference. In contrast, G-Rerank frames reranker training as learning a stochastic, document-wise top- k attention mask. This enables end-to-end optimization by minimizing language loss, ensuring better alignment between training and inference. Retrieval-Augmented Generation (RAG) has shown great potential in natural language processing tasks Lewis et al. (2020) ; Guu et al. (2020) ; Izacard and Grave (2021b) ; Borgeaud et al. (2022) . Despite their remarkable progress, retrieval models in RAG systems—comprising both the retriever and reranker—are typically trained on publicly available datasets and often struggle with long-tail queries requiring domain-specific knowledge. As a result, they necessitate further fine-tuning for specific downstream tasks Glass et al. (2022) ; Shi et al. (2024) . A key challenge in this context is the scarcity of labeled query-document pairs Lee et al. (2019) ; Sachan et al. (2023) . Therefore, a critical research question is how to end-to-end optimize the retrieval models of RAG systems solely relying on the system’s final language modeling loss. Recent efforts to improve retriever or reranker in RAG systems have explored distilling knowledge from LLMs into retrieval components. Techniques such as attention-based distillation Izacard and Grave (2021a) and perplexity-based distillation Sachan et al. (2021) ; Shi et al. (2024) ; Lin et al. (2024) ; Izacard et al. (2023) ; Glass et al. (2022) have yielded notable performance gains. However, these methods still exhibit critical limitations. First, although these methods claim to be end-to-end optimized, they focus on LLM-supervised losses like KL divergence Izacard et al. (2023) ; Glass et al. (2022) or marginalization Sachan et al. (2021) ; Shi et al. (2024) ; Lin et al. (2024) , which do not directly minimize the RAG system’s final generation loss, leading to potential misalignment between training and evaluation objectives. Additionally, attention-based distillation suffers from the distraction problem, where accumulated attention scores do not always reflect document relevance Ke et al. (2024) ; Li et al. (2024) . While perplexity-based distillation methods evaluate each candidate document in isolation, neglecting the interdependencies among retrieved documents. This oversight is particularly detrimental in multi-hop reasoning tasks requiring coherent logical relationships between documents Trivedi et al. (2022) ; Ho et al. (2020) . In this work, we propose a novel end-to-end strategy for training rerankers in RAG systems. We reformulate the reranking task through the lens of attention masks, where selecting the top- k subset from the retrieved candidate documents is viewed as the application of a document-wise top- k attention mask during attention computation. This perspective leads to a shift in the problem formulation: instead of directly learning a more effective reranker, we focus on learning the optimal document-wise top- k attention mask. However, since the hard attention mask is discrete, it can not be directly optimized via gradient descent. To overcome this challenge, we introduce a solution based on the Gumbel Trick Jang et al. (2017) and Relaxed Top- k techniques Chen et al. (2018) . This enables us to design a stochastic , top- k attention mask that is fully differentiable, allowing for end-to-end optimization. We note this approach as D ifferentiable M asked A ttention (DMA). With DMA in place, we reformulate the reranking problem as learning the optimal sampling weight for the corresponding attention mask. This leads to our end-to-end training framework, which we refer to as G umbel Rerank ing (G-Rerank). Unlike previous methods that rely on LLM-supervised losses, G-Rerank directly optimizes the reranker by minimizing the overall language modeling loss of the RAG system, thereby ensuring that the training objective closely aligns with the inference process. Additionally, G-Rerank accounts for interdependencies between retrieved candidate documents, making it suitable for multi-hop QA tasks. We evaluate our training approach across various architectures. Specifically, we conduct experiments using two language models—FiD Izacard and Grave (2021b) and CEPE-Llama2-7B Yen et al. (2024) —as well as two rerankers—BGE-Reranker-Base Xiao and Liu (2023) and RankT5 Zhuang et al. (2023) . Our method is tested on five benchmark datasets, covering both single-hop and multi-hop QA tasks. To comprehensively assess the effectiveness of our approach, we consider three different evaluation settings: mining , reranking , and generation . Our proposed training strategy achieves consistent improvements across all these settings. Furthermore, compared to distillation-based methods, our training approach significantly improves the reranker’s ability to distinguish indirectly relevant documents , leading to a 10.4% improvement in the Recall@5 metric on HotpotQA. Finally, we analyze the necessity of the Gumbel trick and the impact of prior knowledge in rerankers. 2 Related Works Training the Reranking Module in RAG Systems The effectiveness of RAG systems relies heavily on the quality of retrieval and reranking Glass et al. (2022) ; Dong et al. (2024) . Traditional retrieval methods are based on lexical similarity Robertson and Zaragoza (2009) , while recent advances leverage dense vectors and transformer architectures Karpukhin et al. (2020) ; Khattab and Zaharia (2020) . However, retrieval modules fine-tuned on public datasets often require additional adaptation for specific downstream tasks Izacard et al. (2023) ; Salemi and Zamani (2024) . To bridge this gap, recent works explore fine-tuning retrieval and reranking modules for tasks such as open-domain question answering (ODQA). One common strategy distills knowledge from LLMs into retrievers by ranking candidate documents based on generated answer perplexity Shi et al. (2024) ; Glass et al. (2022) ; Lin et al. (2024) . However, such methods overlook inter-document dependencies, crucial for multi-hop reasoning tasks Trivedi et al. (2022) ; Ho et al. (2020) . Alternative approaches use attention scores Izacard and Grave (2021a) or leave-one-out methods Izacard et al. (2023) ; Asai et al. (2022) , but these are not end-to-end optimized for generation quality, leading to a retriever-generation gap Ke et al. (2024) . Stochastic k-Subset Selection and Masked Attention Top- k relaxation has been widely studied for differentiable subset sampling, extending the Gumbel-Softmax trick Jang et al. (2017) ; Xie and