ASRank: Zero-Shot Re-Ranking with Answer Scent for Document Retrieval
Abdelrahman Abdallah, Jamshid Mozafari, Bhawna Piryani, Adam Jatowt · 2025
Read ASRank: Zero-Shot Re-Ranking with Answer Scent for Document Retrieval by Abdelrahman Abdallah, Jamshid Mozafari, Bhawna Piryani, Adam Jatowt (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2501-15245.txt.
Introduction
Abdelrahman Abdallah and co-authors present a re-ranking method for document search in a paper from two thousand twenty-five. Question answering systems that fetch documents first depend on ranking the best ones at the top. Standard search often buries the most relevant document below weaker matches.
Their method scores each fetched document by how well answers drawn from it match an expected answer scent, computed by a large language model with no task training. Gains were marked across six question data sets. On one standard set, top-rank accuracy more than doubled for two common search methods. The authors report results rivaling leading re-rankers.
Catalog record
| CREATOR | Abdelrahman Abdallah, Jamshid Mozafari, Bhawna Piryani, Adam Jatowt |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 68 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2501-15245
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
ASRank: Zero-Shot Re-Ranking with Answer Scent for Document Retrieval Abdelrahman Abdallah, Jamshid Mozafari, Bhawna Piryani, Adam Jatowt Abstract Retrieval-Augmented Generation (RAG) models have drawn considerable attention in modern open-domain question answering. The effectiveness of RAG depends on the quality of the top retrieved documents. However, conventional retrieval methods sometimes fail to rank the most relevant documents at the top. In this paper, we introduce ASRank 1 1 1 https://github.com/DataScienceUIBK/rankify , a new re-ranking method based on scoring retrieved documents using zero-shot answer scent which relies on a pre-trained large language model to compute the likelihood of the document-derived answers aligning with the answer scent. Our approach demonstrates marked improvements across several datasets, including NQ, TriviaQA, WebQA, ArchivalQA, HotpotQA, and Entity Questions. Notably, ASRank increases Top-1 retrieval accuracy on NQ from 19.2 % 19.2\% to 46.5 % 46.5\% for MSS and 22.1 % 22.1\% to 47.3 % 47.3\% for BM25. It also shows strong retrieval performance on several datasets compared to state-of-the-art methods (47.3 Top-1 by ASRank vs 35.4 by UPR by BM25). 1 Introduction Figure 1: After re-ranking the top 1,000 passages retrieved by DPR Karpukhin et al. (2020a) with ASRank , our method surpasses the performance of strong unsupervised models like UPR Sachan et al. (2022) on the Natural Questions and TriviaQA datasets. Document retrieval is a core sub-task in many NLP problems, including open-domain question answering (ODQA), where a document is retrieved and then read to answer an input query. This process tries to find the most relevant documents or passages given the query. The Retrieval-Augmented Generation (RAG) model has achieved a significant improvement in the field of ODQA Lewis et al. (2020) ; Abdallah et al. (2024) ; Gruber et al. (2024) . RAG models combine retrieved documents and advanced pre-trained large language models (LLMs) generating responses based on the retrieved information Lewis et al. (2020) ; Lála et al. (2023) ; Mozafari et al. (2024) ; Abdallah et al. (2023) ; Piryani et al. (2024) . However, the performance of RAG models depends on the top retrieved documents, especially on the first document Setty et al. (2024) ; Zhang et al. (2024) ; Abdallah and Jatowt (2023) . The RAG model usually uses the first retrieved document, which is the primary source for generating the response. In RAG, queries and documents are embedded in a shared representation space to enable efficient search before using a task-specific model to perform a deeper, token-level document analysis. In this paper, we introduce ASRank , a simple, effective, fast, and cost-efficient re-ranking method that leverages the concept of answer scent which is analogous to the way in which animals track the scent of their food or prey Maxwell and Azzopardi (2018) . Cognitive psychologists Winerman (2012) have found that people search for information online in much the same way as animals hunt for food, leading to the establishment of the concept of information scent in the Information Retrieval field. It refers to the trail of relevant information that leads a user to the correct answer. Our proposal is built upon a similar concept of tracing the answer scent. It first utilizes larger LLMs like GPT-3.5 or Llama 3-70B to generate an answer scent. This is done just once, hence is computationally efficient. Subsequently, a smaller model such as T5 is employed to re-rank the documents based on the received answer scent. This two-tiered approach allows leveraging the generative capabilities of a larger LLM to boost the re-ranking capabilities of smaller models thanks to improved contextual understanding. Our method scores retrieved documents using a zero-shot answer scent, which relies on a pre-trained LLM to compute the likelihood of the document-derived answers aligning with the answer scent. This approach allows to rank documents not just based on their initial retrieval scores but also on the likelihood of containing an answer (via answer scent) and the degree to which they contain information that aligns with the expected answer. By applying a cross-attention mechanism to every token in both the question and the passage, ASRank tracks the answer scent within the document corpus. Our approach successfully addresses the challenge of ensuring that the most relevant document is ranked at the top, which is a key component in open-domain question answering and RAG systems (Figure 1 ). 2 Method In this section, we detail the methodology of ASRank , starting with retrieving documents based on either sparse or dense techniques. Subsequently, we introduce our concept of generating an Answer Scent using a large language model (Section 2.2 ), followed by an efficient re-ranking process that employs a smaller model (Section 2.3 ), which enhances the alignment and relevance of the retrieved documents to the query in our RAG system. Figure 2 shows an overview of the ASRank framework. Figure 2: Our ASRank framework, starts with document retrieval, re-ranking using the answer scent from LLMs, and finally passing the top-k document into the RAG system. 2.1 Retriever Let \mathcal{M}=\{\boldsymbol{d}_{1},\ldots,\boldsymbol{d}_{M}\} represent a collection of evidence documents. Given a query \boldsymbol{q} , the retriever’s task is to select a subset of relevant documents \mathcal{D}\subset\mathcal{M} , aiming to include those that likely contain the answer to \boldsymbol{q} . Our framework is designed to operate on documents retrieved by arbitrary methods, hence ones that can either utilize sparse or dense representations. Sparse representation methods such as BM25 Robertson et al. (2009) , a non-neural approach, typically rely on term frequency and inverse document frequency to rank documents. This method is effective for scenarios where lexical matching is crucial, providing a strong baseline due to its simplicity and proven efficiency in various information retrieval tasks. Dense representation methods like Dense Passage Retrieval (DPR) Karpukhin et al. (2020b) employs neural network architectures to encode queries and documents into dense vector spaces. The relevance of documents is assessed based on the similarity of these vectors, allowing to capture of semantic relationships that go beyond keyword matching. Regardless of the retrieval technique employed, the retrieval system identifies the top-K most relevant documents, denoted as \mathcal{D}=\{\boldsymbol{d}_{1},\ldots,\boldsymbol{d}_{K}\} . 2.2 Answer Scent Generation Large language models (LLMs) such as GPT-3 Brown et al. (2020) and LLaMA Touvron et al. (2023) define probability distributions over sequences of tokens. Given a sequence x_{1},...,x_{n} , these models typically predict the sequence’s probability using an autoregressive approach p(x_{1},...,x_{n})=\prod_{i=1}^{n}p(x_{i}|x_{<i}), where x_{<i}:=x_{1},...,x_{i-1} represents the sequence of tokens preceding x_{i} , also referred to as its prefix . This modeling is implemented via a transformer network parameterized by \theta_{1} , typically employing a causal self-attention mask p(x_{1},...,x_{n})=\prod_{i=1}^{n}p_{\theta_{1}}(x_{i}|x_{<i}), which effectively models the conditional probabilities of each token. In our approach, we incorporate the concept of answer scent , which guides the model in generating answers that are contextually appropriate for the query, inspired by the success of In-Context Learning Brown et al. (2020) ; Ram et al. (2023) ; Dong et al. (2022) . This context is derived using a zero-shot approach, where the model infers the scent without explicit prior training on such task: p(x_{1},...,x_{n})=\prod_{i=1}^{n}p(x_{i}|x_{<i},\mathcal{S}(x_{<i})), where \mathcal{S}(x_{<i}) denotes the inferred answer scent in the form of a generated text content. The objective of Scent is to encode the essence of what the answer should represent, enriching the input provided to the LLM reranker. 2.3