DynRank: Improving Passage Retrieval with Dynamic Zero-Shot Prompting Based on Question Classification
Abdelrahman Abdallah, Jamshid Mozafari, Bhawna Piryani · 2024
Read DynRank: Improving Passage Retrieval with Dynamic Zero-Shot Prompting Based on Question Classification by Abdelrahman Abdallah, Jamshid Mozafari, Bhawna Piryani (2024) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2412-00600.txt.
Introduction
Abdelrahman Abdallah and co-authors present DynRank, a framework that adapts search prompts to the type of question being asked. Systems for open-domain question answering retrieve passages to ground each answer. Most systems use one static prompt for every question. That single prompt fits some question types well and fits others poorly. DynRank first classifies each question into a fine-grained type with a trained classification model.
The type then selects a prompt shaped for that kind of question, with no new training runs. The framework plugs into existing retrieval systems as a first step. Tests on several question answering benchmark datasets measure the gain. Prompts matched to the question type retrieve better passages than one-size prompts, the team reports. The gain is largest where question types differ the most.
Catalog record
| CREATOR | Abdelrahman Abdallah, Jamshid Mozafari, Bhawna Piryani |
|---|---|
| YEAR | 2024 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 32 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2412-00600
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
DynRank: Improving Passage Retrieval with Dynamic Zero-Shot Prompting Based on Question Classification Abdelrahman Abdallah, Jamshid Mozafari, Bhawna Piryani Abstract This paper presents DynRank , a novel framework for enhancing passage retrieval in open-domain question-answering systems through dynamic zero-shot question classification. Traditional approaches rely on static prompts and pre-defined templates, which may limit model adaptability across different questions and contexts. In contrast, DynRank introduces a dynamic prompting mechanism, leveraging a pre-trained question classification model that categorizes questions into fine-grained types. Based on these classifications, contextually relevant prompts are generated, enabling more effective passage retrieval. We integrate DynRank into existing retrieval frameworks and conduct extensive experiments on multiple QA benchmark datasets. 1 Introduction Document retrieval plays a crucial role in many NLP tasks, particularly in open-domain question-answering (ODQA) systems Gruber et al. 2024 ; Abdallah et al. 2023 ; Piryani et al. 2024 . In ODQA, a passage is first retrieved and then processed to answer a question. In these systems, the retriever component initially retrieves the most relevant passages from document resources like Wikipedia relevant to the posed question. Subsequently, the reader component examines the retrieved passages to detect an answer. This pipeline causes that the effectiveness of ODQA systems heavily relies on the quality and efficiency of the retriever. Recent advancements in NLP have focused on enhancing the performance of the retriever component by reranking the retrieved passages to improve the likelihood of retrieving the most relevant content and appear on the top Mozafari et al. 2024 ; Abdallah and Jatowt 2023 ; Abdallah et al. 2024 ; Pradeep et al. 2023a ; Sachan et al. 2022 ; Sun et al. 2023 ; Zhuang et al. 2024 ; Pradeep et al. 2023b . Figure 1: Top-1 Accuracy after re-ranking the top 1,000 passages retrieved by BM25 with DynRank comparing with UPR Sachan et al. 2022 on the Natural Questions TriviaQA and WebQA datasets. With the development of large language models (LLMs), many researchers focused on employing LLMs to rerank retrieved passages using diverse strategies and prompting techniques Sun et al. 2023 . Despite the success of LLM-based reranking methods such as UPR Sachan et al. 2022 which generate a question from on each retrieved passage and then re-rank the passages based on how similar their generated questions are to the original question, one significant limitation remains. These models often rely on static and pre-defined instructions during reranking, which can reduce their adaptability across diverse queries and contexts. This static nature restricts the dynamic interaction between the query and retrieved passages, potentially leaving relevant information unused. While methods like RankGPT Sun et al. 2023 have shown performance by generating permutations of passages based on relevance, they still follow a static instruction approach. RankGPT excels in zero-shot passage re-ranking tasks by employing a fixed set of instructions, but it does not dynamically adjust based on the query context. In contrast, our approach aims to overcome this limitation by incorporating more dynamic interactions between the queries and retrieved passages. Similar to UPR Sachan et al. 2022 it generates questions from passages to be used for passage reranking; yet this generation is conditioned by the inferred categories of a target question. A detailed discussion of related works is provided in Appendix D . In response to the above-mentioned challenges, we propose DynRank, a novel re-ranking framework designed to dynamically generate prompts tailored to each specific question. We leverage methods from the Question Classification (QC) task Cortes et al. 2020 , which categorizes questions into two groups: coarse-grained, indicating the major question type, and fine-grained, specifying the minor question type. Table 7 in Appenidx F presents the question types in detail. By incorporating a fine-grained question classification model, DynRank adapts to the context of each query, generating prompts that are more relevant and contextually appropriate. Our contributions are threefold: 1. We introduce a dynamic prompting mechanism that leverages question classification to tailor prompts based on the specific context of each query. 2. We integrate DynRank into existing retrieval framework and demonstrate its compatibility and ease of implementation. 3. We conduct extensive experiments on benchmark QA datasets, demonstrating that DynRank outperforms both traditional and state-of-the-art re-ranking methods. Figure 2: The architecture of the proposed DynRank framework. First, retrieve relevant documents a retriever. The retrieved documents are then re-ranked based on a dynamically generated prompt, tailored to the question’s classification into major ( L_{maj} ) and minor ( L_{min} ) types. A large language model (LLM) processes this prompt to re-rank the top-k documents, improving retrieval accuracy. 2 Method Figure 2 presents an overview of our approach for open-domain retrieval, introducing a novel dynamic zero-shot question generation method ( DynRank ) for re-ranking passages retrieved by any existing retriever. 2.1 Retriever Let \mathcal{D}=\{\boldsymbol{d}_{1},\ldots,\boldsymbol{d}_{M}\} be a collection of evidence documents. Given a question \boldsymbol{q} , the retriever selects a subset of relevant passages \mathcal{Z}\subset\mathcal{D} , one or more of which ideally contains the answer to \boldsymbol{q} . Our method works with passages obtained from any retriever — either based on sparse representations like BM25 or dense representations like DPR. We assume that the retriever provides the K most relevant passages, denoted as \mathcal{Z}=\{\boldsymbol{z}_{1},\ldots,\boldsymbol{z}_{K}\} . 2.2 Question Classification We fine-tune the RoBERTa model Liu et al. 2019 on the UIUC dataset Li and Roth 2002 to build a question classifier. This fine-tuned model categorizes questions based on different levels of granularity, including 5 coarse-grained classes and 50 fine-grained classes. Given an input question \boldsymbol{q} , the question classification model assigns it to a major type l_{maj} and a minor subtype l_{min} . Formally, let Q be the set of all possible questions, and L_{maj} and L_{min} be the sets of major and minor types, respectively. The classification function \mathcal{C}:Q\rightarrow L_{maj}\times L_{min} maps each question to its corresponding type and subtype: (l_{maj},l_{min})=\mathcal{C}(\boldsymbol{q}) (1) 2.3 Dynamic Prompt Generation The dynamic prompt generator constructs a prompt p tailored to each question based on its classification result (l_{maj},l_{min}) . Let \mathcal{T} be a template function that generates prompts. The prompt p is generated as follows: p=\mathcal{T}(l_{maj},l_{min}) (2) For instance, Given the major type l_{maj} as "human" and the minor type l_{min} as "individual," the generated prompt for the passage could be: Document: [passage]. The above Document is about humans, specially on individuals, please write a question based on humans. 2.4 Re-ranking with Pre-trained Language Models Given the dynamically generated prompt p and a set of retrieved passages \mathcal{Z}=\{\boldsymbol{z}_{1},\ldots,\boldsymbol{z}_{K}\} for the question \boldsymbol{q} , the goal of the re-ranking module is to reorder the passages such that the ones containing the correct answer are ranked higher. We define the relevance score s_{i} for each passage \boldsymbol{z}_{i} as the log-likelihood of generating the question \boldsymbol{q} given \boldsymbol{z}_{i} and p . We estimate the relevance score s_{i} as the conditional probability of the question \boldsymbol{q} given the passage \boldsymbol{z}_{i} and the prompt p using a pre-trained language model. The scoring function \mathcal{S} is defined as: s_{i}=\mathcal{S}(\boldsymbol{q}\mid\boldsymbol{z}_{i},p) (3) The log-likelihood \log P(\boldsymbol{q}\mid\boldsymbol{z}_{i},p) is then computed as: \log P(\boldsymbol{q}\mid\boldsymbol{z}_{i},p)=\frac{1}{|\boldsymbol{q}|}\sum_{t=1}^{|\boldsymbol{q}|}\log P(q_{t}\mid\boldsymbol{q}_{<t},\boldsymbol{z}_{i},p;\Theta) (4) where q_{t} is the t -th token of the question \boldsymbol{q} , \boldsymbol{q}_{<t} represents all tokens before q_{t} , and \Theta denotes the parameters of the pre-trained language model. The passages are then re-ranked based on their