MRAG: A Modular Retrieval Framework for Time-Sensitive Question Answering
Siyue Zhang ♢♡ Yuxiang Xue ♠ Yiming Zhang ♣ Xiaobao Wu ♢ Anh Tuan Luu ♢ Chen Zhao ♠★ · 2024
Read MRAG: A Modular Retrieval Framework for Time-Sensitive Question Answering by Siyue Zhang ♢♡ Yuxiang Xue ♠ Yiming Zhang ♣ Xiaobao Wu ♢ Anh Tuan Luu ♢ Chen Zhao ♠★ (2024) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2412-15540.txt.
Introduction
Siyue Zhang and co-authors build a retrieval framework for questions whose answers depend on time, in a paper from 2024. Language models often miss facts that hinge on dates and order. Updating the model itself is costly, so systems fetch outside evidence instead. But standard retrievers falter when a question needs real time-based reasoning.
The team first built TempRAGEval, a benchmark that twists existing questions with time shifts and marks the true evidence. All current retrievers struggled on it. Their framework, MRAG, splits each question into content and a time constraint, fetches and summarizes evidence, then ranks it by both meaning and time fit. With no training at all, it beat rival retrievers and lifted final answer accuracy.
Catalog record
| CREATOR | Siyue Zhang ♢♡ Yuxiang Xue ♠ Yiming Zhang ♣ Xiaobao Wu ♢ Anh Tuan Luu ♢ Chen Zhao ♠★ |
|---|---|
| YEAR | 2024 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 47 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2412-15540
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
MRAG: A Modular Retrieval Framework for Time-Sensitive Question Answering Siyue Zhang ♢♡ Yuxiang Xue ♠ Yiming Zhang ♣ Xiaobao Wu ♢ Anh Tuan Luu ♢ Chen Zhao ♠★ Abstract Understanding temporal concepts and answering time-sensitive questions is crucial yet a challenging task for question-answering systems powered by large language models (LLMs). Existing approaches either update the parametric knowledge of LLMs with new facts, which is resource-intensive and often impractical, or integrate LLMs with external knowledge retrieval ( i.e., retrieval-augmented generation). However, off-the-shelf retrievers often struggle to identify relevant documents that require intensive temporal reasoning. To systematically study time-sensitive question answering, we introduce the TempRAGEval benchmark, which repurposes existing datasets by incorporating complex temporal perturbations and gold evidence labels. As anticipated, all existing retrieval methods struggle with these temporal reasoning-intensive questions. We further propose Modular Retrieval (MRAG), a trainless framework that includes three modules: (1) Question Processing that decomposes question into a main content and a temporal constraint; (2) Retrieval and Summarization that retrieves, splits, and summarize evidence passages based on the main content; (3) Semantic-Temporal Hybrid Ranking that scores semantic and temporal relevance separately for each fine-grained evidence. On TempRAGEval , MRAG significantly outperforms baseline retrievers in retrieval performance, leading to further improvements in final answer accuracy. 1 1 1 Our code and data are available at https://github/anonymous . 1 Introduction Figure 1: A time-sensitive question example that requires temporal reasoning ( as of 6 May 2021 \rightarrow 2019 - 2022 ) to both retrieve documents and generate answers. State-of-the-art retrieval systems struggle to conduct in-depth reasoning to identify relevant documents. We provide a new diagnostic benchmark TempRAGEval , and propose a new modular framework to tackle this challenge. Facts are constantly evolving in our ever-changing world. This dynamic nature highlights the need for natural language processing (NLP) systems capable of updating information ( Liška et al., 2022 ; Zhang et al., 2024 ; Kasai et al., 2024 ) and providing accurate responses to time-sensitive questions ( Chen et al., 2021 ; Chu et al., 2024 ) . For instance, a common query like “Who is the UK Prime Minister?” sees the answer transition from “ Boris Johnson ” to “ Rishi Sunak ” in 2022 (Figure 1 ). With developments of large language models (LLMs), existing approaches rely on the parametric knowledge of LLMs to answer time-sensitive questions directly, and constantly update the parametric knowledge on new facts ( Rozner et al., 2024 ; Wu et al., 2024a ; Wang et al., 2024 ) . However, updating LLM parameters are often resource-intensive. An alternative line of research explores Retrieval-Augmented Generation (RAG), which integrates LLMs with external knowledge ( e.g., Wikipedia) through information retrieval ( Izacard et al., 2020 ) . While RAG allows for the incorporation of new facts with minimal effort, its performance heavily relies on off-the-shelf retrieval systems, which are often limited to keywords or semantic matching. Time-sensitive questions, however, often require intensive temporal reasoning to identify relevant documents, i.e., reasoning-intensive retrieval ( Su et al., 2024a ) . For example, in Figure 1 , retrievers should infer the date “ 24 July 2019 ” as relevant to the constraint “ as of 6 May 2021 ”, rather than only match with date like “ May 2021 ”. Although temporal QA is a well-established domain, there remains a lack of research for temporal reasoning-intensive retrieval systems. We fill that lacuna. We begin by conducting a diagnostic evaluation of existing retrieval approaches for temporal reasoning-intensive retrieval. Following the idea of systematic evaluation with contrast set ( Gardner et al., 2020 ) , we repurpose two existing datasets, TimeQA ( Chen et al., 2021 ) and SituatedQA ( Zhang and Choi, 2021 ) , to introduce the Temp oral QA for RAG Eval uation benchmark ( TempRAGEval ). We manually augment the test questions with complex temporal perturbations ( e.g., modifying the time period to avoid textual overlap). In addition, we annotate gold evidence on Wikipedia for more accurate retrieval evaluation. We observe remarkable degradation of retrieval performance, which points out a serious robustness issue for existing retrievers. To address this issue, we propose a training-free Modular Retrieval framework (MRAG) to enhance temporal reasoning-intensive retrieval. Complementary to recent agentic RAG methods such as R1-Searcher and Search-o1 ( Song et al., 2025 ; Li et al., 2025 ) , MRAG provides an improved retrieval component and reduces the number of invocations for retrieval when handling complex temporal questions. Specifically, MRAG contains three key modules: (1) Question Processing , which decomposes each question into a main content and a temporal constraint; (2) Retrieval and Summarization , which utilizes off-the-shelf retrievers to find evidence passages based on the main content, segments them into independent sentences, and guides LLMs to condense the most relevant passages into query-specific sentences.(3) Semantic-Temporal Hybrid Ranking , which ranks each evidence sentence using a combination of a semantic score measuring semantic similarity, and a temporal score, a novel symbolic component that assesses temporal relevance to the query’s temporal constraint. On TempRAGEval , our proposed MRAG framework achieves substantial improvements in performance, with 9.3% top-1 answer recall and 11% top-1 evidence recall. We also incorporate state-of-the-art (SOTA) answer generators ( Asai et al., 2024 ; Yan et al., 2024 ; Wei et al., 2022 ) , and demonstrate that the improvements in retrieval from MRAG propagate to enhanced final QA accuracy, with 4.5% for both exact match and F1. Detailed case studies further confirms MRAG’s robustness to temporal perturbations qualitatively. Our contributions can be summarized as follows. • We introduce TempRAGEval , a time-sensitive RAG benchmark to diagnostically evaluate each component of existing retrieval-augmented generation systems. • We propose MRAG, a modular retrieval framework to separately determine semantic and temporal relevance. • On TempRAGEval , MRAG significantly outperforms all baseline retrieval systems, and the improvements lead to better answer generation. 2 Background In this section, we first define the time-sensitive question answering task (§ 2.1 ), and then introduce the baseline retrieval-augmented generation QA based systems (§ 2.2 ). 2.1 Temporal Question Answering There has been extensive research on Temporal Question Answering over Knowledge Graphs (TKGQA). Prior works have developed knowledge graph-based benchmarks (e.g., CronQuestions ( Saxena et al., 2021 ) and TimeQuestions ( Jia et al., 2021 ) ), graph neural network models ( Chen et al., 2022 ; Liu et al., 2023 ; Sharma et al., 2023 ) , and TKG-augmented LLMs ( Qian et al., 2024 ) . However, these methods heavily rely on knowledge graphs, which are costly to build, domain-specific, and prone to becoming outdated ( Kau et al., 2024 ) . Therefore, we focus on the free-text setting, which is more universal and up-to-date. Instead of building temporal-specific graph models, we improve general-purpose text retrievers for temporal queries. We use Wikipedia as the main knowledge corpus D . 2 2 2 December 2021 Wikipedia dump, comprising 33.1 million text chunks. Our approach is broadly applicable to other collections, such as the New York Times Annotated Archive ( Sandhaus, 2008 ) and ClueWeb ( Overwijk et al., 2022 ) . 2.2 Retrieval-Augmented Generation The goal of RAG is to address the limitations in the parametric knowledge of LLMs by incorporating external knowledge. Passage retrieval