MSRS : Evaluating Multi-Source Retrieval-Augmented Generation

Rohan Phanse, Yijie Zhou, Kejian Shi, Wencai Zhang, Yixin Liu · 2025

Read MSRS : Evaluating Multi-Source Retrieval-Augmented Generation by Rohan Phanse, Yijie Zhou, Kejian Shi, Wencai Zhang, Yixin Liu (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2508-20867.txt.

Introduction

Rohan Phanse and co-authors built benchmarks for AI that must weave answers from many sources in this 2025 paper. Retrieval-augmented systems fetch documents before writing. Most tests let one document hold the whole answer, but real questions scatter facts across sources that no single page covers.

Their framework spins up such tests at scale. Two benchmarks came out of it: one on story synthesis, one on meeting summaries, both pulling from large collections. Trials with varied fetchers and top language models show answer quality riding on fetch quality. Even with perfect fetching, weaving stays hard. Reasoning models, though, clearly out-wove standard ones at that step.

Catalog record

CREATORRohan Phanse, Yijie Zhou, Kejian Shi, Wencai Zhang, Yixin Liu
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE77 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2508-20867

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

MSRS : Evaluating Multi-Source Retrieval-Augmented Generation Rohan Phanse, Yijie Zhou, Kejian Shi, Wencai Zhang, Yixin Liu Abstract Retrieval-augmented systems are typically evaluated in settings where information required to answer the query can be found within a single source or the answer is short-form or factoid-based. However, many real-world applications demand the ability to integrate and summarize information scattered across multiple sources, where no single source is sufficient to respond to the user’s question. In such settings, the retrieval component of a RAG pipeline must recognize a variety of relevance signals, and the generation component must connect and synthesize information across multiple sources. We present a scalable framework for constructing evaluation benchmarks that challenge RAG systems to integrate information across distinct sources and generate long-form responses. Using our framework, we build two new benchmarks on M ulti- S ource R etrieval and S ynthesis: MSRS-Story and MSRS-Meet , representing narrative synthesis and summarization tasks, respectively, that require retrieval from large collections. Our extensive experiments with various RAG pipelines—including sparse and dense retrievers combined with frontier LLMs—reveal that generation quality is highly dependent on retrieval effectiveness, which varies greatly by task. While multi-source synthesis proves challenging even in an oracle retrieval setting, we find that reasoning models significantly outperform standard LLMs at this distinct step. https://github.com/yale-nlp/MSRS 1 Introduction Traditional Retrieval-Augmented Generation (RAG) systems are often applied in settings where the necessary information to answer a query is contained within a single retrieved source ( Fan et al. 2024 ) . However, many real-world applications often require capturing and synthesizing information scattered across many sources. In these scenarios, the required information is often distributed, complementary, or even contradictory across sources, demanding multi-document retrieval and reasoning capabilities Tang & Yang 2024 ; Lála et al. 2023 ; Asai et al. 2024b ; Zhao et al. 2025 ; Su et al. 2024 ; Wang et al. 2025 . Specifically, the earlier testbeds of RAG systems, such as multi-hop QA ( Yang et al. 2018 ; Khashabi et al. 2018 ; Trivedi et al. 2022 ) are limited to task settings with two-hop questions and short-form answers. More recent studies have introduced benchmarks with increased query complexities and reasoning requirements ( Fan et al. 2024 ) . However, they are mostly restricted to short answers ( Tang & Yang 2024 ) or rely on synthesizing information from a narrow set of source documents ( Edge et al. 2024 ) . We argue that open-domain multi-document summarization (MDS) is a more challenging and realistic testbed, as it requires RAG systems to retrieve and synthesize key information from multiple, complementary long-form documents. In addition, this setting specifically targets the generation capabilities of LLMs for comprehensive, long-form summaries, rather than short-form multi-hop question-answering Trivedi et al. 2022 ; Tang & Yang 2024 . Open-domain multi-document summarization Liu et al. 2018 ; Giorgi et al. 2023 is a retrieval-based task, consisting of joint retrieval and summarization, where relevant documents to an information need or query are first retrieved, and then a summary of relevant information is generated and provided to the user. The source documents have to be retrieved from a large corpus of documents (as opposed to regular multi-document summarization Fabbri et al. 2019 where the set of related input documents is already provided). In addition, such information may be disjoint, overlapping, or complementary between multiple documents. Figure 1 : Overview of the creation process of MSRS-Story and MSRS-Meet datasets. While this task can be a suitable testbed for evaluating RAG systems in multi-source settings, progress is impeded by a lack of realistic benchmarks due to the difficulty of collecting data that includes both multiple complementary relevant documents to a given query and a corresponding ground-truth output. We aim to bridge this gap, particularly focusing on the limitations observed in recent work on open domain summarization (§ 2 ). Specifically, two primary limitations Tang et al. 2021 ; Giorgi et al. 2023 include the use of unrealistic, pseudo-queries (using the reference summary of documents or document titles as the retrieval query) and the sub-optimal construction of the document corpus. To address these limitations, we propose a new method for data collection and introduce two new benchmarks for M ulti- S ource R etrieval and S ynthesis: MSRS-Story and MSRS-Meet (§ 3 ). Bootstrapped from existing long-context, query-focused summarization datasets, our benchmarks aim to meet the main requirements of a realistic multi-source RAG task: (1) necessitate retrieving and synthesizing information from a large corpus; (2) the information to address the query must come from multiple documents; and (3) the queries should be realistic, reflecting a user’s real-world information need. We experiment with a wide range of RAG pipelines, which include sparse and dense retrievers, and state-of-the-art LLMs as generators. The effectiveness of our tested retrievers varies significantly, highlighting the need for better solutions to address document quality and relevance variability challenges in complex RAG problems. Furthermore, the final generation performance is highly dependent on retrieval quality. The multi-source synthesis step remains challenging, even in an oracle setting of perfect retrieval, but achieves meaningful gains when using reasoning models over standard LLMs. Our contributions are summarized below: • We propose a scalable approach (§ 3 ) to construct multi-source RAG datasets from query-focused MDS datasets. • We release MSRS-Story and MSRS-Meet , two benchmarks designed to test RAG systems in retrieving complementary information from large data sources and providing a coherent answer to the user query. • We conduct extensive experiments on RAG system components, specifically retrieval (§ 4.1 ) and generation (§ 4.2 ), providing insights into their capabilities and limitations. 2 Related Work Query-Focused MDS and Retrieval-Augmented Generation. Multi-document summarization (MDS) is a classic NLP task where models synthesize information from multiple input sources ( Yasunaga et al. 2017 ; Liu et al. 2018 ; Fabbri et al. 2019 ; Liu & Lapata 2019 ; Li et al. 2020 ; Lu et al. 2020 ; DeYoung et al. 2021 ) . Query-Focused MDS (QMDS) specializes this task by generating summaries from given documents that address a specific user query Kulkarni et al. 2020 ; Pasunuru et al. 2021 . When QMDS operates in open domains (also known as Open-Domain MDS Giorgi et al. 2023 ), it requires first retrieving relevant information before summarization. This setting provides a natural testbed for RAG systems when answers span multiple documents. While similar to recent deep-research systems OpenAI 2025d , open-domain MDS typically addresses narrower information needs and produces concise summaries rather than long, detailed reports seen in the output of deep research systems. High-quality QMDS datasets remain scarce. Existing benchmarks like AQuaMuSe ( Kulkarni et al. 2020 ) and QMDSIR ( Pasunuru et al. 2021 ) automatically retrieve documents through corpus mining or web search but lack gold-standard relevance labels. QMDSCNN ( Pasunuru et al. 2021 ) repurposes existing datasets by treating titles as pseudo-queries and using four retrieved documents per query, though this approach cannot guarantee document relevance or authentic information needs. Other benchmarks ( Giorgi et al. 2023 ) suffer from unrealistic queries—using ground-truth summaries instead of real-world questions—and sparse corpora that yield many irrelevant documents. In contrast, MSRS presents more significant challenges in information retrieval, necessitating models to first

Read the complete text