Beyond Relevance: On the Relationship Between Retrieval and RAG Information Coverage DOI: 10.1145/1122445.1122456 Conference: SIGIR ’26: International ACM SIGIR Conference on Research and Development in Information Retrieval; July 20–24, 2026; Naarm, Australia SIGIR ’26: ACM SIGIR Conference on Research and Development in Information Retrieval, July 20–24, 2026, Naarm, Australia CCS: Information systems Information retrieval diversity CCS: Information systems Summarization CCS: Computing methodologies Natural language generation
Saron Samuel, Alexander Martin, Eugene Yang, Andrew Yates, Dawn Lawrie, Ian Soboroff, Laura Dietz, Benjamin Van Durme · 2026
Read Beyond Relevance: On the Relationship Between Retrieval and RAG Information Coverage DOI: 10.1145/1122445.1122456 Conference: SIGIR ’26: International ACM SIGIR Conference on Research and Development in Information Retrieval; July 20–24, 2026; Naarm, Australia SIGIR ’26: ACM SIGIR Conference on Research and Development in Information Retrieval, July 20–24, 2026, Naarm, Australia CCS: Information systems Information retrieval diversity CCS: Information systems Summarization CCS: Computing methodologies Natural language generation by Saron Samuel, Alexander Martin, Eugene Yang, Andrew Yates, Dawn Lawrie, Ian Soboroff, Laura Dietz, Benjamin Van Durme (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2603-08819.txt.
Introduction
Saron Samuel and colleagues test whether retrieval scores predict how much ground a generated answer covers. Across two text benchmarks and one video benchmark, with many retrieval stacks and four pipelines, coverage-based retrieval metrics track nugget coverage in outputs at topic and system levels. The link holds best when retrieval aims match generation aims, though iterative pipelines partly loosen the tie.
Catalog record
| CREATOR | Saron Samuel, Alexander Martin, Eugene Yang, Andrew Yates, Dawn Lawrie, Ian Soboroff, Laura Dietz, Benjamin Van Durme |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 51849 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2603.08819
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Beyond Relevance: On the Relationship Between Retrieval and RAG Information Coverage DOI: 10.1145/1122445.1122456 Conference: SIGIR ’26: International ACM SIGIR Conference on Research and Development in Information Retrieval; July 20–24, 2026; Naarm, Australia SIGIR ’26: ACM SIGIR Conference on Research and Development in Information Retrieval, July 20–24, 2026, Naarm, Australia CCS: Information systems Information retrieval diversity CCS: Information systems Summarization CCS: Computing methodologies Natural language generation Saron Samuel, Alexander Martin, Eugene Yang, Andrew Yates, Dawn Lawrie, Ian Soboroff, Laura Dietz, Benjamin Van Durme Abstract . Retrieval-augmented generation (RAG) systems combine document retrieval with a generative model to address complex information-seeking tasks like report generation. While the relationship between retrieval quality and generation effectiveness seems intuitive, it has not been systematically studied. We investigate whether upstream retrieval metrics can serve as reliable early indicators of the final generated response’s information coverage. Through experiments across two text RAG benchmarks (TREC NeuCLIR 2024 and TREC RAG 2024) and one multimodal benchmark (WikiVideo), we analyze 15 text retrieval stacks and 10 multimodal retrieval stacks across four RAG pipelines and multiple evaluation frameworks (Auto-ARGUE and MiRAGE). Our findings demonstrate strong correlations between coverage-based retrieval metrics and nugget coverage in generated responses at both topic and system levels. This relationship holds most strongly when retrieval objectives align with generation goals, though more complex iterative RAG pipelines can partially decouple generation quality from retrieval effectiveness. These findings provide empirical support for using retrieval metrics as proxies for RAG performance. 1. Introduction As information systems have moved from merely providing source information for human consumption to providing a synthesized text that summarizes the source information, system architecture has changed dramatically. When the system outputs a ranked list or a search engine result page (SERP), gathering documents that are most likely to be relevant to the user query has been the typical design goal of the systems. The community generally refer to this class of problems as adhoc retrieval . However, when users expect only a piece of text synthesized from the source information, a succinct and well-organized summary is usually more preferable than a simple concatenation of relevant documents ( 66 ) . Similar and related information are expected to be grouped, rephrased, or merged ( 6 ) ; multiple facets should be included if they are all relevant to the user query ( 42 ) . Such preference implies that source documents providing the same set of relevant information, while still equally relevant to the query, are not as useful after the first one has been processed by the downstream summarization model, usually powered by a large language model (LLM) ( 3 ; 17 ; 16 ; 63 ; 11 ) . Therefore, the objective of the information system has changed – it should provide a summary of information covering multiple aspects or facets without redundancy. This kind of retrieval problem is generally being referred as report generation ( 33 ) . These systems are usually implemented with retrieval-augmented generation (RAG), which combines the strengths of document retrieval with LLM generation to address complex information seeking tasks. Adhoc retrieval, in this case, becomes an upstream component in the system by providing source information to generation model. However, given the final objective of report generation, it should be sufficient for the upstream retrieval model to provide a relatively small number of documents that cover all needed aspect in the final generation. To date, these LLMs are not able to faithfully leverage all advertised context window, exhibiting information loss ( 34 ) and hallucination ( 26 ) when exceeding certain effective limit. In fact, as long as the LLM has an effective context window, there is need to filter the collection to a smaller subset of documents for the downstream model. Therefore, minimizing the number of document the generation model needs to process should provide benefits to the final generated responses. The problem of retrieving for multiple aspects with penalization on redundant information has been studied in prior retrieval literature on diversity ranking ( 8 ) , which is equivalent to information coverage ( 67 ) . However, the relationship between information coverage of the upstream retrieval and the final generation quality, though logical, has not been systematically studied. Furthermore, end-to-end evaluation of generated responses requires running complete RAG pipelines, which incurs substantial computational cost. Unlike document-level judgments, judgments of generated responses are not easily reusable, so evaluating them incurs substantial further costs in the form of collecting new judgments from a human or LLM. Moreover, the LLM itself adds variability and noise, as different LLMs and generation strategies can produce divergent outputs even when given identical retrieved context, resulting in noisy signals when attributing effectiveness to each component in the pipeline ( 4 ; 7 ) . In this paper, we aim to provide a systematic study on the relationship between upstream retrieval and the downstream generation quality. Particularly, we focus on information coverage (usually materialized as nugget coverage in recent literature ( 1 ; 46 ) ) of the final generation quality since it is the primary purpose of employing a retrieval model in the generation pipeline. Other qualities, such as fluency and faithfulness, can already be attributed to the LLM itself ( 40 ) , thus are excluded in this study. Particularly, we aim to answer the question: “Is the upstream retrieval quality an early indicator of the information coverage of the downstream generation responses?” With our empirical evidence, we show that there is a strong relationship between the two stages. This evidence provides empirical grounds for simplifying the evaluation of information coverage to focus on the upstream retrieval model, reducing both computational costs and experimental noise. Such a simplification is common in the literature such as focusing on recall for first-stage retrieval model ( 24 ) and precision at one for question-answering and conversational search (which need to be announced to users through audio). Our analysis operates at multiple levels. Topic-level analysis examines whether better retrieval results (or a ranked list) for a specific query lead to a better generated response. System-level analysis assesses whether employing a more effective retrieval models yields a more effective RAG pipeline in general, measuring by information coverage. For robustness, we examine these relationships across different generation pipelines, from simple retrieve-then-generate approaches ( 63 ; 15 ; 30 ) to more complex multi-query and iterative strategies ( 2 ; 9 ) , to understand whether generation complexity can compensate for weaker retrieval. Our contributions are: • We demonstrate that nugget-oriented retrieval metrics serve as reliable indicators of RAG information coverage, with strong correlations observed at both topic and system levels across text and multimodal benchmarks. • We show that RAG pipeline complexity affects the retrieval-generation relationship. Simpler linear pipelines benefit directly from retrieval improvements, while complex iterative pipelines can partially decouple generation quality from retrieval effectiveness by adapting queries to retrieval system capabilities. • We validate our findings across multiple generation strategies (GPT-Researcher, Bullet List, LangGraph), evaluation frameworks (Auto-ARGUE, MiRAGE), and modalities (text and video), demonstrating the robustness and generalizability of coverage-based retrieval metrics as proxies for RAG performance. 2. Background 2.1. Retrieval and Evaluation Traditional adhoc retrieval evaluation has focused on document relevance. Metrics such as MRR, MAP, and nDCG