More Documents, Same Length: Isolating the Challenge of Multiple Documents in RAG
Shahar Levy, Nir Mazor, Lihi Shalmon, Gabriel Stanovsky · 2025
Read More Documents, Same Length: Isolating the Challenge of Multiple Documents in RAG by Shahar Levy, Nir Mazor, Lihi Shalmon, Gabriel Stanovsky (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2503-04388.txt.
Introduction
Adding more documents to a retrieval system hurts language models even when the total text length stays the same, shows a 2025 arXiv study by Shahar Levy and co-authors. The team held context length and the position of the key facts fixed, then varied only how many documents carried the text.
Performance fell by up to 20 percent as document counts rose. One model family held steady, which hints that handling many documents is a skill of its own, apart from handling long text. The authors release their datasets and code, and they argue that multi document reading needs its own tests and training.
Catalog record
| CREATOR | Shahar Levy, Nir Mazor, Lihi Shalmon, Gabriel Stanovsky |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 13 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2503-04388
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
More Documents, Same Length: Isolating the Challenge of Multiple Documents in RAG Shahar Levy, Nir Mazor, Lihi Shalmon, Gabriel Stanovsky Abstract Retrieval-augmented generation (RAG) provides LLMs with relevant documents. Although previous studies noted that retrieving many documents can degrade performance, they did not isolate how the quantity of documents affects performance while controlling for context length. We evaluate various language models on custom datasets derived from a multi-hop QA task. We keep the context length and position of relevant information constant while varying the number of documents, and find that increasing the document count in RAG settings poses significant challenges for LLMs. Additionally, our results indicate that processing multiple documents is a separate challenge from handling long contexts. We also make the datasets and code available 1 1 1 https://github.com/shaharl6000/MoreDocsSameLen to facilitate further research in multi-document retrieval. 1 Introduction Figure 1: More Documents, Same Length. We create various sets containing the same questions but differing in the number of distractor documents. Each set includes a multi-hop question, all of the supporting documents that contain the information to answer the question (pink), and varying distractor documents (blue). We begin with a 20-document version (left) and then reduce the number of documents while maintaining a fixed context size. When fewer documents are used, the remaining documents are extended (blue without text) so that concatenating them yields the same total length. The RAG approach enriches prompts with relevant documents, retrieved according to an input query Karpukhin et al. 2020 . For example, given a question about a certain historical period, RAG techniques can retrieve documents related to the time from a large historical corpus. Recent work has noted a drop in RAG performance when retrieving many documents. For example, in multi-hop QA, LLMs struggle when the number of retrieved documents grows, even when presented with all the needed information Press et al. 2022 ; Liu et al. 2023 ; Levy et al. 2024 ; Wang et al. 2024 . Such deficiencies were observed without controlling for the number of tokens in which the information is conveyed, i.e., when the number of documents grew, so did the number of overall tokens, thus conflating between the challenge of long context and multi document. Figure 2: Increasing the number of retrieved document can hurt performance. In retrieval setups with fixed context windows, adding more documents could reduce performance by up to 10 percent. Two models (Llama- 3.1 and Gemma-2) showed worse performance, while Qwen-2 remained unaffected. The smaller versions of the LLMs (7–9B) show a similar trend as their larger counterparts but the effect is weaker. The hues of the bars represent the amount of retrieved documents. In this work, we address the following question: Assuming a fixed input length, how is LLM performance affected by the number of retrieved documents? This disentangles the challenge of long context from the challenge in processing collections of related documents – which often contain redundancies, conflicting information, and implicit inter-document relations ( Hirsch et al. 2023 ; Lior et al. 2024 ) . From a practical perspective, answering this question can help understand a breadth versus depth tradeoff — i.e., whether to strive to retrieve shorter context out of many documents or whether to aim to retrieve longer context out of fewer documents. An ideal experimental setup would have the exact information conveyed in the same number of tokens across varying number of documents, from a long and self-contained single document to a large, multi-document corpus. We find that the custom sets we constructed from MuSiQue ( Trivedi et al. 2022 ) , a multi-hop QA dataset, serve as a convenient approximation, allowing us to explore the relationship between long-context and multi-document comprehension in a controlled environment with real-world texts. Each instance in MuSiQue consists of a question and a set of 20 documents, where each document is an excerpt from a Wikipedia article retrieved according to the input question. MuSiQue is constructed such that the question can be answered based on only a few of the input documents (2-4), while the other documents serve as realistic distractors in retrieval settings, as they revolve around the question’s topic but do not contain information required to answer the question. We vary the number of documents in the input by gradually removing the distractor documents. When removing a distractor document, we respectively extend each of the remaining documents with distracting content from their corresponding Wikipedia article. Importantly, the process preserves the position of the relevant information within the cotext. This process is illustrated in Fig. 1 . If the context length is the sole challenge, we should expect the performance to remain similar regardless of the number of input documents. Conversely, if processing multiple related documents presents an additional challenge, we would expect an inverse correlation between performance and the number of input documents. The results of evaluating several state-of-the-art models (Llama-3.1, Qwen2, and Gemma2) which are presented in Fig. 2 , indicate that in most cases, reducing the number of documents improves performance by up to 10%. An exception is Qwen2, which may indicate that it better handles multi-document collections. Our work has several major implications and avenues for future work. First, from a practical perspective, RAG systems should take the number of retrieved documents into consideration, as the introduction of additional documents into the prompt may hurt performance. Second, future work should explore novel approaches for multi-document processing, which according to our findings presents a separate challenge from mere long context. Such work can make use of our paradigm and data for training and evaluation. 2 Multi-Document Evaluation with Controlled Token Count Our goal is to understand how the number of retrieved documents affects LLM performance when controlling the input length. To this end, we evaluate several models on multi-document multi-hop question answering, which requires models to find relevant information within a given context to answer a specific question. In particular, we make controlled adjustments to the number of documents in the input, while preserving the position of the key information needed to answer the questions, and keeping the context length consistent. Our dataset is based on the validation set of MuSiQue ( Trivedi et al. 2022 ) , a multi-hop QA dataset that consists of 2,417 answerable questions. Each question is associated with 20 paragraphs sampled from individual documents, retrieved from Wikipedia according to the question. Of these paragraphs, 2–4 contain the supporting information necessary to answer the question, while the remaining paragraphs serve as realistic distractors in a RAG setup, as they are retrieved from related topics but do not contain relevant information to answer the question. Fig. 1 shows an example query, and a list of retrieved documents, where three are relevant to the question (marked in pink), and the rest are distractors (marked in blue). Leveraging MuSiQue’s structure, we constructed several data partitions to investigate the impact of the number of retrieved documents in a controlled manner. The process involved the following steps: 1. Select the total number of documents: We reduced the number of documents from the original 20 to 15, then 10, 8, and finally down to the 2–4 documents consisting of the relevant information to answer the question. 2. Choose the supporting