Scaling External Knowledge Input Beyond Context Windows of LLMs via Multi-Agent Collaboration
Zijun Liu, Zhennan Wan, Peng Li, Ming Yan, Fei Huang, Yang Liu · 2025
Read Scaling External Knowledge Input Beyond Context Windows of LLMs via Multi-Agent Collaboration by Zijun Liu, Zhennan Wan, Peng Li, Ming Yan, Fei Huang, Yang Liu (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2505-21471.txt.
Introduction
Zijun Liu and co-workers present ExtAgents, a multi-agent framework that lets language models use more outside knowledge than fits in one context window, in a 2025 paper. Retrieved facts can improve hard tasks, but the window size caps how much a model can read at once. Longer windows lose information.
Their system splits the knowledge across cooperating agents, fixing two bottlenecks they found in past designs. It needs no extra training for long contexts and runs tasks in parallel for speed. Tests on a harder multi-hop question set and on survey writing showed clear gains over rival no-training methods. Gains held whether the knowledge fit the window or not.
Catalog record
| CREATOR | Zijun Liu, Zhennan Wan, Peng Li, Ming Yan, Fei Huang, Yang Liu |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 82 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2505-21471
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Scaling External Knowledge Input Beyond Context Windows of LLMs via Multi-Agent Collaboration Zijun Liu, Zhennan Wan, Peng Li, Ming Yan, Fei Huang, Yang Liu Abstract With the rapid advancement of post-training techniques for reasoning and information seeking, large language models (LLMs) can incorporate a large quantity of retrieved knowledge to solve complex tasks. However, the limited context window of LLMs obstructs scaling the amount of external knowledge input, prohibiting further improvement. Existing context window extension methods inevitably cause information loss. LLM-based multi-agent methods emerge as a new paradigm to handle massive input in a distributional manner, where we identify two core bottlenecks in existing agent orchestration designs. In this work, we develop a multi-agent framework, ExtAgents , to overcome the bottlenecks and enable better scalability in inference-time knowledge integration without longer-context training. Benchmarked with our enhanced multi-hop question answering test, ∞ \boldsymbol{\infty} Bench+ , and other public test sets including long survey generation, ExtAgents significantly enhances the performance over existing non-training methods with the same amount of external knowledge input, regardless of whether it falls within or exceeds the context window . Moreover, the method maintains efficiency due to high parallelism. We believe further study in the coordination of LLM agents on increasing external knowledge input could benefit real-world applications. 1 1 1 Code and data are available at https://github.com/THUNLP-MT/ExtAgents . 1 Introduction Large Language Models (LLMs) have recently witnessed dramatic progress in parameter scales and context lengths, culminating in context windows that span more than a book-length of text ( DeepSeek-AI, 2025b ; OpenAI, 2025b ; Anthropic, 2025 ) . Yet even these impressive limits remain insufficient for many real-world tasks—multi-hop question answering with Internet, reasoning over enterprise knowledge bases, or writing surveys based on massive academic research—where more external knowledge input often results in better outcomes. Especially, recent research on post-training LLMs to generate long chains of thoughts on reasoning ( OpenAI, 2025d ; DeepSeek-AI, 2025a ) and information seeking ( OpenAI, 2025a ; Li et al., 2025 ; Song et al., 2025 ; Jin et al., 2025 ) tasks, has shown that increasing the amount of retrieved knowledge within the context window could lead to better task performance ( Yue et al., 2025b ) . Figure 1: Performance of scaling external knowledge input with ExtAgents (Ours) and LLM \times MapReduce ( Zhou et al., 2025 ) on \infty Bench+ (detailed in Figure 4 ). For larger input beyond the context window length, the situation is more complicated. When such knowledge is crudely truncated, essential evidence is lost and downstream performance suffers. A natural solution is to train ever longer-context models ( Chen et al., 2023b ; Peng et al., 2024 ; Xu et al., 2025 ; Shang et al., 2025 ) , but this is economically prohibitive and practically brittle: (i) the quadratic complexity of attention ( Vaswani et al., 2017 ) becomes intractable; and (ii) longer-context training data is scarce. Consequently, practitioners turn to retrieval-augmented generation (RAG) ( Lewis et al., 2020 ; Gao et al., 2024 ; Packer et al., 2024 ) or context compression ( Jiang et al., 2024 ; Qian et al., 2024 ; Xiao et al., 2024b ; Wang et al., 2024c ; Hao et al., 2025 ) pipelines. Unfortunately, both strategies inevitably introduce information loss: RAG is limited by ranking errors that could exclude essential evidence during retrieval, while compressors may discard subtle cues that are only useful once the reasoning chain unfolds. Recent approaches ( Trivedi et al., 2023 ; Zhao et al., 2024b ; Zhang et al., 2024b ; Zhou et al., 2025 ) let LLM-based agents collaborate to process long contexts distributedly, reaching state-of-the-art performance on long-context tasks. In this work, we take a step further by asking a question: Could LLMs consistently improve task performance by scaling the amount of external knowledge input beyond the context window? Achieving high scalability of external knowledge implies two requirements: (i) a scalable context extension method needs to accept the massive input, and (ii) the knowledge should be effectively integrated in the orchestration of LLMs and agents. Since it is impractical to re-train short-context LLMs, we mainly focus on the scalability of inference-time knowledge integration beyond context windows . We focus on a few tasks that require massive external knowledge, including multi-hop question answering (QA), both over long documents and large knowledge bases, and long survey generation. We found current benchmarks on long-context tasks constructed with biases, that a quantity of queries could be answered by sweeping a small context window over the attached document. For comprehensive validation, we enhance the existing long-context benchmark, \infty Bench ( Zhang et al., 2024a ) , with an automated pipeline, to obtain a long-document-based multi-hop QA evaluation set, \boldsymbol{\infty} Bench+ , alongside with public multi-hop QA ( Yang et al., 2018 ) and long survey generation ( Wang et al., 2024d ) benchmarks. In preliminary experiments, we find that the current state-of-the-art LLM-based multi-agent system ( Zhou et al., 2025 ) fails to consistently improve task performance with scaled external knowledge input, and even degrades the performance compared to truncated input (Figure 1 ). We systematically analyzed existing multi-agent methods, and then spotted two core bottlenecks in the shared designs of agent orchestration: (i) knowledge synchronization that agents comprehend the distributed contexts and provide condensed information, where the bottleneck is the “bandwidth” of accessible agents for each agent. and (ii) knowledge-integrated reasoning , where the bottleneck is the ratio of redundant information in the reasoning process. To overcome the bottlenecks, we develop a scalable multi-agent framework, ExtAgents . Following prior distributional paradigm, the framework partitions the full input into agent-specific context chunks, each sized to fit a small window. ExtAgents simplifies the roles of agents into two: Seeking Agents and Reasoning Agent; featuring two key components: global knowledge synchronization , where Seeking Agents to globally exchange and update salient intermediate results instead of locally sharing entire context chunks ( Zhao et al., 2024b ; Zhou et al., 2025 ) , and knowledge-accumulating reasoning , which gradually integrates and increases the updated knowledge from Seeking Agents to Reasoning Agent throughout multiple rounds of reasoning. We demonstrate the effectiveness and efficiency of ExtAgents with comprehensive experiments on the aforementioned benchmarks. We show that ExtAgents consistently improves task performance with scaled external knowledge input, outperforming the state-of-the-art non-training methods and achieves increasing performance when the input exceeds context windows. We show the generalization of ExtAgents across different QA and long generation tasks, and its compatibility with different LLM families. We also measure the efficiency gain of ExtAgents from high parallelism. In summary, our contributions are: • We introduce and define the problem of scaling external knowledge input beyond context windows , filling a critical gap in current LLM deployment. We also construct an enhanced multi-hop QA benchmark, \boldsymbol{\infty} Bench+ , for corresponding evaluation. • We systematically study existing LLM-based multi-agent systems for context window extension, and overcome their bottlenecks by proposing a novel framework, ExtAgents . • We demonstrate the effectiveness and efficiency of ExtAgents on QA and survey generation tasks. With external knowledge input scaling beyond context windows, it consistently improves task performance and significantly outperforms baseline methods. 2 Related Work Context Window Extension Methods for