DeepSieve: Information Sieving via LLM-as-a-Knowledge-Router
Minghao Guo, Qingcheng Zeng, Xujiang Zhao, Yanchi Liu, Wenchao Yu, Mengnan Du, Haifeng Chen, Wei Cheng · 2025
Read DeepSieve: Information Sieving via LLM-as-a-Knowledge-Router by Minghao Guo, Qingcheng Zeng, Xujiang Zhao, Yanchi Liu, Wenchao Yu, Mengnan Du, Haifeng Chen, Wei Cheng (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2507-22050.txt.
Introduction
Language models answer hard questions better when they can look things up, but ordinary lookup often drags in noise. Minghao Guo and co-authors present DeepSieve, a framework that treats retrieval like passing material through a sieve. A big question is first broken into smaller, structured sub-questions. Each piece is then routed to the knowledge source that fits it best.
A language model acts as the router, choosing sources and filtering out what does not help. The sifting happens in stages, so the reasoning stays deep while the evidence stays clean. On multi-hop questions that need facts from mixed sources, the method beat standard retrieval setups on accuracy and clarity. The design is modular, so builders can swap sources in and out.
Catalog record
| CREATOR | Minghao Guo, Qingcheng Zeng, Xujiang Zhao, Yanchi Liu, Wenchao Yu, Mengnan Du, Haifeng Chen, Wei Cheng |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 62 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2507-22050
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
DeepSieve: Information Sieving via LLM-as-a-Knowledge-Router Minghao Guo, Qingcheng Zeng, Xujiang Zhao, Yanchi Liu, Wenchao Yu, Mengnan Du, Haifeng Chen, Wei Cheng Abstract Large Language Models (LLMs) excel at many reasoning tasks but struggle with knowledge-intensive queries due to their inability to dynamically access up-to-date or domain-specific information. Retrieval-Augmented Generation (RAG) has emerged as a promising solution, enabling LLMs to ground their responses in external sources. However, existing RAG methods lack fine-grained control over both the query and source sides, resulting in noisy retrieval, shallow reasoning, and limited adaptability to heterogeneous knowledge sources. In this work, we introduce DeepSieve , a novel RAG method that incorporates information sieving via LLM-as-a-knowledge-router. DeepSieve breaks down complex queries into structured sub-queries and recursively routes each to the most appropriate knowledge source, filtering out irrelevant information through a multi-stage information sieving process. This modular and transparent approach ensures that DeepSieve remains adaptable across diverse information needs. Experiments on three multi-hop QA benchmarks involving heterogeneous sources show that DeepSieve achieves greater reasoning depth, retrieval precision, and interpretability compared to conventional RAG approaches. Our codes are available at https://github.com/MinghoKwok/DeepSieve . 1 Introduction Large language models (LLMs) have achieved remarkable progress across a wide range of natural language tasks and have demonstrated strong reasoning abilities, including in domains such as mathematics Guo et al. 2025 , fake news detection Yi et al. 2025 ; Jin et al. 2024b , and commonsense reasoning Yao et al. 2023 . Yet these LLMs often falter when facing knowledge-intensive questions that require up-to-date or domain-specific information Guu et al. 2020 ; Asai et al. 2024 ; Fan et al. 2024 . This limitation stems from the fixed nature of LLM parameters, which prevents dynamic access to external knowledge and frequently results in hallucinations or factually inaccurate outputs Huang et al. 2023 . Retrieval-Augmented Generation (RAG) has emerged as a powerful paradigm for equipping LLMs with access to external knowledge Lewis et al. 2020 ; He et al. 2023 ; Su et al. 2025 , enabling them to tackle complex information-seeking tasks more effectively. Recent advances such as GraphRAG, HippoRAG, and so on Gutierrez et al. 2024 ; Edge et al. 2024 ; Zhang et al. 2025 demonstrate the promise of structured and memory-augmented retrieval in improving factual accuracy and multi-hop reasoning. Figure 1: Motivation and Overview: Left: Compositional queries are hard to answer under source heterogeneity (e.g., structured, private, and unmergeable databases). Right: DeepSieve performs decomposition, source-aware routing, and iterative fusion to enable structured reasoning. However, existing RAG systems still struggle with a fundamental limitation: the lack of an effective information sieving mechanism across the reasoning pipeline. In practice, this deficiency manifests in two critical forms: query-side sieving and source-side sieving . On the query side, most systems treat user queries as atomic units and directly retrieve information without decomposing or analyzing their underlying semantic structure Trivedi et al. 2023 ; Yao et al. 2023 ; Lin et al. 2025 . This prevents the model from isolating key subgoals or reasoning steps, which is essential for multi-hop or compositional question answering. On the source side, knowledge can be heterogeneous both across different sources (e.g., unstructured corpora, structured APIs, private databases) and within a single source that contains multiple domains or topics. However, most existing RAG systems retrieve from a flat, unified index without considering differences in domain, format, access modality, or granularity Chen et al. 2017 ; Izacard and Grave 2021 ; Gao et al. 2023 . This often results in mismatched content, irrelevant retrievals, and unnecessary computational cost. Moreover, many knowledge sources cannot be merged into a single retrieval index because of privacy constraints, structural incompatibility, or deployment considerations Mavi et al. 2023 ; Xu et al. 2025 , further highlighting the need for selective and modular retrieval strategies. To bridge this knowledge gap, we propose DeepSieve , a framework that performs retrieval-augmented reasoning through multi-stage information sieving. As shown in the Figure 1 , DeepSieve consists of four key components: question decomposition , thought generation , data-source routing , and recursive reflexion . The pipeline begins by decomposing a complex input query into a set of structured sub-questions using an LLM-based planner Lin et al. 2025 ; Trivedi et al. 2022 . Each sub-question is then independently processed to generate a “thought”, a latent reasoning step that guides retrieval planning Yao et al. 2023 ; Xu et al. 2023 . Based on this thought, DeepSieve identifies the most appropriate source (e.g., API, SQL, RAG corpus) and generates a corresponding action plan. This source-aware routing builds on the idea of using LLMs as controllers for modular tool access Schick et al. 2023 ; Xu et al. 2025 . Once evidence is retrieved, the system evaluates whether the information sufficiently addresses the sub-question. If the answer is invalid, DeepSieve enters a reflection loop to reassess the thought, revise the action plan, or reselect the source, iterating until resolution or timeout Madaan et al. 2023 ; Baek et al. 2025 . Finally, after all sub-questions are addressed, DeepSieve aggregates the answers into a coherent output using LLM module, which also generates intermediate rationales Wei et al. 2022 ; Creswell et al. 2023 . This iterative, modular design allows DeepSieve to sieve and integrate knowledge progressively, making it particularly suited for multi-hop reasoning and heterogeneous information access. Empirically, DeepSieve consistently outperforms the baselines across all three benchmarks, achieving average F1/EM scores of 58.9/49.3 with DeepSeek-V3 and 51.2/41.4 with GPT-4o, surpassing both RAG (e.g., HippoRAG Gutierrez et al. 2024 ) and agentic (e.g., ReAct Yao et al. 2023 ) baselines, while using significantly fewer tokens. To sum up, our paper makes the following key contributions: • We identify the structural and semantic heterogeneity of real-world knowledge sources as a core challenge in RAG. To address this, we propose the DeepSieve framework, which introduces information sieving and, for the first time, uses an LLM-as-a-knowledge-router to dynamically decompose queries and dispatch sub-questions to heterogeneous sources. • Our experiments demonstrate that DeepSieve not only satisfies multi-source settings but also improves performance in standard single-source scenarios. Even when operating over a unified corpus, our system yields better retrieval precision and answer accuracy. • We design DeepSieve as a modular and extensible method. It supports plug-and-play integration with diverse tools, retrieval backends, and RAG models. It provides a flexible backbone for future RAG architectures. 2 Methodology Figure 2: DeepSieve workflow with multi-step reasoning: A complex query is first decomposed into diverse subqueries, like a directed acyclic graph (DAG) . For each path in the DAG, an LLM generates a plan to select knowledge sources via routing. Failed retrievals will trigger re-routing or re-decomposition in the workflow. Retrieved subanswers are stored in memory and later fused across paths to form a final answer. We propose DeepSieve , a novel RAG method that performs information sieving across both query and source spaces. DeepSieve addresses the challenge of reasoning over heterogeneous knowledge sources, spanning diverse formats, access modalities, and domains, by employing a layered process of decomposition, routing, reflection, and fusion. As shown in Figure 1 and Algorithm 1 , DeepSieve decomposes input queries into subqueries, routes each to an appropriate (Tool, Corpus) pair, iteratively refines insufficient answers via reflexion, and ultimately fuses