LevelRAG: Enhancing Retrieval-Augmented Generation with Multi-hop Logic Planning over Rewriting Augmented Searchers

Zhuocheng Zhang, Yang Feng, Min Zhang · 2025

Read LevelRAG: Enhancing Retrieval-Augmented Generation with Multi-hop Logic Planning over Rewriting Augmented Searchers by Zhuocheng Zhang, Yang Feng, Min Zhang (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2502-18139.txt.

Introduction

Zhuocheng Zhang, Yang Feng, and Min Zhang improve how chat models search for facts before answering, in a research paper from 2025. Search-backed answering cuts made-up replies, but hard questions often need several search steps. Older methods rewrite the question in ways that clash with some search tools.

Their system, LevelRAG, first splits a hard question into small, plain sub-questions. Separate searchers then handle each one: one for exact keywords, one for web pages, and one for meaning-based matches. Tests on five data sets, with both simple and multi-step questions, show the system beating prior methods and a leading commercial model.

Catalog record

CREATORZhuocheng Zhang, Yang Feng, Min Zhang
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE39 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2502-18139

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

LevelRAG: Enhancing Retrieval-Augmented Generation with Multi-hop Logic Planning over Rewriting Augmented Searchers Zhuocheng Zhang, Yang Feng, Min Zhang Abstract Retrieval-Augmented Generation (RAG) is a crucial method for mitigating hallucinations in Large Language Models (LLMs) and integrating external knowledge into their responses. Existing RAG methods typically employ query rewriting to clarify the user intent and manage multi-hop logic, while using hybrid retrieval to expand search scope. However, the tight coupling of query rewriting to the dense retriever limits its compatibility with hybrid retrieval, impeding further RAG performance improvements. To address this challenge, we introduce a high-level searcher that decomposes complex queries into atomic queries, independent of any retriever-specific optimizations. Additionally, to harness the strengths of sparse retrievers for precise keyword retrieval, we have developed a new sparse searcher that employs Lucene syntax to enhance retrieval accuracy. Alongside web and dense searchers, these components seamlessly collaborate within our proposed method, LevelRAG . In LevelRAG, the high-level searcher orchestrates the retrieval logic, while the low-level searchers (sparse, web, and dense) refine the queries for optimal retrieval. This approach enhances both the completeness and accuracy of the retrieval process, overcoming challenges associated with current query rewriting techniques in hybrid retrieval scenarios. Empirical experiments conducted on five datasets, encompassing both single-hop and multi-hop question answering tasks, demonstrate the superior performance of LevelRAG compared to existing RAG methods. Notably, LevelRAG outperforms the state-of-the-art proprietary model, GPT4o, underscoring its effectiveness and potential impact on the RAG field. 1 Key Laboratory of Intelligent Information Processing, Institute of Computing Technology, Chinese Academy of Sciences (ICT/CAS) 2 Key Laboratory of AI Safety, Chinese Academy of Sciences 3 University of Chinese Academy of Sciences, Beijing, China 4 Harbin Institute of Technology (Shenzhen), Shenzhen, China [email protected], [email protected], [email protected] Code — https://github.com/ictnlp/LevelRAG Introduction Figure 1: Overview of the LevelRAG. The user query is initially processed by the high-level searcher. The decomposed atomic queries are handled by low-level searchers, which may rewrite and refine the queries before sending them into their corresponding retrievers. The retrieved documents are aggregated and summarized by the high-level searcher, then fed to the generator to generate the response. Both the high-level searcher and low-level searchers employs the feedback from the retrieved documents to refine or supplement their outputs. The introduction of scaling laws ( Kaplan et al. 2020 ; Hoffmann et al. 2022 ) has led to significant advancements in Large Language Models (LLMs) ( Brown et al. 2020 ; Touvron et al. 2023a ; Touvron et al. 2023b ; Dubey et al. 2024 ; Abdin et al. 2024 ) , enabling them to achieve remarkable successes across a wide range of tasks ( Zhang et al. 2023 ; Jiang et al. 2024a ; Yang et al. 2023 ; Yao et al. 2023 ; Lee et al. 2024 ) . Despite their extensive training and vast knowledge base, LLMs continue to encounter challenges in accurately handling certain types of information, including less commonly known facts, specialized domain knowledge, and events that occurred after their pre-training phase ( Mallen et al. 2023a ; Huang et al. 2023 ) . In these cases, LLMs often fail to provide accurate responses and may even produce hallucinated answers. Hence Retrieval-Augmented Generation (RAG) is proposed to enhance LLM-generated responses by appending real-time retrieval of relevant external knowledge to user input of LLMs, offering a promising solution to address these challenges ( Gao et al. 2024 ; Ding et al. 2024 ) . However, due to the constraints of the search techniques and the coverage of databases, the retrieval results are often inaccurate and incomplete, affecting the effectiveness of RAG systems. Many researchers have made efforts to optimize RAG systems by improving the accuracy and completeness of retrieval results. To improve accuracy, researchers have developed numerous query rewriting techniques ( Ma et al. 2023 ; Asai et al. 2023 ; Chan et al. 2024 ) which try to refine the query fed to a retriever to make the query better match with the retriever and better align to user intents. On the other hand, to improve completeness, hybrid retrieval is employed to broaden the scope of retrieval, which integrates multiple retrievers and databases to maximize the advantages of diverse retrievers ( Lu et al. 2022 ; Wang et al. 2023 ; Yu et al. 2022 ; Zhang et al. 2020 ; Li et al. 2024 ) . To enhance both the completeness and accuracy of the retrieval process, it is a good choice to combine the superiority of the two methods mentioned above. However, a key challenge arises from the fact that current query rewriting techniques are closely tied to the dense retriever, rendering them unsuitable for hybrid retrieval scenarios. Meanwhile, to achieve complete retrieval results, we need to sort out all the retrieval results for hybrid retrieval to avoid overlapping and contradictory knowledge input to LLMs. Besides, multi-hop planning for retrieval is also necessary as not all the user intents can be satisfied with one retrieval when fed with complex user input. To address these limitations, we propose LevelRAG , a novel approach that decouples retrieval logic from retriever-specific rewriting to allow for greater flexibility for either function to achieve better expertise, and meanwhile involves interactions between them for global optimization. As shown in Figure 1 , LevelRAG is composed of a high-level searcher which is responsible for multi-hop logic planning along with information combing, and multiple low-level searchers which is the operator supporting query rewriting. The high-level searcher conducts planning to decompose the user query into atomic queries for low-level searchers to retrieve, then summarizes the retrieved results of low-level searchers for information combing. In the process of summarization, it also determines whether to invoke next-hop planning to supplement new atomic queries according to the completeness of summarized information for the user input. The low-level searchers operate the real retrieval over databases with atomic queries from the high-level searcher, which may be rewritten to better fit the corresponding retriever. Specifically, we employ a sparse searcher, a dense searcher, and a web searcher as the low-level searchers, among which the sparse searcher is good at exact retrieval for entities with Lucene syntax, the dense searcher is skilled at retrieval with complex queries which are transformed into pseudo-documents, and the web searcher is employed to leverage the vast knowledge from the internet to supplement the local database. In conjunction with the high-level searcher, these searchers collectively guarantee the completeness and accuracy of the retrieval process. We conducted empirical experiments on five datasets, including both single-hop question answering (QA) tasks and multi-hop QA tasks. The experimental results demonstrate that LevelRAG significantly outperforms current RAG methods across these tasks. Notably, the response quality of our proposed method surpasses the state-of-the-art proprietary model, GPT4o, providing strong evidence for the superiority of our approach. Additionally, the mere utilization of our proposed sparse searcher outperforms numerous existing methods, which further reveals the effectiveness of our novel rewriting approach. In conclusion, our key contributions are as follows: • To address the challenge associated with the tight coupling of query rewriting to dense retrievers in hybrid retrieval scenarios, we propose LevelRAG, a method that integrates the high-level searcher with low-level searchers (sparse, web, and dense). This integration allows for optimal retrieval by

Read the complete text