RAG vs. GraphRAG: A Systematic Evaluation and Key Insights
Haoyu Han, Yu Wang, Harry Shomer, Yongjia Lei, Kai Guo, Zhigang Hua, Bo Long, Hui Liu, Jiliang Tang · 2025
Read RAG vs. GraphRAG: A Systematic Evaluation and Key Insights by Haoyu Han, Yu Wang, Harry Shomer, Yongjia Lei, Kai Guo, Zhigang Hua, Bo Long, Hui Liu, Jiliang Tang (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2502-11371.txt.
Introduction
Haoyu Han and co-authors benchmarked in 2025 two ways of giving language models outside facts: plain retrieval and retrieval that follows a graph of linked facts. Plain retrieval is widely used for text tasks. Graph retrieval was built for structured data and has since spread to text, but past tests used mixed setups.
Their study sets one fair protocol for data handling, retrieval, and generation across question answering and summary tasks. Each method showed distinct strengths by task and by measure. Blending the two, by picking or joining them, gave steady gains. Failure modes, speed costs, and bias in scoring are also examined, with advice for future test design.
Catalog record
| CREATOR | Haoyu Han, Yu Wang, Harry Shomer, Yongjia Lei, Kai Guo, Zhigang Hua, Bo Long, Hui Liu, Jiliang Tang |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 44 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2502-11371
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
RAG vs. GraphRAG: A Systematic Evaluation and Key Insights Haoyu Han, Yu Wang, Harry Shomer, Yongjia Lei, Kai Guo, Zhigang Hua, Bo Long, Hui Liu, Jiliang Tang Abstract Retrieval-Augmented Generation (RAG) enhances the performance of LLMs across various tasks by retrieving relevant information from external sources, particularly on text-based data. For structured data, such as knowledge graphs, GraphRAG has been widely used to retrieve relevant information. However, recent studies have revealed that structuring implicit knowledge from text into graphs can benefit certain tasks, extending the application of GraphRAG from graph data to general text-based data. Despite their successful extensions, most applications of GraphRAG for text data have been designed for specific tasks and datasets, lacking a systematic evaluation and comparison between RAG and GraphRAG on widely used text-based benchmarks. In this paper, we systematically evaluate RAG and GraphRAG on well-established benchmark tasks, such as Question Answering and Query-based Summarization. Our results highlight the distinct strengths of RAG and GraphRAG across different tasks and evaluation perspectives. Inspired by these observations, we investigate strategies to integrate their strengths to improve downstream tasks. Additionally, we provide an in-depth discussion of the shortcomings of current GraphRAG approaches and outline directions for future research. 1 Introduction Retrieval-Augmented Generation (RAG) has emerged as a powerful approach to enhance downstream tasks by retrieving relevant knowledge from external data sources. It has achieved remarkable success in various real-world applications, such as healthcare ( Xu et al. 2024 ) , law ( Wiratunga et al. 2024 ) , finance ( Zhang et al. 2023 ) , and education ( Miladi et al. 2024 ) . This success has been further amplified with the advent of Large Language Models (LLMs), as integrating RAG with LLMs significantly improves their faithfulness by mitigating hallucinations, reducing privacy risks, and enhancing robustness Zhao et al. 2023 ; Huang et al. 2023 . In most existing RAG systems, retrieval is primarily conducted from text databases using lexical and semantic search. Graphs, as a fundamental data structure, encode rich relational information and have been extensively utilized across real-world domains, including knowledge representation, social network analysis, and biomedical research Wu et al. 2020 ; Ma and Tang 2021 ; Wu et al. 2023 . Motivated by this, GraphRAG has recently gained attention for retrieving graph-structured data, such as knowledge graphs (KGs) and molecular graphs ( Han et al. 2024 ; Peng et al. 2024 ) . Beyond leveraging existing graphs, GraphRAG has also demonstrated its effectiveness for text-based tasks after structuring implicit knowledge from text into graph representations, benefiting applications such as global summarization Edge et al. 2024 ; Zhang et al. 2024 , planning Lin et al. 2024 and reasoning Han et al. 2025 . While previous studies have demonstrated the potential of GraphRAG for text-based tasks by converting sequential text into graphs, most of them primarily focus on specific tasks and well-designed datasets. Consequently, the applicability of GraphRAG to broader, real-world text-based tasks remains unclear, particularly when compared to RAG, which has seen widespread adoption across diverse applications. This raises a critical question: What are the advantages and disadvantages of applying GraphRAG to general text-based tasks compared to RAG? To bridge this gap, we systematically evaluate the performance of RAG and GraphRAG on general text-based tasks using widely adopted datasets, including Question Answering and Query-based Summarization. Specifically, we assess two representative GraphRAG methods: (1) Knowledge Graph-based GraphRAG Liu 2022 , which extracts a Knowledge Graph (KG) from text and performs retrieval solely based on the KG and (2) Community-based GraphRAG ( Edge et al. 2024 ) , which retrieves information not only from the constructed KG but also from hierarchical communities within the graph. For the Question Answering task, we conduct experiments on both single-hop and multi-hop QA under single-document and multi-document scenarios. Similarly, for the Query-based Summarization task, we evaluate both single-document and multi-document summarization to comprehensively assess the effectiveness of RAG and GraphRAG. Based on our comprehensive evaluation, we conduct an in-depth analysis of the strengths and weaknesses of RAG and GraphRAG across different tasks. Our findings reveal that RAG and GraphRAG are complementary, each excelling in different aspects. For the Question Answering task, we observe that RAG performs better on single-hop questions and those requiring detailed information, while GraphRAG is more effective for multi-hop questions. In the Query-based Summarization task, RAG captures fine-grained details, whereas GraphRAG generates more diverse and multi-faceted summaries. Building on these insights, we investigate two strategies from different perspectives to integrate their unique strengths and enhance the overall performance. Our main contributions are as follows: • Systematical Evaluation : This is the very first work to systematically evaluate and compare RAG and GraphRAG on text-based tasks using widely adopted datasets and evaluations. • Task-Specific Insights : We provide an in-depth analysis of the distinct strengths of RAG and GraphRAG, demonstrating their complementary advantages across different types of queries and objectives. • Hybrid Retrieval Strategies: Based on our findings on the unique strengths of RAG and GraphRAG, we propose two strategies to improve overall performance: (1) Selection, where queries are dynamically assigned to either RAG or GraphRAG based on their characteristics, and (2) Integration, where both methods are integrated to leverage their complementary strengths. • Challenges and Future Directions: We discuss the limitations of current GraphRAG approaches and outline potential future research directions for broader applicability. 2 Related Works 2.1 Retrieval-Augmented Generation Retrieval-Augmented Generation (RAG) has been widely applied to enhance the performance of Large Language Models (LLMs) by retrieving relevant information from external sources, addressing the limitation of LLMs’ restricted context windows, improving factual accuracy, and mitigating hallucinations Fan et al. 2024 ; Gao et al. 2023 . Most RAG systems primarily process text data by first splitting it into chunks Finardi et al. 2024 . When a query is received, RAG retrieves relevant chunks either through lexical search Ram et al. 2023 or by computing semantic similarity Karpukhin et al. 2020 , embeddings both the query and text chunks into a shared vector space. Advanced techniques, such as pre-retrieval processing Ma et al. 2023 ; Zheng et al. 2023a and post-retrieval processing Dong et al. 2024 ; Xu et al. 2023 , as well as fine-tuning strategies Li et al. 2023 , have further enhanced RAG’s effectiveness across various domains, including QA) Yan et al. 2024 , dialogue generation Izacard et al. 2023 , and text summarization Jiang et al. 2023 . Several studies have evaluated the effectiveness of RAG systems across various tasks Yu et al. 2024 ; Chen et al. 2024 ; Es et al. 2023 , such as multi-hop question answering Tang and Yang 2024 , biomedical question answering Xiong et al. 2024 , and text generation Liu et al. 2023 . However, no existing study has simultaneously and systematically evaluated and compared RAG and GraphRAG on these general text-based tasks. Figure 1: The illustration of RAG, KG-based GraphRAGs and Community-based GraphRAGs. 2.2 Graph Retrieval-Augmented Generation While RAG primarily processes text data, many real-world scenarios involve graph-structured data, such as knowledge graphs (KGs), social graphs, and molecular graphs Xia et al. 2021 ; Ma and Tang 2021 . GraphRAG Han et al. 2024 ; Peng et al. 2024 aims to retrieve information from various