Comparing RAG and GraphRAG for Page-Level Retrieval Question Answering on Math Textbook
Eason Chen, Chuangji Li, Shizhuo Li, Zimo Xiao, Jionghao Lin, and Kenneth R. Koedinger · 2025
Read Comparing RAG and GraphRAG for Page-Level Retrieval Question Answering on Math Textbook by Eason Chen, Chuangji Li, Shizhuo Li, Zimo Xiao, Jionghao Lin, and Kenneth R. Koedinger (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2509-16780.txt.
Introduction
Eason Chen and co-authors compare two ways to ground a tutor in a textbook in a 2025 paper. Plain retrieval fetches pages that match a question. A graph variant also uses links between ideas in the book. The team test both for page-level answers on an undergraduate math text.
The graph method gave a clear gain in answer quality. The gain matters most where budgets are tight, since graph methods can cost more to run. Knowing the trade-off helps schools pick a design. A tutor that cites the right page builds trust with students and teachers. The study gives builders hard numbers for that choice.
Catalog record
| CREATOR | Eason Chen, Chuangji Li, Shizhuo Li, Zimo Xiao, Jionghao Lin, and Kenneth R. Koedinger |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 35 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2509-16780
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Comparing RAG and GraphRAG for Page-Level Retrieval Question Answering on Math Textbook Eason Chen, Chuangji Li, Shizhuo Li, Zimo Xiao, Jionghao Lin, and Kenneth R. Koedinger Abstract Technology-enhanced learning environments often help students retrieve relevant learning content for questions arising during self-paced study. Large language models (LLMs) have emerged as novel aids for information retrieval during learning. While LLMs are effective for general-purpose question-answering, they typically lack alignment with the domain knowledge of specific course materials such as textbooks and slides. We investigate Retrieval-Augmented Generation (RAG) and GraphRAG, a knowledge graph-enhanced RAG approach, for page-level question answering in an undergraduate mathematics textbook. While RAG has been effective for retrieving discrete, contextually relevant passages, GraphRAG may excel in modeling interconnected concepts and hierarchical knowledge structures. We curate a dataset of 477 question–answer pairs, each tied to a distinct textbook page. We then compare the standard embedding-based RAG methods to GraphRAG for evaluating both retrieval accuracy—whether the correct page is retrieved—and generated answer quality via F1 scores. Our findings show that embedding-based RAG achieves higher retrieval accuracy and better F1 scores compared to GraphRAG, which tends to retrieve excessive and sometimes irrelevant content due to its entity-based structure. We also explored re-ranking the retrieved pages with LLM and observed mixed results, including performance drop and hallucinations when dealing with larger context windows. Overall, this study highlights both the promises and challenges of page-level retrieval systems in educational contexts, emphasizing the need for more refined retrieval methods to build reliable AI tutoring solutions in providing reference page numbers. 1 Introduction When students learn in technology-based learning environment, they often need to reference or retrieve relevant information in reading excerpts, textbooks, and other structured text content. Artificially intelligent solutions can help students quickly find and reference relevant sources and excerpts related to questions, facts, and other study objectives during this learning process. Increasingly so, these references are generated by large language models (LLMs), which have shown notable expertise in domains such as mathematics and science [ 1 , 25 ] . However, while these models excel at providing answers, they often lack direct alignment with specific course materials (e.g., textbooks or lecture notes) and struggle to handle specialized domain knowledge, resulting in suboptimal performance [ 2 , 20 , 23 ] . In the worst case, these models provide incorrect and hallucinated sources to back up claims [ 15 ] . How can our field move toward more reliable and accurate curation of learning material in personalized learning supported by LLMs? Retrieval-augmented generation (RAG) has emerged as a promising solution by combining information retrieval mechanisms as context for LLM to generate outputs [ 28 , 9 , 2 ] . In RAG, a retriever module first identifies potentially relevant documents or pages from a textbook, followed by a generator module that combines the retrieved content with the user’s query to produce an answer. This two-stage pipeline mitigates “hallucination” and helps direct the model’s attention to domain-specific information [ 12 ] . A further refinement of RAG is the variant GraphRAG [ 6 ] , which extends RAG with a knowledge graph to capture entity relationships and provide more comprehensive retrieved content. In an educational context, it is important that an AI tutor furnishes correct answers and precisely references the textbook pages and content students need to refer to [ 5 , 27 ] . Ensuring alignment at the page level not only requires the model to retrieve information from and ground its responses in a clearly defined corpus—rather than relying on pre-trained knowledge, but also enables users to easily locate the referenced material in the textbook to review the original context and verify the AI’s response. In particular, when deploying RAG approaches in mathematics education, enhancing retrieval faithfulness by ensuring AI-generated answers are both accurately sourced and effectively address the question can improve learning outcomes [ 4 , 17 ] . Providing explicit pointers, like page references, might further help students reference for deeper understanding or verify correctness. In our paper, we explore how RAG-based pipelines can be adapted for math textbook question answering, focusing on the following R esearch Q uestion: “To what extent can an AI-based retrieval system identify the correct textbook page for a question derived from that specific page?” Specifically, we compare two RAG approaches: a standard embedding-based RAG and a knowledge-graph-based GraphRAG [ 6 ] approach that uses graph-structured relationships between mathematical entities. We benchmark these methods on a custom-built dataset of 477 question-answer pairs derived from An Infinite Descent into Pure Mathematics [ 22 ] , an undergraduate mathematics textbook. By measuring both retrieval correctness (the ability to locate the appropriate page) and generative quality (the ability to produce the correct answer), our experiments show how different RAG pipelines align large language models more closely with course-specific materials. The findings have direct implications for building reliable AI tutoring systems that not only answer questions but also point learners exactly to where in the textbook they can find references, enabling transparent instruction and mitigating potential hallucinations in LLMs by working in conjunction with existing, verified learning material. 2 Related Work 2.1 Limitations of LLMs in Question Answering and the Emergence of Retrieval-Augmented Generation (RAG) in Education Pre-trained LLMs often struggle to reference specific contextual information. For instance, [ 20 ] employed GPT-3 to evaluate student-generated short-answer questions, revealing that the model tends to overestimate question quality and lacks precise alignment with course content. Another study [ 23 ] investigated ChatGPT-based feedback with ChatGPT in a digital learning game, finding that the mdoel encountered difficulties with complex number-line problems. These challenges highlight the importance of using augmentation methods like RAG to ensure alignment with educational content. In response to these limitations, research [ 2 , 9 , 28 ] indicates that Retrieval-Augmented Generation (RAG) can effectively enhance LLM performance by providing domain-specific context. RAG leverages vector-based embedding retrieval to gather relevant reference documents from an external database, and then feeds the retrieved information to an LLM as input for generation. By grounding model outputs in these context documents, RAG not only helps mitigate hallucination but also boosts factual accuracy. Generally, a RAG pipeline involves two core components: a retriever , which indexes and locates relevant snippets or documents (e.g., via semantic embeddings), and a generator , which produces the final answer using both the user query and retrieved content. This structure makes RAG particularly appealing for educational settings, where aligning answers with textbook pages, lecture slides, or other course materials can offer significant pedagogical benefits. Some recent studies have shown the effectiveness of RAG solutions in building LLM-based tutors [ 12 , 19 , 8 , 14 , 26 , 16 ] . Research suggests that these retrieval-based methods not only enhance the accuracy and contextual relevance of automated tutoring but also mitigate hallucinations and other errors in generated responses [ 12 , 30 ] . For instance, RetLLM-E [ 19 ] is a RAG approach designed for university course forums, integrating past Q&A posts and official course materials to improve LLM-generated responses. Similarly, CourseAssist [ 8 ] is an LLM-based tutor that grounds feedback in lecture slides, notes, and assignments for computer science courses. Nevertheless, certain challenges persist—such as