TOBUGraph: Knowledge Graph-Based Retrieval for Enhanced LLM Performance Beyond RAG
Savini Kashmira, Jayanaka L. Dantanarayana, Joshua Brodsky, Ashish Mahendra, Yiping Kang, Krisztián Flautner, Lingjia Tang, Jason Mars · 2024
Read TOBUGraph: Knowledge Graph-Based Retrieval for Enhanced LLM Performance Beyond RAG by Savini Kashmira, Jayanaka L. Dantanarayana, Joshua Brodsky, Ashish Mahendra, Yiping Kang, Krisztián Flautner, Lingjia Tang, Jason Mars (2024) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2412-05447.txt.
Introduction
Savini Kashmira and a large team replace text-chunk retrieval with knowledge graph retrieval for language models, in their 2024 arXiv paper. Standard retrieval-augmented generation matches query text to stored chunks, which misses deeper relationships, depends heavily on chunking choices and invites hallucination. Their framework, TOBUGraph, first builds a knowledge graph from unstructured data automatically, using language models to extract entities and relationships.
Retrieval then walks the graph instead of matching chunks. The system runs in production inside a personal memory organization app. On real user data it beat several standard retrieval implementations in both precision and recall, improving the experience through more accurate answers.
Catalog record
| CREATOR | Savini Kashmira, Jayanaka L. Dantanarayana, Joshua Brodsky, Ashish Mahendra, Yiping Kang, Krisztián Flautner, Lingjia Tang, Jason Mars |
|---|---|
| YEAR | 2024 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 26 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2412-05447
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
TOBUGraph: Knowledge Graph-Based Retrieval for Enhanced LLM Performance Beyond RAG Savini Kashmira, Jayanaka L. Dantanarayana, Joshua Brodsky, Ashish Mahendra, Yiping Kang, Krisztián Flautner, Lingjia Tang, Jason Mars Abstract Retrieval-Augmented Generation (RAG) is one of the leading and most widely used techniques for enhancing LLM retrieval capabilities, but it still faces significant limitations in commercial use cases. RAG primarily relies on the query-chunk text-to-text similarity in the embedding space for retrieval and can fail to capture deeper semantic relationships across chunks, is highly sensitive to chunking strategies, and is prone to hallucinations. To address these challenges, we propose TOBUGraph , a graph-based retrieval framework that first constructs the knowledge graph from unstructured data dynamically and automatically. Using LLMs, TOBUGraph extracts structured knowledge and diverse relationships among data, going beyond RAG’s text-to-text similarity. Retrieval is achieved through graph traversal, leveraging the extracted relationships and structures to enhance retrieval accuracy, eliminating the need for chunking configurations while reducing hallucination. We demonstrate TOBUGraph’s effectiveness in TOBU , a real-world application in production for personal memory organization and retrieval. Our evaluation using real user data demonstrates that TOBUGraph outperforms multiple RAG implementations in both precision and recall, significantly improving user experience through improved retrieval accuracy. 1 Introduction Integrating Large Language Models (LLMs) with external knowledge sources improves retrieval accuracy and enhances reliability Niu et al. 2024 . The state-of-the-art approach for such integration is Retrieval Augmented Generation (RAG) Lewis et al. 2021 ; Gao et al. 2024 . RAG preprocesses documents by chunking text and storing the chunks in a vector database. During retrieval, it retrieves the top-ranked chunks based on vector similarity, and an LLM leverages those selected chunks to generate a response accordingly. While traditional RAG based approaches allow LLMs to incorporate external knowledge, this methodology faces several key limitations: • RAG relies on query-chunk similarity in vector embeddings, comparing the query to each chunk individually without capturing broader contextual connections among text chunks. However, in many domains, data can be interconnected. Failing to represent and leverage such relationships and structures beyond text-to-text similarity across multiple chunks often leads to low retrieval accuracy by RAG Peng et al. 2024 . • Chunking and embedding strategies, such as chunk length and overlap size can significantly affect retrieval performance Qu et al. 2024 . • When relevant chunks do not exist in the database for a given query, RAG may hallucinate Huang et al. 2025 . Indeed, our evaluation of RAG approaches using production data in a real-world application clearly highlights these limitations (Section 3 ). To address these limitations, it is important to uncover the relationships among unstructured data and leverage such relationships to improve retrieval performance. A promising approach is to structure data as knowledge graphs Su et al. 2024 ; Hogan et al. 2021 . Prior work Jin et al. 2024 ; Wu et al. 2024b introduces a graph-augmented retrieval technique that uses LLM reasoning of the knowledge graph through a chain-of-thought process. While this approach enhances retrieval, it assumes the existence of a predefined knowledge graph and overlooks its construction, which remains labor-intensive and lacks adaptability to dynamic data Hofer et al. 2024 . Designing a holistic graph-based retrieval framework that enables automatic knowledge graph construction and graph-based retrieval to capture deeper semantic relationships remains an open challenge. In this work, we propose TOBUGraph, a novel graph-based retrieval augmentation framework. TOBUGraph leverages LLMs to automatically construct a knowledge graph from unstructured data. Unlike traditional RAG that stores data chunks in a vector database and compares query-chunk’s text similarly, TOBUGraph extracts structured knowledge and diverse relationships among data and represents the structures and connections of data in a graph. Our novel graph structure is composed of semantic nodes , representing the key semantic information of data chunks, and relationship nodes , to represent diverse semantic relationships between semantic nodes. During retrieval, TOBUGraph leverages relationship nodes to prune irrelevant data and prioritize the retrieval on highly relevant data, improving retrieval precision. By traversing the pruned graph of all relevant interconnected nodes, we mitigate the limitations of traditional chunking and ensure completeness and high recall for the retrieval. We implemented TOBUGraph in a real-world application called TOBU, designed for storing and retrieving personal memories. We define "personal memory" as user-provided images and videos coupled with details, context and narratives around them. When users upload an image, TOBUGraph will first leverage a multimodal LLM to extract key details and generate a summary of the image. Users can provide more details and refinements through a conversational AI assistant. TOBUgraph constructs a knowledge graph of such memories and facilitates users to interact and query about them. Figure 1: Memory capturing workflow in TOBUGraph framework implemented in TOBU app. Using real-world user data of the TOBU app, we evaluated TOBUGraph approach against multiple RAG baseline implementations. TOBUGraph consistently outperformed these baselines in retrieval accuracy, efficiency, and user experience, receiving higher preference ratings across diverse memory retrieval scenarios. The main contributions of this paper are as follows. 1. A novel approach to extracting structured knowledge and diverse relationships among unstructured data and representing the structures and connections of data in a graph. 2. A novel approach to leverage such knowledge graph to enable a more effective and efficient retrieval mechanism. 3. Applying TOBUGraph in a real-world application for personal memory organization and retrieval. 4. A comprehensive evaluation against RAG systems using real-world user data. TOBUGraph achieves 93.74% precision (vs. 89.23% best baseline), 91.96% recall (vs. 82.26% best baseline), and 92.84% F1-score (vs 85.56% best baseline). Our user experience evaluation shows that whenever TOBUGraph appears as a response option, evaluators are 75% likely to choose it over RAG baselines. We plan to open-source our dataset and experimentation for further study. 2 TOBUGraph Figure 2: Memory retrieving workflow in TOBUGraph framework implemented in TOBU app. In this section, we introduce TOBUGraph, a novel graph-based approach for information capture and retrieval. TOBUGraph overcomes the limitations of RAG by structuring information in a dynamic graph-based representation, effectively capturing relationships between various data points. Here, we describe the implementation of TOBUGraph in the TOBU app for personal memory capture and retrieval. During the capturing process (Figure 1 ), TOBUGraph uses an LLM to automatically extract semantics from user inputs, creating context-rich memories and establishing relationships between them to form a structured memory graph. During retrieval (Figure 2 ), users interact with a conversational AI assistant as TOBUGraph combines LLMs with the memory graph, efficiently navigating it for comprehensive recall that outperforms RAG approaches. The following sections detail each step of this process. 2.1 Memory Input Data Collection Our system combines a multimodal LLM with a conversational AI assistant to help users effortlessly create memory entries (Figure 1 ). When users provide multimedia inputs, such as images, videos, or audio, the multimodal LLM applies object recognition, emotion detection, scene recognition, and geolocation estimation to extract key contextual details, including date, location, people, activities, and emotions associated with the memory. Based on these extracted details, the LLM generates an initial summary. The TOBU AI assistant then engages the user in a conversation, gathering additional details and refining the extracted information as needed. As users provide more input, the summary dynamically updates to reflect the most accurate