Interpretable Question Answering with Knowledge Graphs

Kartikeya Aneja, Manasvi Srivastava, Subhayan Das, Nagender Aneja · 2025

Read Interpretable Question Answering with Knowledge Graphs by Kartikeya Aneja, Manasvi Srivastava, Subhayan Das, Nagender Aneja (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2510-19181.txt.

Introduction

Kartikeya Aneja and three co-authors build a question answering system that works only from a knowledge graph, without large language model retrieval pipelines. Instead, a small paraphraser model turns the entity and relation edges found in the graph into a natural final answer.

Documents are first turned into question-answer pairs, then into a knowledge graph that is searched with embeddings and fuzzy matching, re-ranked, and paraphrased. Judged by a language model on a standard benchmark, the system performs competitively. The design offers an interpretable route to answers, where each step can be traced through the graph.

Catalog record

CREATORKartikeya Aneja, Manasvi Srivastava, Subhayan Das, Nagender Aneja
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE15 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2510-19181

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Interpretable Question Answering with Knowledge Graphs Kartikeya Aneja, Manasvi Srivastava, Subhayan Das, Nagender Aneja Abstract This paper presents a question answering system that operates exclusively on a knowledge graph retrieval without relying on retrieval augmented generation (RAG) with large language models (LLMs). Instead, a small paraphraser model is used to paraphrase the entity relationship edges retrieved from querying the knowledge graph. The proposed pipeline is divided into two main stages. The first stage involves pre-processing a document to generate sets of question-answer (QA) pairs. The second stage converts these QAs into a knowledge graph from which graph-based retrieval is performed using embeddings and fuzzy techniques. The graph is queried, re-ranked, and paraphrased to generate a final answer. This work includes an evaluation using LLM-as-a-judge on the CRAG benchmark, which resulted in accuracies of 71.9% and 54.4% using LLAMA-3.2 and GPT-3.5-Turbo, respectively. 1 INTRODUCTION Question Answering (QA) systems aim to provide precise and contextually relevant answers to user queries, often using structured or unstructured knowledge sources. Recently, there has been interest in Retrieval-Augmented Generation (RAG) [ 1 ] , where a user’s query retrieves relevant text chunks, which are then passed to an LLM for answer generation. However, these systems are heavily reliant on unstructured documents and prone to hallucination [ 2 , 3 ] and have limited transparency. In contrast, Knowledge Graphs (KGs) provide a structured representation of information in the form of entities (nodes) and their relationships (edges). The KGs can be created from both structured and unstructured data. This formalism supports interpretable reasoning and contextual association, making them particularly suitable for knowledge-intensive QA tasks. Traditional KG-based QA systems often require complex semantic parsing and rule-based components, limiting their scalability and adaptability. This research presents a Knowledge Graph-based QA system that does not rely on traditional chunk retrieval using RAG but instead constructs and queries a structured knowledge graph generated from QA pairs. Unlike RAG frameworks that use text chunking and dense retrieval, this approach builds a graph-based abstraction of knowledge, enabling semantic and entity-based search across the document’s contents [ 4 , 5 , 6 ] . Our methodology consists of two primary phases. First, QA pairs are generated from documents using a prompt-driven language model. These pairs are then converted into a knowledge graph via entity-relation extraction using LLMGraphTransformer by Langchain with GPT-3.5-Turbo . In the second stage, user queries are answered by retrieving relevant subgraphs based on embedding similarity and fuzzy matching. Retrieved information is passed through a lightweight paraphrasing model tuner007/pegasus_paraphrase to produce coherent, human-readable answers. We evaluated the performance using LLM-as-a-judge. By combining structured KG retrieval with paraphrased natural language responses, the proposed research offers an interpretable alternative to traditional RAG pipelines. It demonstrates practical benefits in legal and technical domains where traceability and factual consistency are critical. Table 1: QA Graph-Based Answering Flow Component Description Document Input PDF containing contractual clauses QA Pairs Extracted using prompt-based QA generation Graph Creation GPT-3.5 identifies entities, relations Embeddings text-embedding-3-large on nodes and types Retrieval Cosine and fuzzy search on entities Reranker BAAI/bge-reranker-large on triples Answer tuner007/pegasus_paraphrase paraphrasing 2 Related Work This section explains related work on knowledge graph-based question answering while avoiding LLM-based RAG. Related work can be categorized into two broad approaches: (i) semantic parsing that converts a question into a logical query, e.g., Cypher in the case of Neo4j, and executes it on the KG, and (ii) retrieval/embedding methods that can find relevant subgraphs or entities and paths without building a full logical query. Gu et al. 2022 reviewed research on question answering over knowledge bases for semantic-parsing and proposed joint entity linking and parsing. Most semantic parsing systems target either SPARQL, a semantic query language for data stored in Resource Description Framework (RDF) format, or use Cypher, a standard query language for property graph databases. Du et al. 2023 proposed constructing a dual relation graph where relations are nodes, and edges connect relations sharing head or tail entities. Their approach alternates between (1) reasoning on the primal entity graph, (2) propagating information on the dual relation graph, and (3) enabling interaction between the two graphs. Zhao et al. 2022 proposed linking relation phrases to relation paths instead of single relations. The authors introduce a path ranking model that aligns both textual (word embeddings) and structural (KG embeddings) information, utilizing a gated attention mechanism with external paraphrase dictionaries to handle vague phrases. Yang et al. 2024 highlighted the limitations of existing RAG approaches, showing that even advanced LLMs with RAG achieve only 44–63% accuracy on dynamic, diverse questions. Although CRAG was built to benchmark RAG, our KG-only pipeline attains competitive accuracy on CRAG. The proposed system leverages structured KG construction from QA pairs, followed by embedding-based retrieval, fuzzy entity matching, reranking, and paraphrasing. This design emphasizes interpretability, traceability, and reduced hallucination, providing clearer reasoning paths compared to black-box RAG methods. 3 RESEARCH METHODOLOGY 3.1 QA Generation from Document We used two datasets: one is a PDF document, and the other is the CRAG dataset [ 10 ] . For our PDF document, we generated question-answer (QA) pairs. The text from the PDF is first divided into semantically meaningful units using a rule-based segmentation strategy. This strategy leverages structural cues, such as paragraph breaks, bullet points, clause identifiers, and length thresholds, to ensure that each chunk represents a logically coherent segment. These segments serve as atomic knowledge blocks for downstream indexing and retrieval. For each extracted chunk, the Hugging Face model iarfmoose/t5-base-question-generator is used to generate a list of relevant question-answer pairs automatically. These pairs are designed to extract atomic facts and knowledge from the text and are stored in JSON format for further processing. In the CRAG, there are two tasks with 2706 question-answer pairs. 3.2 Knowledge Graph Creation This section describes the methodology for constructing a Knowledge Graph from question–answer (QA) pairs and the subsequent process of querying it for relevant information. Given an input question, the system identifies and retrieves the associated entities and relationships from the graph. These retrieved entities and relationships are then fed into a lightweight paraphrasing model, which synthesizes the final answer in natural language. Table 1 presents a high-level overview of the Knowledge Graph-based answer generation workflow. Knowledge Graph Construction Sets of question–answer (QA) pairs are processed through the LLMGraphTransformer of LangChain, utilizing the GPT-3.5 Turbo language model to extract entities and their semantic relationships. The additional information to consider Clauses and Numerical Values as nodes was passed to the LLMGraphTransformer as a prompt shown in Fig. 1 . For this research, sets of twenty QA pairs are passed to LLMGraphTransformer at a time. ⬇ prompt = "Clauses and Numerical Values, etc count as nodes" llm_transformer = LLMGraphTransformer ( llm = llm , additional_instructions = prompt ) Figure 1: LLM Graph Transformer setup These nodes and relationships are used to construct a Knowledge Graph, which is stored in a Neo4j graph database. After processing all QA pairs, the resulting Knowledge Graph encapsulates the document’s knowledge in a structured form. In this graph, nodes represent entities, while edges denote the relationships between them. We created two knowledge graphs, one for each dataset. We then generated embeddings for each node and node type using transformer-based sentence embeddings via the

Read the complete text