KEA Explain: Explanations of Hallucinations using Graph Kernel Analysis
Reilly Haskins, Benjamin Adams · 2025
Read KEA Explain: Explanations of Hallucinations using Graph Kernel Analysis by Reilly Haskins, Benjamin Adams (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2507-03847.txt.
Introduction
Reilly Haskins and Benjamin Adams present a system that not only catches language model hallucinations but explains them, in this 2025 paper. Their framework builds a knowledge graph from the model's answer and compares it with trusted graphs from Wikidata or supplied documents. Graph kernels measure how the structures differ, and semantic clustering groups the mismatches into readable reasons.
Detection accuracy held up on both open and closed question sets. More importantly, the output is contrastive: it shows what the answer claims against what the record supports. That makes the flag useful to a human checker rather than a bare score. The authors aim the work at high-stakes uses where a wrong sentence costs real harm, and sketch multi-source checks as the next step.
Catalog record
| CREATOR | Reilly Haskins, Benjamin Adams |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 39 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2507-03847
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
KEA Explain: Explanations of Hallucinations using Graph Kernel Analysis Reilly Haskins, Benjamin Adams Abstract Large Language Models (LLMs) frequently generate hallucinations: statements that are syntactically plausible but lack factual grounding. This research presents KEA (Kernel-Enriched AI) Explain: a neurosymbolic framework that detects and explains such hallucinations by comparing knowledge graphs constructed from LLM outputs with ground truth data from Wikidata or contextual documents. Using graph kernels and semantic clustering, the method provides explanations for detected hallucinations, ensuring both robustness and interpretability. Our framework achieves competitive accuracy in detecting hallucinations across both open- and closed-domain tasks, and is able to generate contrastive explanations, enhancing transparency. This research advances the reliability of LLMs in high-stakes domains and provides a foundation for future work on precision improvements and multi-source knowledge integration. 1 Introduction Despite the impressive capabilities of Large Language Models (LLMs), they frequently generate outputs that are grammatically correct but semantically incorrect—referred to as hallucinations ( Huang et al. 2025 ) . These hallucinations are a significant challenge, particularly for high-stakes domains such as healthcare, legal advice, and education, where misleading information can seriously degrade user trust ( Oelschlager 2024 ) . Thus, detecting and explaining hallucinations is critical to LLM reliability. In this paper, we propose a neurosymbolic framework, KEA (Kernel-Enriched AI) Explain, that not only detects semantic hallucinations in LLM-generated texts under both open-domain (general knowledge) and closed-domain (constrained context) conditions but also provides explanations of why the statements are deemed hallucinatory. By integrating symbolic AI methods with state-of-the-art neural techniques, our approach combines the strengths of both paradigms to address key limitations in existing methods. Specifically, we use graph kernels ( Vishwanathan et al. 2010 ) to compare the structural similarity between knowledge graphs (KG), while semantic word embeddings ( Almeida and Xexéo 2019 ) are employed to cluster semantically similar labels and select relevant triples. For the open-domain problem, these techniques are combined with structured knowledge from Wikidata ( Vrandečić and Krötzsch 2014 ) , a comprehensive and widely-used knowledge base, to form a robust detection system. We extract entities and relations from the LLM output and query these against the knowledge base, systematically identifying and explaining hallucinations based on discrepancies in structural and semantic similarity. For the closed-domain problem, we construct a KG based on both the LLM output and the provided context in place of the Wikidata knowledge. There is evidence that people prefer explanations that are contrastive; that is, they explain why one outcome occurred rather than another plausible alternative ( Miller 2019 ) . By providing why a given output was classified as a hallucination while also providing what fact would make it not a hallucination, KEA Explain offers a more explainable and intuitive solution. The key contributions of this paper are as follows: • Introduces the use of graph kernels over symbolic knowledge graphs to detect and explain hallucinations in LLM outputs. • Presents a novel neurosymbolic framework to enable comparison of semanticity between entity and relation labels of a knowledge graph pair. • Improves upon existing methods by introducing explainable classifications, which are driven by the symbolic structure of the graph-based representation. 2 Related Work Hallucinations in LLMs stem from biased training data, lack of in-built logic, vague prompts, and insufficient data or overfitting ( Perković et al. 2024 ) . Common techniques to mitigate the production of hallucinations include fine-tuning and Retrieval-Augmented Generation (RAG) ( Gao et al. 2023 ) , which supplements LLM outputs with external data. However, RAG depends on prompt quality, potentially retrieving relevant but insufficient context ( Guan et al. 2024 ) . Given these limitations, hallucination detection is pertinent. Here, we review recent advances in hallucination detection. 2.1 LLM-based and Probabilistic Techniques Many recently developed methods use LLMs or probabilistic measures to detect hallucinations. ChainPoll ( Friel and Sanyal 2023 ) aggregates votes from multiple LLM instances, but it is computationally intensive and lacks ground truth validation, so biases in the training data of the LLMs being used could propagate through. SelfCheckGPT ( Manakul et al. 2023 ) estimates hallucination probability based on response entropy, assuming that domain knowledge leads to consistent outputs. While effective, both methods lack explainability and require multiple LLM queries, increasing computational cost. Belief Tree Propagation (BTPROP) ( Hou et al. 2025 ) enhances detection by constructing a tree of logically related statements, analyzed via a hidden Markov model. This improves accuracy but is also resource-intensive, as tree expansion incurs exponential time complexity. 2.2 Knowledge Graphs for Hallucination Detection and Prevention Open Domain. AlignScore ( Zha et al. 2023 ) uses a unified information alignment function, a model designed to assess the alignment between two pieces of text, and is trained on 4.7M examples spanning seven language tasks, including paraphrasing, fact verification, and summarization. However, its reliance on synthetic training data introduces questions about real-world generalizability. Knowledge Graph-based Retrofitting (KGR) ( Guan et al. 2024 ) reduces factual hallucinations in LLMs by refining outputs using KGs. Unlike methods that only retrieve facts based on user queries, KGR is able to extract, validate, and correct facts within the LLM’s reasoning process, enabling it to catch inaccuracies generated during intermediate reasoning, however, entity detection noise can prevent accurate claim validation. Closed Domain. FactAlign ( Rashad et al. 2024 ) constructs a KG from a given source and generated text, and uses word embeddings to align individual claim triples with source triples to identify factual misalignments, categorizing them as hallucinations if the similarity is low. FactAlign differentiates between intrinsic hallucinations (distortions of source information) and extrinsic hallucinations (unsupported additions) by employing a contradiction score from a natural language inference (NLI) model. The approach achieves high accuracy scores without requiring training or fine-tuning. However, it excludes non-named entities in the KG, which reduces coverage, and encounters scalability challenges due to the computational overhead of computing pairwise similarities between each generated triplet and all source triplets. Additionally, the triple-level comparison methodology in Rashad et al.’s approach prevents wider context from the knowledge graph from contributing to similarity calculations, a limitation that our graph kernel-based approach overcomes. GraphEval ( Sansford et al. 2024 ) decomposes LLM outputs into entities and relationships, validating them against the grounding context provided using NLI. On established benchmarks, it performs well on long outputs, but is limited to closed-domain settings, where the method is provided with the output alongside both the original prompt and the contextual information that was used to arrive at that output. This limits GraphEval’s real-world applicability. 2.3 Summary Across these studies, three common limitations emerge: 1. Generalizability : The majority of methods are designed to rely on specific open- or closed-domain conditions, but not both. In addition, many make use of synthetic training data, limiting applicability beyond their training domain. 2. Lack of ground-truth : Many of these methods use proxies for ground truth, such as measuring the entropy of LLM responses, or neural NLI models trained on synthetic data, introducing biases. 3. Lack of explainability : Neural methods alone are not transparent, making it difficult to understand why certain outputs are classified as hallucinations, degrading user trust. 3 Method Our proposed method systematically detects and explains hallucinations in LLM outputs through comparison of KG representations. Initially, an LLM