From Nodes to Narratives: Explaining Graph Neural Networks with LLMs and Graph Context

Peyman Baghershahi, Gregoire Fournier, Pranav Nyati, Sourav Medya · 2025

Read From Nodes to Narratives: Explaining Graph Neural Networks with LLMs and Graph Context by Peyman Baghershahi, Gregoire Fournier, Pranav Nyati, Sourav Medya (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2508-07117.txt.

Introduction

Peyman Baghershahi, Gregoire Fournier, Pranav Nyati, and Sourav Medya present a method that explains graph neural network predictions in plain language. Their 2025 paper tackles a trust problem: these networks learn from linked data such as citation graphs, but their reasons stay hidden. Existing explanation tools struggle when nodes carry rich text.

Their framework feeds the network's internal node vectors into a large language model, mixing them with text drawn from the graph itself. The model then writes an explanation and points to a small subgraph that supports it. Tests on real text-linked data sets show a good balance between faithfulness to the network and sparseness of the cited subgraph. Human-centered scores, such as how insightful readers find the text, also improved.

Catalog record

CREATORPeyman Baghershahi, Gregoire Fournier, Pranav Nyati, Sourav Medya
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE51 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2508-07117

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

From Nodes to Narratives: Explaining Graph Neural Networks with LLMs and Graph Context Peyman Baghershahi, Gregoire Fournier, Pranav Nyati, Sourav Medya Abstract Graph Neural Networks (GNNs) have emerged as powerful tools for learning over structured data, including text-attributed graphs, which are common in domains such as citation networks, social platforms, and knowledge graphs. GNNs are not inherently interpretable and thus, many explanation methods have been proposed. However, existing explanation methods often struggle to generate interpretable, fine-grained rationales, especially when node attributes include rich natural language. In this work, we introduce Logic , a lightweight, post-hoc framework that uses large language models (LLMs) to generate faithful and interpretable explanations for GNN predictions. Logic projects GNN node embeddings into the LLM embedding space and constructs hybrid prompts that interleave soft prompts with textual inputs from the graph structure. This enables the LLM to reason about GNN internal representations and produce natural language explanations along with concise explanation subgraphs. Our experiments across four real-world TAG datasets demonstrate that Logic achieves a favorable trade-off between fidelity and sparsity, while significantly improving human-centric metrics such as insightfulness. Logic sets a new direction for LLM-based explainability in graph learning by aligning GNN internals with human reasoning. 1 Introduction Graph neural networks (GNNs) have witnessed rapid adoption across a wide range of critical real-world applications, including healthcare Zitnik et al. 2018 ; Zitnik et al. 2019 , drug design Xiong et al. 2021 ; Liu et al. 2022 ; Sun et al. 2020 , recommender systems Chen et al. 2022 , and fraud detection Rao et al. 2021 . The high-stakes nature of these domains demands that the predictions made by GNNs be not only accurate but also trustworthy. One powerful approach to building trust in deep learning models is to provide meaningful explanations that justify their predictions Shneiderman 2020 . Such explanations, which may highlight important substructures in the input Luo et al. 2020 ; Schlichtkrull et al. 2021 ; Yuan et al. 2021 , or offer counterfactual scenarios Lucic et al. 2022a ; Tan et al. 2022a ; Chen et al. 2022 . However, generating explanations for GNNs is inherently challenging due to the combinatorial nature of graph data and the joint influence of node attributes and edge connections. These complexities make GNN explainability a non-trivial problem and have led to the development of a wide array of explanation techniques Kakkad et al. 2023a ; Guo et al. 2023 . 1.1 Need for LLM-based GNN explainers There has been a growing interest in the integration of GNNs and LLMs to enhance GNN task performance on text attribute graphs (TAGs). Essentially, these methods leverage LLMs in different architectural orders to 1) attain rich representations of the input TAGs, task descriptions, and task-specific few-shot examples for the GNN predictors Liu et al. 2024a ; Fang et al. 2024 , or 2) reason over the TAGs directly Fatemi et al. 2024 or along with GNN-enhanced soft prompts to make predictions Chen et al. 2024a ; Tang et al. 2024 . However, there is a lack of focus on using LLMs to explain GNNs predictions and clarify the reasoning behind these black-box models. Traditional GNN explainers aim to identify influential subgraphs or feature subsets that contribute to predictions Ying et al. 2019a ; Luo et al. 2020 . While effective for graphs with simple node features, these methods struggle in TAGs, where node information is embedded in natural language. Moreover, often these explanations are in the form of subgraphs that are not human-tangible for domain users. LLMs, on the other hand, excel at reasoning over textual content and can generate coherent, human-interpretable rationales. Thus, their integration into GNNs explanation pipelines can improve the interpretability of the GNN models outputs on TAGs. 1.2 Limitations of Existing LLM-based GNN Explainers The existing frameworks (please see Sec. 2 ) that use LLMs to build GNN explanations have the following shortcomings: (1) Template dependency and alignment complexity: Aligning the GNN explainers’ output with the LLM acceptable input necessitates rigid templates, handcrafted scores, or extensive training/fine-tuning of the GNN explainer. (2) Missed GNN internals: Existing approaches fail to directly leverage the rich internal representations of the GNN, making the explanations generic or unfaithful to the internal working of the GNNs. (3) Bias from suboptimal explainers: Involving a pre-existing GNN explainer can bias the reasoning and judgment of the LLM. This is critical when the GNN explainers are suboptimal and they signal noisy information to LLM, while the LLM can potentially better infer and reason the pivotal causes of why the GNN has made specific predictions, particularly for TAGs where the textual attributes carry rich information about the graph entities. 1.3 Our Contributions In this work, we present Logic ( L anguage m O del-guided G NN I nterpretation with C ontext), a lightweight, post-hoc explanation framework for text-attributed graphs (TAGs) that seamlessly integrates the representational power of GNNs with the reasoning capabilities of LLMs. Our contributions are as follows: • Novel Method : We introduce a method that bypasses traditional GNN explainer modules by directly using LLMs as interpreters of GNN behavior. Logic generates fine-grained, natural language explanations without relying on handcrafted templates or perturbation-based saliency masks, reducing bias from external explainers. • Soft Prompt Integration : Logic aligns the internal representations of a trained GNN with the token-level embedding space of an LLM via a novel embedding projector. This projection enables GNN features to be treated as soft prompts in a hybrid prompt that blends structural and textual cues, allowing the LLM to reason over the graph’s latent space. • Interpretable Explanation Subgraphs : Unlike prior methods that return subgraph masks, Logic produces interpretable subgraphs based on node-level binary support decisions derived from natural language explanations. • Training-Agnostic and Plug-and-Play Deployment : Logic does not require fine-tuning the GNN or the LLM, making it easily deployable on pre-trained models without retraining. This makes it adaptable for real-world applications. 2 Related Work We describe the related studies on the integrated LLM and GNN explainer methods. The explanation capability of LLMs has been applied to graph problems in task-oriented contexts such as explainable recommendation systems Li et al. 2025 ; Ma et al. 2024 . There has been attempts to guide GNNs using the explanations generated by LLMs, for example, in He et al. 2024a , an LLM initially generates node predictions and textual explanations on the reasoning behind the predictions. These explanations then guide a second LLM to produce new node features, which augment the original text features as input for a GNN predictor. LLM explanation to enhance explainers. LLMs have been utilized to enhance the explanations of pre-existing GNN explainers, for example Zhang et al. 2024 uses LLMs to evaluate explanations of an explainer, its scores guide the weighted gradients used for optimizing the explainer through Bayesian Variation Inference to address learning bias. Similarly, to improve the understandability of explanations, Pan et al. 2024 first trains a pseudo-label generator LLM that takes explanations from a pre-existing GNN explainers and generates pseudo-labels for them. The generator is tuned in an expert iteration procedure with tailored objectives for faithfulness and brevity, and the optimized generator generates pseudo-labels to fine-tune an explainer LLM to build end-to-end

Read the complete text