Explaining Fine-Tuned LLMs via Counterfactuals: A Knowledge Graph–Driven Framework
Yucheng Wang, Ziyang Chen, Md Faisal Kabir · 2025
Read Explaining Fine-Tuned LLMs via Counterfactuals: A Knowledge Graph–Driven Framework by Yucheng Wang, Ziyang Chen, Md Faisal Kabir (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2509-21241.txt.
Introduction
Explaining how LoRA fine tuning reshapes a language model reasoning is an open challenge in interpretability. Behind the work are Yucheng Wang and 2 co-authors. Fine tuning with LoRA lets a model learn a domain with small added weights. Users still cannot see how that step changes the model reasoning. The authors seek a clear way to show the change.
They build BioToolKG, a knowledge graph for bioinformatics tools. A knowledge graph links facts as nodes and edges. Their explainer learns soft masks over those nodes and edges. Small changes that cause large shifts in meaning reveal what the tuning relied on. They test the method on a fine tuned LLaMA model. The work balances sparse structure, large meaning shift, and clear rules for reading the result.
Catalog record
| CREATOR | Yucheng Wang, Ziyang Chen, Md Faisal Kabir |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 46 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2509-21241
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Explaining Fine-Tuned LLMs via Counterfactuals: A Knowledge Graph–Driven Framework Yucheng Wang, Ziyang Chen, Md Faisal Kabir Abstract The widespread adoption of Low-Rank Adaptation (LoRA) has enabled large language models (LLMs) to acquire domain-specific knowledge with remarkable efficiency. However, understanding how such a fine-tuning mechanism alters a model’s structural reasoning and semantic behavior remains an open challenge. This work introduces a novel framework that explains fine-tuned LLMs via counterfactuals grounded in knowledge graphs. Specifically, we construct BioToolKG, a domain-specific heterogeneous knowledge graph in bioinformatics tools and design a counterfactual-based fine-tuned LLMs explainer ( CFFTLLMExplainer ) that learns soft masks over graph nodes and edges to generate minimal structural perturbations that induce maximum semantic divergence. Our method jointly optimizes structural sparsity and semantic divergence while enforcing interpretability preserving constraints such as entropy regularization and edge smoothness. We apply this framework to a fine-tuned LLaMA-based LLM and reveal that counterfactual masking exposes the model’s structural dependencies and aligns with LoRA-induced parameter shifts. This work provides new insights into the internal mechanisms of fine-tuned LLMs and highlights counterfactual graphs as a potential tool for interpretable AI. 1 Introduction In recent years, with the continuous advancement of NLP technologies and large language models (LLMs) [ 1 ] , which exhibit remarkable generative capabilities [ 1 ] and cross-domain generalization [ 2 ] , an increasing number of application-level techniques have begun to play a significant role across various domains [ 3 ] . Structured knowledge graphs [ 4 ] have become a compelling direction in a wide range of Natural Language Processing(NLP) applications. As a form of heterogeneous information network (HIN), knowledge graphs have supported numerous early use cases, including recommendation systems, link prediction, and information fusion [ 5 ] . Recently, the integration of knowledge graphs with LLMs has given rise to novel applications such as domain-specific knowledge graph construction [ 6 ] and graph-enhanced retrieval-augmented generation (RAG) [ 7 ] , significantly improving the factuality and controllability of LLM outputs. However, due to the low information purity and the presence of substantial irrelevant content in many knowledge graphs, using RAG may lead LLMs to hallucinate or fail to extract accurate subgraphs as reliable external knowledge [ 8 , 9 ] . Furthermore, the extracted subgraphs often contain cycles, which can pose significant challenges in tasks such as knowledge path reasoning and graph ordering [ 10 ] . Meanwhile, with the rise of explainable AI (XAI) techniques, there has been growing attention to the user-friendliness and interpretability of AI systems in real-world applications [ 11 ] . A variety of explainability frameworks have been proposed for predictive models, utilizing diverse strategies. Among these, many rely on perturbing input features to evaluate their influence on model outputs [ 12 , 11 , 13 , 14 , 15 ] . However, it is important to note that alternative approaches—such as gradient-based methods [ 16 ] or attention-based [ 17 ] explanations—also exist. However, there are additional challenges in interpreting LLMs. Firstly, the output of an LLM is not a simple classification result, but rather contains rich semantic content. Secondly, due to the complexity of LLM architectures and the massive scale of training data, fine-grained explanations based on attention mechanisms alone are often insufficient or intractable [ 18 ] . Finally, prompt-based self-explanation tools for LLMs have shown limited effectiveness, primarily due to the prevalence of hallucinations and the inherent uncertainty in LLM outputs [ 19 , 20 ] . To improve and extend existing methods, this paper proposes the following innovations in the subsequent sections: 1. We introduce BioToolKG , a domain-specific and semantically structured knowledge graph that organizes bioinformatics tools, algorithms, databases, and more related entities; 2. A novel counterfactual-based interpretability framework ( CFFTLLMExplainer ) is proposed, specifically tailored for fine-tuned large language models, enabling structure-aware explanation of model behavior; 3. The counterfactual generation is formalized as an unsupervised optimization problem , where a multi-objective loss jointly balances semantic divergence, structural sparsity, prompt relevance, and graph smoothness; 4. To uncover the internal decision mechanisms of fine-tuned LLMs, we conducted a multi-perspective interpretability analysis by aligning learned structural masks, token-level attention scores, and LoRA-induced embedding shifts. Following the introduction, the paper is organized into several sections. Section 2 reviews relevant works on LLM interpretability and outlines the motivation for our study. Section 3 introduces our proposed counterfactual-based interpretability framework tailored for fine-tuned LLMs. Section 4 introduces experiments for revealing the mechanisms in LoRA fine-tuning process. In Section 5 presents preliminary experimental results that demonstrate the effectiveness of our approach. Finally, Section 6 and Section 7 discuss potential directions for future work and summarize main contributions. 2 Related Works and Motivations With the rapid development of large language models (LLMs), the widespread adoption of Low-Rank Adaptation (LoRA) fine-tuning has played a significant role in enabling LLMs to acquire domain-specific knowledge efficiently, thereby facilitating the construction of expert systems. This technique is particularly important for the broader deployment of LLMs in specialized applications. Specifically, LoRA introduces a low-rank matrix as an incremental module into the existing model architecture, effectively altering the model’s behavior. By inserting these trained adapter layers into the original model, LoRA enables the acquisition of new knowledge while updating only a small subset of parameters, thus ensuring efficiency and adaptability [ 21 ] . With the rapid advancement of Explainable AI (XAI), a variety of XAI tools, such as SHAP, LIME and DiCE, have been widely applied to interpret and analyze different AI models and systems. In LIME, feature importance is indicated by the weights assigned under feature perturbation [ 11 ] , and in SHAP, it is represented by computing Shapley values [ 12 ] . Under this background, the interpretability of large language models (LLMs) has increasingly attracted widespread attention alongside their rapid development [ 22 ] . Meanwhile, counterfactual explanations—such as those generated by DiCE—enhance model interpretability and human alignment by producing diverse and informative counterfactuals. Specifically, DiCE formulates counterfactual generation as a multi-objective optimization problem, where a carefully designed loss function minimizes the distance between the counterfactuals and the original instances while simultaneously altering the model’s prediction [ 13 ] . TokenSHAP, tries to explain LLM output from token level. Specifically, it extends the application of SHAP by introducing Shapley value-based attribution to natural language processing tasks, enabling a deeper understanding of how different components of an input prompt influence the model’s output. By incorporating Monte Carlo Shapley Estimation, it achieves a balance between computational efficiency and estimation accuracy [ 23 ] . Some recent studies have proposed embedding interpretability frameworks directly into the architecture of large language models (LLMs) to overcome the limitations of traditional black-box interpretability, which relies solely on analyzing model outputs. Concept Bottleneck Large Language Models (CB-LLMs) introduce an inherently interpretable framework for LLMs, demonstrating clear advantages in terms of scalability and transparency—both critical for the responsible development. Specifically, the core idea of CB-LLMs lies in training a Concept Bottleneck Layer (CBL) that maximizes the concept score (i.e., similarity) of input samples, thereby generating transparent concept weights that explain the model’s outputs in a semantically meaningful and interpretable manner [ 24 ] . Overall, current research on the interpretability of large language models (LLMs) generally follows two