Enabling Global, Human-Centered Explanations for LLMs: From Tokens to Interpretable Code and Test Generation
Dipin Khati, Daniel Rodriguez-Cardenas, David N. Palacio, Alejandro Velasco, Michele Tufano, Denys Poshyvanyk · 2025
Read Enabling Global, Human-Centered Explanations for LLMs: From Tokens to Interpretable Code and Test Generation by Dipin Khati, Daniel Rodriguez-Cardenas, David N. Palacio, Alejandro Velasco, Michele Tufano, Denys Poshyvanyk (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2503-16771.txt.
Introduction
Dipin Khati and co-authors push code-generating models to explain themselves globally in a 2025 software engineering paper. Accuracy scores say nothing about how a model reasons, and the usual token-level explanations stay too local and scattered for a developer to learn anything general from them.
Their framework, CodeQ, lifts thousands of token-level rationales into high-level programming categories and aggregates them statistically. Aggregation cut explanation uncertainty by more than half, exposing habits of a generation model, which leaned on shallow cues like indentation over deeper logic. The authors argue such global views are what trust in coding assistants will require.
Catalog record
| CREATOR | Dipin Khati, Daniel Rodriguez-Cardenas, David N. Palacio, Alejandro Velasco, Michele Tufano, Denys Poshyvanyk |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 75 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2503-16771
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Enabling Global, Human-Centered Explanations for LLMs: From Tokens to Interpretable Code and Test Generation Dipin Khati, Daniel Rodriguez-Cardenas, David N. Palacio, Alejandro Velasco, Michele Tufano, Denys Poshyvanyk Abstract . As Large Language Models for Code (LM4Code) become integral to software engineering, establishing trust in their output becomes critical. However, standard accuracy metrics obscure the underlying reasoning of generative models, offering little insight into how decisions are made. Although post-hoc interpretability methods attempt to fill this gap, they often restrict explanations to local, token-level insights, which fail to provide a developer-understandable global analysis. Our work highlights the urgent need for global, code-based explanations that reveal how models reason across code. To support this vision, we introduce code rationales ( Code ℚ \mathbb{Q} ), a framework that enables global interpretability by mapping token-level rationales to high-level programming categories. Aggregating thousands of these token-level explanations allows us to perform statistical analyses that expose systemic reasoning behaviors. We validate this aggregation by showing it distills a clear signal from noisy token data, reducing explanation uncertainty (Shannon entropy) by over 50%. Additionally, we find that a code generation model ( codeparrot-small ) consistently favors shallow syntactic cues (e.g., indentation ) over deeper semantic logic. Furthermore, in a user study with 37 participants, we find its reasoning is significantly misaligned with that of human developers. These findings, hidden from traditional metrics, demonstrate the importance of global interpretability techniques to foster trust in LM4Code. Conference: 2026 IEEE/ACM 48th International Conference on Software Engineering; April 12–18, 2026; Rio de Janeiro, Brazil 2026 IEEE/ACM 48th International Conference on Software Engineering (ICSE ’26), April 12–18, 2026, Rio de Janeiro, Brazil DOI: 10.1145/3744916.3787764 ISBN: 979-8-4007-2025-3/2026/04 icse2026-p1047 CCS: Software and its engineering Software development techniques Automatic programming CCS: Software and its engineering Software development techniques Program synthesis CCS: Computing methodologies Artificial intelligence † † cc-license: by 1. Introduction LM4Code such as GPT-4 ( Brown et al., 2020 ) , Claude 3 ( Anthropic, 2024 ) , and StarCoder ( Li et al., 2023 ) promise significant developer productivity as developers integrate them into their software engineering (SE) workflows. As these models move from research curiosities to indispensable tools for tasks such as code completion and test generation, their reliability, safety, and interpretability become paramount. The central question for the AI for Software Engineering (AI4SE) community is no longer “Can these models generate correct code?” but rather “Can we trust generative models?" ( Khati et al., 2025 ) . Answering this question demands that we scrutinize the underlying reasoning of generative models ( i.e., LLMs), which aligns with calls for a more rigorous science of interpretable machine learning ( Doshi-Velez and Kim, 2017 ; Palacio et al., 2024b ) . The primary obstacle to establishing code-based trust lies in our current evaluation paradigm. The de facto standard for evaluating LLMs consists of a suite of accuracy-based metrics, including Exact Match and F1 for general evaluation; CodeBLEU for code synthesis ( Ren et al., 2020 ) ; HumanEval, with pass@k and correctness rate, for code reasoning ( Chen et al., 2021 ) . Although these metrics possess useful information for measuring functional correctness, they offer no insight into the model’s decision-making process. For instance, a model might generate correct code by relying on spurious, non-generalizable patterns in the input, a behavior that accuracy scores alone cannot detect ( Ribeiro et al., 2020 ) . Overreliance on correctness can overestimate the true capabilities of the model ( Burnell et al., 2023 ) and fails to meet the growing demand for a more holistic evaluation of LLMs ( Liang et al., 2023 ) . To look beyond precision, the AI community has increasingly turned to post-hoc interpretability techniques such as LIME ( Ribeiro et al., 2016 ) and Shapley values ( Lundberg and Lee, 2017 ) that can explain individual predictions by attributing importance to input features. Although valuable, these methods are often computationally intensive, conceptually ill-suited for generative tasks, and provide only local, token-level explanations. Similarly, attention mechanisms initially offered a promising window into model reasoning ( Mohankumar et al., 2020 ) , however, subsequent work challenged their reliability, showing that attention does not accurately represent the internal logic of a model ( Jain and Wallace, 2019 ; Serrano and Smith, 2019 ) . Consequently, practitioners lack tools to systematically analyze a model’s behavior at a global level to uncover consistent biases or diagnose systemic reasoning flaws and successes. To address these limitations, we introduce Code \mathbb{Q} , a interpretability framework that enables global post-hoc analysis of LLMs for code. Our framework was inspired by rationalization theory, identifying the minimal subset of input tokens that are the most influential for a prediction ( Vafa et al., 2017 ) . However, for explanations to be truly useful to practitioners, they must be presented in human-understandable terms ( Doshi-Velez and Kim, 2017 ) . Explanations at the raw token level are often too fine-grained and lack the abstract structure that developers use to reason about programs ( Ghorbani et al., 2019 ) . Our key contribution, therefore, is a framework that goes beyond tokens by mapping these low-level rationales to a category of higher-level programming categories derived from syntax decomposition ( Palacio et al., 2024a ) . This abstraction of individual tokens into meaningful categories ( e.g., from ‘if’ to [conditionals] ) is the critical step that allows us to aggregate explanations across thousands of outputs. Our research enables a global statistical view ( i.e., reasoning patterns) of a given generative model ( e.g., encoder-decoder and decoder-only), directly addressing the limitations of previous local methods ( e.g., Shapley, SHAP, and Lime). To establish and rigorously validate our framework, we focus on two well-known models representing distinct decoder-only ( codeparrot-small ) and encoder-decoder ( BART-large ) architectures for Python code completion and Java test case generation, respectively. This work is outside the scope of benchmarking specific state-of-the-art models; instead, we demonstrate the flexibility and power of our transformer-agnostic Code \mathbb{Q} framework and highlight the need for practitioner-friendly global explanations. The global statistical view of our study yields immediate and telling insights into the LLM’s behavior. Applying our framework, we find that LLMs systematically rely on shallow syntactic patterns, such as code indentation and punctuation , far more frequently than on deep semantic categories, such as conditional logic . We also observe a strong self-reliance phenomenon in which structures like loops are primarily influenced by existing loop-related tokens in the prompt. Perhaps most critically, our user study of 37 participants reveals a significant misalignment between these machine-generated rationales and the explanations provided by human developers (Jaccard similarity 0.074 for code completion, 0.306 for test generation), highlighting a fundamental gap in their respective reasoning processes. These findings, invisible to accuracy metrics, underscore the need for category-level global interpretability to foster appropriate trust and guide future model development. The main contributions of our work are as follows: (1) We developed an interpretability framework Code \mathbb{Q} , which transforms token rationales at the low level into a structured interpretability tensor based on high-level programming concepts; (2) We performed an exploratory analysis using two different models to demonstrate the applicability of Code \mathbb{Q} to estimate global code-based explanations; (3) We conducted a user