Toward Explaining Large Language Models in Software Engineering Tasks CCS: Software and its engineering Software creation and management
Antonio Vitale, Khai Nguyen-Nguyen, Denys Poshyvanyk, Rocco Oliveto, Simone Scalabrino, Antonio Mastropaolo · 2025
Read Toward Explaining Large Language Models in Software Engineering Tasks CCS: Software and its engineering Software creation and management by Antonio Vitale, Khai Nguyen-Nguyen, Denys Poshyvanyk, Rocco Oliveto, Simone Scalabrino, Antonio Mastropaolo (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2512-20328.txt.
Introduction
Antonio Vitale and co-authors address a trust problem: large language models write and summarize code well, but nobody can see why they produce a given output. Their approach, FeatureSHAP, explains model behaviour for software engineering tasks. It is fully automated and works with both open and proprietary models.
FeatureSHAP builds on Shapley values from game theory. It perturbs high-level input features one at a time and measures how the output changes, using task-specific similarity checks. On code generation and code summarization, it blames irrelevant features less and tracks the model more faithfully than benchmarks. A survey of thirty-seven practitioners found the explanations genuinely helped interpretation and decisions.
Catalog record
| CREATOR | Antonio Vitale, Khai Nguyen-Nguyen, Denys Poshyvanyk, Rocco Oliveto, Simone Scalabrino, Antonio Mastropaolo |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 92 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2512-20328
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Toward Explaining Large Language Models in Software Engineering Tasks CCS: Software and its engineering Software creation and management Antonio Vitale, Khai Nguyen-Nguyen, Denys Poshyvanyk, Rocco Oliveto, Simone Scalabrino, Antonio Mastropaolo Abstract . Recent progress in Large Language Models (LLMs) has substantially advanced the automation of software engineering (SE) tasks, enabling complex activities such as code generation and code summarization. However, the black-box nature of LLMs remains a major barrier to their adoption in high-stakes and safety-critical domains, where explainability and transparency are vital for trust, accountability, and effective human supervision. Despite increasing interest in explainable AI for software engineering, existing methods lack domain-specific explanations aligned with how practitioners reason about SE artifacts. To address this gap, we introduce FeatureSHAP , the first fully automated, model-agnostic explainability framework tailored to software engineering tasks. Based on Shapley values, FeatureSHAP attributes model outputs to high-level input features through systematic input perturbation and task-specific similarity comparisons, while remaining compatible with both open-source and proprietary LLMs. We evaluate FeatureSHAP on two bi-modal SE tasks: code generation and code summarization. The results show that FeatureSHAP assigns less importance to irrelevant input features and produces explanations with higher fidelity than baseline methods. A practitioner survey involving 37 participants shows that FeatureSHAP helps practitioners better interpret model outputs and make more informed decisions. Collectively, FeatureSHAP represents a meaningful step toward practical explainable AI in software engineering. FeatureSHAP is available at https://github.com/deviserlab/FeatureSHAP . 1. Introduction Automation in software engineering (SE) is rapidly evolving due to two main drivers: the unprecedented volume and sheer amount of code data available from open-source ecosystems such as GitHub and the rise of Large Language Models (LLMs) capable of understanding and generating code ( 13 ; 58 ; 40 ; 28 ; 25 ; 30 ) . Among these, Neural Code Models (NCMs) represent a focused branch of LLM research–models specifically optimized for coding-related tasks, offering finer control and domain awareness in SE applications ( 19 ; 24 ; 70 ; 23 ; 62 ) . Trained in a vast corpus of text enriched with large-scale open-source code, these models have achieved impressive performance in a wide range of software engineering applications, including code completion ( 14 ; 13 ) , program repair ( 33 ; 17 ; 32 ; 75 ) , code summarization ( 5 ; 6 ; 65 ; 69 ) , and test case generation ( 59 ; 36 ; 7 ) . Through scaling to billions and even trillions of parameters 1 1 1 For example, recent large-scale models such as OpenAI’s GPT-4, Anthropic’s Claude 3, and Google’s Gemini 1.5 are reported to contain on the order of one to several trillion parameters, depending on the configuration disclosed. , LLMs are capable of capturing complex relationships between code and natural language, allowing them to generalize effectively across programming languages, coding styles, and diverse development contexts. This capability has placed LLMs at the core of commercial software engineering tools, such as GitHub Copilot ( 21 ) , which promise to enhance developer productivity throughout the software development life-cycle ( 48 ; 49 ; 66 ) . This signals a fundamental shift in SE automation, where AI-driven models are no longer confined to experimental settings, but are actively embedded in IDEs and integrated into development workflows. Figure 1 . Token-level (top) and feature-level (bottom) attributions for the same input; darker rectangles represent higher attribution scores. However, LLMs operate as black boxes. Given a suggestion ( e.g., a generated code block), a developer can either blindly trust the model, which actually results in a productivity improvement, or not, which implies double-checking the output and, thus, spending more time on the task. In addition, LLMs are non-deterministic, i.e., the same input can result in different outputs. Such challenges pose significant barriers to the trustworthy adoption of such models. Ideally, developers should neither blindly trust the model – because of non-determinism – nor completely mistrust it – since it would dramatically reduce the benefits of using LLMs in the first place. Instead, we conjecture that the best option would be an evidence-based trust : predictions should be accompanied by easy-to-check pieces of evidence that report to what extent the user can trust the output and its parts. Explainable AI (xAI) for SE methods emerge as a practical and promising solution to this problem. xAI methods reveal which inputs most influence an LLM’s output, enabling practitioners to better assess the reliability of generated code, debug unexpected behavior, and refine prompts to align the model with the user’s intended goals. Among existing xAI techniques, two of the most popular are LIME ( 57 ) and SHAP ( 42 ) . LIME ( 57 ) explains a prediction by generating perturbed versions of the input ( e.g., by masking or removing tokens) and training a simple interpretable model to approximate how these perturbations affect the generated output. SHAP ( 42 ) , instead, estimates the importance of each input token by measuring how the inclusion or exclusion of that element changes the output across all possible subsets of tokens. Existing explainability techniques typically operate at the level of individual tokens, which limits their applicability in software engineering contexts. For example, consider the attributions for the prompt shown in Fig. 1 . In the token-level view, the attribution is spread across many tokens, resulting in an explanation that is difficult to interpret: Developers do not reason one token at a time, and the distribution of importance across subwords offers little insight into which conceptual parts of the instruction actually contributed to model’s output. Instead, the feature-level view provides a different perspective. Grouping tokens into semantically coherent units, such as the task description, the constraint on case sensitivity, and the definition of the vowel set, produces an explanation that aligns more closely with how developers understand and reason about prompts. This higher-level representation makes the explanation substantially more actionable. For instance, when a specific feature receives negligible attribution ( e.g., the final requirement in the example), the developer may immediately infer two possibilities: Either the model ignored an intended constraint, which may imply a potential issue in the generated code, or the model produced the correct result regardless, explaining that the overlooked requirement was not contributing in the first place ( i.e., the requirement could have been omitted without affecting the model’s behavior). In both cases, feature-level attribution provides a clearer, more practical basis for explaining the model’s behavior than token-level analysis and naturally points toward the need for xAI techniques that operate over semantically meaningful, task-dependent features —units. When the input consists of source code, such units may align with program structures, such as method signatures, code blocks, documentation fragments, that define the semantics developers rely on when reasoning about the way the model processed and responded to the input. To this end, we introduce FeatureSHAP , a model- and task-agnostic explainability framework that estimates the importance of an input i in generating the related output M(i) . Grounded in the theoretical foundations of Shapley values ( 60 ) , FeatureSHAP provides a principled mechanism for estimating feature attribution beyond specific model architectures. Unlike recent efforts such as doCode ( 53 ) , ASTrust ( 52 ) ,