Don’t Go Breaking My LLM: The Impact of Pruning Attention Layers on Explanation Faithfulness and Confidence Calibration
Pietro Tropeano [email protected], Maria Maistro [email protected], Tuukka Ruotsalo [email protected], Christina Lioma [email protected] · 2026
Read Don’t Go Breaking My LLM: The Impact of Pruning Attention Layers on Explanation Faithfulness and Confidence Calibration by Pietro Tropeano [email protected], Maria Maistro [email protected], Tuukka Ruotsalo [email protected], Christina Lioma [email protected] (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2606-24970.txt.
Introduction
Pietro Tropeano, Maria Maistro, Tuukka Ruotsalo, and Christina Lioma ask what pruning attention layers does to a language model's trustworthiness, beyond accuracy. Pruning cuts memory and inference costs, and attention layers are the most resource hungry parts of these models. Earlier work showed up to a third of attention layers can go with minimal accuracy loss. Nobody had checked the interpretability side: whether explanations stay faithful and confidence stays calibrated. The authors study exactly that, across five large language models and eight datasets.
Accuracy often survives pruning, as expected. Faithfulness and calibration often do not. Both can swing widely even while accuracy holds steady, exposing a mismatch between a model's confidence, its interpretability, and its correctness. A pruned model may answer as well as before while explaining itself worse and misjudging its own certainty. Tropeano and colleagues recommend that checks of pruned models include explanation and calibration metrics as standard.
Catalog record
| CREATOR | Pietro Tropeano [email protected], Maria Maistro [email protected], Tuukka Ruotsalo [email protected], Christina Lioma [email protected] |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 118 KB |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2606.24970
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Don’t Go Breaking My LLM: The Impact of Pruning Attention Layers on Explanation Faithfulness and Confidence Calibration Pietro Tropeano [email protected], Maria Maistro [email protected], Tuukka Ruotsalo [email protected], Christina Lioma [email protected] Abstract Pruning Large Language Models (LLMs) reduces memory and inference costs by removing parts of the network, producing smaller models that retain most of their accuracy. As attention layers are the most resource-intensive parts of LLMs, pruning them is a promising compression strategy. Prior work shows that up to 33% of attention layers can be pruned with minimal accuracy loss. Nevertheless, the impact of attention pruning on model interpretability, specifically faithfulness and confidence calibration, remains unstudied. To address this gap, we study how pruning attention layers affects explanation faithfulness and confidence calibration across five LLMs and eight datasets. While the pruned models often maintain high accuracy, we find that their faithfulness and calibration often degrade. Notably, faithfulness and calibration can fluctuate significantly, even when accuracy remains stable, highlighting a misalignment between model confidence, interpretability, and accuracy. Our findings suggest that layer pruning can affect LLMs’ interpretability and reliability in ways not captured by accuracy and efficiency measures alone. We recommend including explainability and calibration metrics when evaluating pruned models. The code is available at: https://github.com/pietrotrope/Dont_Go_Breaking_My_LLM 1 Introduction Large Language Models (LLMs) have achieved exceptional results across tasks, yet their high computational demands make them costly to deploy. Pruning is a family of model compression techniques that reduces inference time and memory requirements by removing parts of the model, while aiming to preserve performance. A specific type of pruning is layer removal, also known as layer drop ( Gromov et al. 2025 ; He et al. 2026 ) , which cuts entire layers from the model. This method is simple and model agnostic, making it relevant in current research. Layer removal in LLMs can reduce computational costs with minimal accuracy loss, especially when pruning attention layers rather than MLPs; up to 33% of attention layers can be removed without harming performance ( Siddiqui et al. 2024 ) . However, the impact of layer pruning on key aspects of interpretability, such as faithfulness (do model rationales accurately reflect the reasoning process of the model?) and confidence calibration (do predicted probabilities reflect true likelihoods?), remains underexplored ( Jacovi & Goldberg 2020 ; Naeini et al. 2015 ) . To fill this gap, we present the first systematic study of how pruning attention layers affects explanation faithfulness and confidence calibration. Specifically, we make three contributions. First, we provide the first evaluation of the impact of attention layer pruning on the faithfulness of LIME ( Ribeiro et al. 2016 ) and Kernel SHAP ( Lundberg & Lee 2017 ) feature attributions, measured using comprehensiveness and sufficiency ( DeYoung et al. 2020 ) . Second, we present the first analysis of how attention layer pruning influences the alignment between model confidence and accuracy by computing the Estimated Calibration Error (ECE) ( Naeini et al. 2015 ) . Third, we relate changes in explanation faithfulness and confidence alignment to changes in model accuracy, investigating whether accuracy degradation due to attention layer pruning is a good proxy for its effects on explanation faithfulness and confidence calibration. We find that pruning attention layers negatively affects LLMs’ explanation faithfulness. Notably, changes in faithfulness are often unstable and can occur independently of changes in accuracy. For both pruned and unpruned models, we analyse the calibration between model confidence and accuracy and find that removing more layers generally causes them to follow different trends. As with faithfulness, the effect of pruning on calibration is not aligned with the effect on accuracy, highlighting the limitations of using only accuracy to evaluate pruned models. Overall, we provide novel evidence that attention layer pruning can unpredictably degrade model interpretability and confidence calibration in ways that are not captured by accuracy. 2 Related work Model pruning is a family of compression approaches used to improve the efficiency of LLMs by removing parts of the model, such as parameters or structures (e.g., neurons, features, layers). Pruning methods aim to reduce model size, lower computational costs, and accelerate inference, while preserving the model’s performance. Pruning methods can be either unstructured or structured. Unstructured Pruning. Unstructured pruning methods remove individual parameters (weights) from a neural network without enforcing any specific sparsity pattern. There exist several unstructured pruning methods for LLMs, such as SparseGPT ( Frantar & Alistarh 2023 ) , Wanda ( Sun et al. 2024 ) , and RIA ( Zhang et al. 2024d ) . These methods can preserve most of the models’ performance while increasing their sparsity. However, even though unstructured pruning reduces the number of parameters, it often does not lead to faster inference on standard hardware (e.g., CPUs or GPUs), since the resulting sparse patterns are irregular and hard to leverage ( Li et al. 2017 ) . Recent pruners, therefore, include a semi-structured variant, which imposes some structure on the sparsity patterns and can, in principle, enable hardware acceleration ( Li et al. 2017 ; Frantar & Alistarh 2023 ; Sun et al. 2024 ; Zhang et al. 2024d ; Zhang et al. 2024b ) . Structured Pruning and Layer Removal. Structured pruning methods remove entire structures, such as attention heads, neurons, or entire layers, from a neural network. Several structured pruning methods have been developed for LLMs ( Ma et al. 2023 ; Xia et al. 2024 ; Michel et al. 2019 ; Ashkboos et al. 2024 ; Men et al. 2024 ) . These methods can lead to tangible efficiency improvements; however, they tend to yield worse performance to sparsity trade-offs compared to unstructured pruning ( Hu et al. 2025 ) . A subset of the structured pruning literature specifically targets the removal of entire layers from the network. Siddiqui et al. 2024 investigate the removal of both MLP and attention layers in decoder-based architectures, finding that pruning attention layers leads to smaller accuracy drops compared to pruning MLPs. Remarkably, up to 33% of attention layers can be removed without compromising performance on specific downstream tasks. Additionally, pruning attention layers reduces the computational cost associated with the KV-cache. In another example, Gromov et al. 2025 remove layers using the similarity between their input and output representations as a proxy for layer importance. Yet, studies on how layer pruning affects model performance have primarily focused on measures like accuracy and inference time ( Men et al. 2024 ; He et al. 2026 ; Zhang et al. 2024c ; Siddiqui et al. 2024 ; Kim et al. 2024 ; Song et al. 2024 ) . Pruned models evaluation. Most pruning studies evaluate models using accuracy and inference time ( Frantar & Alistarh 2023 ; Sun et al. 2024 ; Zhang et al. 2024d ; Men et al. 2024 ; He et al. 2026 ; Zhang et al. 2024c ; Siddiqui et al. 2024 ; Kim et al. 2024 ; Song et al. 2024 ) . Other work has examined internal properties of pruned models, such as the similarity of representations across layers ( Gromov et al. 2025 ; Song et al. 2024 ; Zhang et al. 2024a ) . Prior work also focuses on pruned models’ fact recall and retrieval abilities ( Jin et