Attention Consistency for LLMs Explanation
Tian Lan, Jinyuan Xu, Xue He, Jenq-Neng Hwang, Lei Li · 2025
Read Attention Consistency for LLMs Explanation by Tian Lan, Jinyuan Xu, Xue He, Jenq-Neng Hwang, Lei Li (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2509-17178.txt.
Introduction
Tian Lan, Jinyuan Xu, Xue He, Jenq-Neng Hwang, and Lei Li wrote a 2025 arXiv paper on explaining the choices of large language models. Knowing which input words swayed a model matters for trust, but many explanation tools are slow or blurry. The team wanted a method that is light, cheap, and still faithful to the model.
Their score, called MACS, grades each input token by how steady the model's peak attention stays across its layers. In tests, MACS matched the faithfulness of much heavier tools. It also used notably less video memory and ran with lower delay than those tools. The authors offer it as a practical add-on that can be dropped into decoder-based models with little fuss.
Catalog record
| CREATOR | Tian Lan, Jinyuan Xu, Xue He, Jenq-Neng Hwang, Lei Li |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 49 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2509-17178
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Attention Consistency for LLMs Explanation Tian Lan, Jinyuan Xu, Xue He, Jenq-Neng Hwang, Lei Li Abstract Understanding the decision-making processes of large language models (LLMs) is essential for their trustworthy development and deployment. However, current interpretability methods often face challenges such as low resolution and high computational cost. To address these limitations, we propose the Multi-Layer Attention Consistency Score (MACS) , a novel, lightweight, and easily deployable heuristic for estimating the importance of input tokens in decoder-based models. MACS measures contributions of input tokens based on the consistency of maximal attention. Empirical evaluations demonstrate that MACS achieves a favorable trade-off between interpretability quality and computational efficiency, showing faithfulness comparable to complex techniques with a 22% decrease in VRAM usage and 30% reduction in latency. 1 Introduction The rapid advancements in Large Language Models (LLMs) have revolutionized natural language processing, powering diverse applications ( Li et al. 2025b ; He et al. 2025 ; Jin et al. 2025 ; Yao et al. 2025 ; Cai et al. 2025b ; Cai et al. 2025a ; Li et al. 2025a ) . As these models become increasingly integrated into critical systems, understanding their decision-making processes is crucial for ensuring trustworthiness, reliability, and enabling targeted improvements. Explainable AI (XAI), particularly methods attributing outputs to input tokens, is thus essential. However, existing approaches for interpreting deep decoder-only Transformers face notable limitations: full attention aggregation methods like Attention Rollout can produce noisy or diffuse attributions, potentially due to phenomena like Over-squashing, Over-mixing, and softmax dispersion which can obscure important signals in deep models and long sequences. Concurrently, many other XAI techniques, including gradient and perturbation-based approaches, are often too computationally intensive, limiting their practicality for real-time diagnostics and require model modifications or specialized computation paths. Table 1 summarizes the comparison of MACS and these state-of-the-art XAI methods, outlining their respective advantages and limitations across key interpretability dimensions. 1 1 1 * Both authors contributed equally to this research. 2 2 2 †The corresponding author. Metric MACS Rollout AttnLRP AtMan Faithfulness Usability Comp. Eff. VRAM Eff. Table 1: Comparative summary of MACS (ours) and other XAI methods, highlighting their strengths and weaknesses across key interpretability criteria. ( ✓ ) denotes strong performance, ( ✗ ) indicates poor performance. MACS demonstrates competitive results across most metrics. To address the need for efficient and insightful interpretability, we propose the Multi-Layer Attention Consistency Score (MACS) , a novel, lightweight heuristic for estimating the importance of input tokens in Transformer models. Unlike conventional attribution techniques, MACS quantifies token contribution by evaluating the consistency of the strongest attention connection from an output query to each input token across all layers. This consistency is derived via layer-wise max pooling followed by element-wise (Hadamard) multiplication. MACS can be applied directly during inference without requiring any model modifications. Our main contributions are as follows: • MACS Methodology : We propose the Multi-Layer Attention Consistency Score (MACS), a lightweight and computationally efficient heuristic for input token attribution in Transformer-based models, leveraging cross-layer attention consistency. • Theoretical Motivation : We provide a theoretical grounding for MACS as a response to known information propagation challenges (Over-squashing, Over-mixing, Softmax Dispersion), which often limit the effectiveness of existing attribution techniques. • Empirical Evaluation : Through experiments on Question Answering tasks, we demonstrate that MACS identifies salient tokens more effectively than Attention Rollout and achieves faithfulness comparable to more complex attribution methods. • Practical Utility : We demonstrate the real-time capability of MACS by highlighting its efficiency and showcase its potential applicability beyond text-only models to other Transformer architectures, as preliminarily explored in the context of Visual Question Answering (see Appendix B ). We show that MACS achieves a favorable trade-off between computational efficiency and attribution quality, making it a practical component for interpreting LLMs. 2 Background Understanding the decision-making process of Transformer-based models, especially Large Language Models (LLMs), is a critical research objective in the field of Explainable AI (XAI) Liu et al. 2025 ; Shi et al. 2025 . Existing attribution methods that link model predictions to input tokens can be broadly categorized by their primary mechanisms, including attention aggregation, perturbation, and backpropagation. 2.1 Explainability via Attention Aggregation and its Challenges The attention mechanism has been a natural candidate for interpretability due to its explicit weighting of token interactions Zhang et al. 2023 . Early studies demonstrated its usefulness in tasks such as machine translation Zenkel et al. 2019 and document classification Yang et al. 2016 . Subsequent analyses of models like BERT revealed that individual attention heads can capture linguistically meaningful patterns Clark et al. 2019 However, interpreting raw attention from single layers or heads in deep, multi-head Transformers is often insufficient. To obtain a more holistic view, some methods focusing on attention aggregation across layers, like Attention Rollout Abnar and Zuidema 2020 , have been proposed. Rollout typically propagates attention by multiplying attention matrices across layers, thereby tracing and accumulating influence through all paths. Although more comprehensive than raw attention, such full-aggregation techniques may still yield "noisy" or diffuse attribution maps, making it challenging to identify critical input tokens. Recent theoretical studies have highlighted challenges associated with aggregating attention signals in deep decoder-based Transformer models Jia and Li 2024 . Barbero et al. 2024 ; Barbero et al. 2025 formalized phenomena like Over-squashing, where influence from distant inputs is severely attenuated, and Over-mixing/Representational Collapse, where token representations lose distinctiveness. Furthermore, Veličković et al. 2024 demonstrate that standard softmax attention inherently disperses and loses its ability to focus sharply as sequence length increases out-of-distribution. These theoretical limitations suggest that methods like Rollout, by design, may struggle to produce clear attributions in deep, long-context models. This motivates the need for alternative heuristics like the one proposed in this work. 2.2 Perturbation and Backpropagation-Based Explainability Beyond direct attention aggregation, prominent XAI paradigms include perturbation-based and gradient-based methods, each offering distinct approaches to attribution. Perturbation-based methods assess feature importance by modifying input features or, in Transformer-specific variants like AtMan Deiseroth et al. 2025 , by altering pre-softmax attention weights and measuring the impact on outputs. This family also includes local surrogate models such as LIME Ribeiro et al. 2016 and LORE Guidotti et al. 2018 , which approximate the model locally using interpretable proxies, and SHAP Lundberg and Lee 2017 , which leverages game-theoretic principles. While generally model-agnostic, these methods require numerous forward passes, making them computationally expensive. Moreover, surrogate-based explanations may fail to capture the internal representations of the original model. Gradient-based methods propagate signals from the output back to the input using gradient information. Techniques such as SmoothGrad Smilkov et al. 2017 and Integrated Gradients Sundararajan et al. 2017 belong to this family, however their effectiveness can degrade in large Transformers Achtibat et al. 2024 . Other approaches, like gradient-weighted attention Chefer et al. 2021 (e.g., Grad-CAM Selvaraju et al. 2019 variants), and rule-based techniques like Layer-wise Relevance Propagation (LRP) Binder et al. 2016 and AttnLRP Achtibat et al. 2024 , provide more structured attributions but face challenges in handling Transformer-specific non-linearities while maintaining theoretical guarantees like relevance conservation. Both families are typically post hoc and computationally demanding, often requiring backward passes or repeated evaluations. This limits their scalability for real-time or interactive scenarios, motivating the development of efficient