Learning to Explain: Supervised Token Attribution from Transformer Attention Patterns
George Mihaila · 2026
Read Learning to Explain: Supervised Token Attribution from Transformer Attention Patterns by George Mihaila (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2601-14112.txt.
Introduction
George Mihaila trains models to explain which words drive a transformer's answer in 2026. Explainable AI has become critical as transformers enter high-stakes fields. Healthcare, legal systems, and financial services need trust and accountability. A transformer's self-attention weights have proven valuable for interpretability.
Attention shows where a model focuses as it reads. Mihaila's work learns supervised token attribution from transformer attention patterns. Supervision uses known answers to teach which tokens mattered. Clearer token credit could help experts audit model choices before acting on them. The maps can be checked by hand for a few key cases. That check is what lets experts trust the bulk of the scores.
Catalog record
| CREATOR | George Mihaila |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 49 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2601-14112
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Learning to Explain: Supervised Token Attribution from Transformer Attention Patterns George Mihaila Abstract Explainable AI (XAI) has become critical as transformer-based models are deployed in high-stakes applications including healthcare, legal systems, and financial services, where opacity hinders trust and accountability. Transformers’ self-attention mechanisms have proven valuable for model interpretability, with attention weights successfully used to understand model focus and behavior Xu et al. 2015 ; Wiegreffe and Pinter 2019 . However, existing attention-based explanation methods rely on manually defined aggregation strategies and fixed attribution rules Abnar and Zuidema 2020a ; Chefer et al. 2021 , while model-agnostic approaches (LIME, SHAP) treat the model as a black box and incur significant computational costs through input perturbation. We introduce Explanation Network (ExpNet), a lightweight neural network that learns an explicit mapping from transformer attention patterns to token-level importance scores. Unlike prior methods, ExpNet discovers optimal attention feature combinations automatically rather than relying on predetermined rules. We evaluate ExpNet in a challenging cross-task setting and benchmark it against a broad spectrum of model-agnostic methods and attention-based techniques spanning four methodological families. 1 Introduction Transformer architectures Vaswani et al. 2017 have revolutionized Natural Language Processing, achieving state-of-the-art performance across diverse tasks. Despite their success, transformers remain complex and difficult to interpret, as predictions emerge from multiple layers of self-attention and nonlinear transformations. This opacity raises critical concerns for transparency and trustworthiness, particularly when models are deployed in sensitive domains such as healthcare diagnostics, legal decision support, and financial risk assessment Doshi-Velez and Kim 2017 . Rigorous, human-centered interpretability has therefore become essential for accountable deployment of these powerful yet opaque models. Existing XAI approaches fall into two main categories. Model-agnostic methods such as LIME Ribeiro et al. 2016 and SHAP Lundberg and Lee 2017 treat models as black boxes, offering broad applicability but ignoring internal reasoning and incurring significant computational costs. In NLP specifically, these perturbation-based methods face a fundamental challenge: masking or replacing words creates semantically invalid examples that lose linguistic meaning Feng et al. 2018 . The transformer’s self-attention mechanism offers an alternative explanation source. Attention weights have proven valuable for interpretability since their introduction Bahdanau et al. 2014 ; Xu et al. 2015 ; Wiegreffe and Pinter 2019 , leading to numerous attention-based explanation methods Abnar and Zuidema 2020a ; Qiang et al. 2022 . However, these methods rely on hand-crafted heuristic rules or fixed aggregation functions to extract explanations from attention patterns. Our contribution treats attention-based explanation as a learning problem. This learned approach generalizes across tasks and consistently outperforms established baseline methods while maintaining computational efficiency. Our work contributes: (1) a novel framework that learns attention-based explanations from human rationales rather than relying on fixed heuristics, (2) demonstration of cross-task explainability generalization in NLP, (3) comprehensive ablation studies justifying architectural choices, and (4) extensive empirical evaluation showing state-of-the-art performance. 2 Related Work 2.1 Model-Agnostic Explanation Methods Model-agnostic techniques treat models as black boxes, requiring only input-output access. LIME Ribeiro et al. 2016 learns local linear approximations by probing models on perturbed inputs, while SHAP Lundberg and Lee 2017 provides game-theoretic Shapley value-based attributions with formal guarantees of consistency and local accuracy. While offering broad applicability, these methods face limitations in NLP contexts. They ignore model internals including transformer attention mechanisms, incur significant computational costs through extensive model queries, and perturbation-based approaches can produce out-of-distribution inputs that confound causal interpretation Feng et al. 2018 . 2.2 Attention as Explanation Attention mechanisms have proven valuable for interpretability since their introduction, with attention weights widely visualized to understand model focus Bahdanau et al. 2014 ; Xu et al. 2015 ; Vig 2019 . Research has shown that attention patterns capture meaningful linguistic phenomena including syntactic dependencies and semantic relationships Wiegreffe and Pinter 2019 . This motivates attention-based explanation methods that leverage these internal signals, which are common to transformer architectures and unavailable to model-agnostic approaches. 2.3 Gradient-Based and Attention-Based Methods Gradient-based methods leverage backpropagation to attribute importance. Integrated Gradients Sundararajan et al. 2017a accumulates gradients along paths from baseline to actual inputs, satisfying axioms including sensitivity and implementation invariance, but operates independently of attention mechanisms. Methods combining gradients with attention leverage complementary signals from backpropagation and attention mechanisms. Gradient-weighted approaches multiply attention scores by loss gradients, analogous to Grad-CAM in computer vision Selvaraju et al. 2016 ; Barkan et al. 2021 . AttCAT Qiang et al. 2022 integrates attention weights, token gradients, and skip connections. Layer-wise Relevance Propagation (LRP) methods conservatively redistribute output scores through systematic backpropagation Bach et al. 2015 . Transformer-LRP Chefer et al. 2020 adapted LRP for transformers, Generic Attention Explainability (GAE) Chefer et al. 2021 extended it to encoder-decoder models, and Mask-LRP Song et al. 2024 refined the approach by masking uninformative heads. Attention aggregation approaches combine attention signals from multiple heads or layers using predetermined rules. Attention Rollout and Attention Flow Abnar and Zuidema 2020b aggregate attention across layers to trace token influence, addressing entangled representations in deeper layers Brunner et al. 2019 . These approaches improved correlation with ablation studies through multi-layer composition rules. These methods extract explanations through predetermined procedures—gradient computations, propagation algorithms, or aggregation functions—that are specified apriori based on mathematical or heuristic principles. Each method embodies specific assumptions about how to derive token importance from model internals. 2.4 Explanation Evaluation Evaluation of explanation quality faces fundamental challenges. Faithfulness metrics assess whether explanations reflect model reasoning through perturbation DeYoung et al. 2019 , but suffer from critical problems in NLP: token removal creates out-of-distribution inputs Feng et al. 2018 , architectural redundancy enables compensation where alternative pathways maintain predictions despite removing important features Voita et al. 2019 ; Serrano and Smith 2019 , and circular validation without ground truth undermines validity Hooker et al. 2018 . Plausibility metrics, which measure alignment with human rationales Jacovi and Goldberg 2020 , provide a more principled evaluation framework for high-stakes applications where human interpretability is paramount Doshi-Velez and Kim 2017 . Our approach embraces this paradigm by learning explanations directly from human rationales. 2.5 Positioning ExpNet ExpNet builds upon attention-based explanation methods, leveraging the insight that transformer attention patterns contain valuable signals for interpretability Bahdanau et al. 2014 ; Wiegreffe and Pinter 2019 ; Abnar and Zuidema 2020b . Rather than applying fixed aggregation rules or heuristics, ExpNet treats explanation extraction as a supervised learning problem, training on human-annotated rationales to discover which attention configurations correlate with human-judged token importance. This approach enables cross-task generalization and direct optimization for human-aligned explanations while reducing the need for human rationales when applied to new datasets. 3 Methodology 3.1 Overview and Base Model We use BERT-base Devlin et al. 2019 as our transformer model for all experiments. BERT processes input text by tokenizing it into subword units and prepending a special classification token [CLS] to each sequence. During classification, BERT’s final hidden state corresponding to [CLS] is passed to a classification head to produce the prediction. This design makes [CLS] a natural aggregation point: through self-attention across all layers, [CLS] gathers task-relevant information from input tokens. Our goal is to learn which input tokens are important for BERT’s decisions by analyzing attention patterns involving [CLS] . Specifically, we train a lightweight neural network—ExpNet—that maps BERT’s self-attention patterns to token-level importance