Regularization, Semi-supervision, and Supervision for a Plausible Attention-Based Explanation

Duc Hau Nguyen, Cyrielle Mallart, Guillaume Gravier, Pascale Sébillot · 2025

Read Regularization, Semi-supervision, and Supervision for a Plausible Attention-Based Explanation by Duc Hau Nguyen, Cyrielle Mallart, Guillaume Gravier, Pascale Sébillot (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2501-12775.txt.

Introduction

Duc Hau Nguyen and co-authors test ways to make AI attention maps easier for people to trust, in this 2025 paper. Attention is the part of a language model that weighs which input words drive its answer. The weights can be shown as a map, offered as an explanation of the output. Yet in many models the weights scatter widely, so the map looks unconvincing to regular users.

The team adds three kinds of constraint to training so the maps sharpen. The routes lean on cleaner training signals: penalties that tidy the map, partly labeled data, and full labels where they exist. Their goal is an explanation plain users can follow and accept. The work was partly funded by a French research grant.

Catalog record

CREATORDuc Hau Nguyen, Cyrielle Mallart, Guillaume Gravier, Pascale Sébillot
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE25 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2501-12775

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Regularization, Semi-supervision, and Supervision for a Plausible Attention-Based Explanation Thanks: Work partially funded by grant ANR-19-CE38-0011-03 from the French national research agency (ANR). Duc Hau Nguyen, Cyrielle Mallart, Guillaume Gravier, Pascale Sébillot Abstract Attention mechanism is contributing to the majority of recent advances in machine learning for natural language processing. Additionally, it results in an attention map that shows the proportional influence of each input in its decision. Empirical studies postulate that attention maps can be provided as an explanation for model output. However, it is still questionable to ask whether this explanation helps regular people to understand and accept the model output (the plausibility of the explanation). Recent studies show that attention weights in the RNN encoders are hardly plausible because they spread on input tokens. We thus propose 3 additional constraints to the learning objective function to improve the plausibility of the attention map: regularization to increase the attention weight sparsity, semi-supervision to supervise the map by a heuristic and supervision by human annotation. Results show that all techniques can improve the attention map plausibility at some level. We also observe that specific instructions for human annotation might have a negative effect on classification performance. Beyond the attention map, the result of experiments on text classification tasks also shows that no matter how the constraint brings the gain, the contextualization layer plays a crucial role in finding the right space for finding plausible tokens. 1 Introduction Attention mechanisms [ 15 ] play a crucial role in recent success across many natural language processing (NLP) tasks and are present in most recent neural models. As a layer in a complex neural network, the mechanism attributes weight to each input token and encodes by weighted summing the input vectors into a context vector. When this context vector is used for prediction, the weight vector, also called attention map [ 11 ] , can be considered as an explanation by showing the degree of influence of each input token in the prediction. Despite the potential of attention maps as a form of explanation, there are concerns [ 11 ] about their validity on two properties that are not guaranteed: faithfulness and plausibility. Faithfulness, a widely discussed problem [ 19 , 3 ] , focuses on whether the weight associated with a token reflects its influence on the prediction. Plausibility refers to the extent to which the attention map can resemble human reasoning [ 19 , 31 ] . While plausibility is an interesting feature that allows to present an easily comprehensible way to individuals with limited knowledge of neural models without additional computational costs, the contributions in this direction remain limited and rare. Given that multiple studies have suggested that raw attention weights lack plausibility (see, e.g., [ 20 ] ), the issue of forcing their plausibility is an obvious one that calls for further exploration. As it is proven possible to incorporate constraints on attention while maintaining satisfactory performance [ 25 , 12 , 31 ] , we propose three approaches for enforcing plausibility constraints on attention maps, namely, sparsity regularization, semi-supervised learning, and supervised learning. The main contributions in this paper are : (1) we can to some extent force the model plausibility (as demonstrated by supervision) at no accuracy cost, (2) both regularization and semi-supervision can optimize the plausibility but the latter offers a solution without compromising performance and (3) the deep contextualization is harmful to attention plausibility. The last result provides insight into why this hardly transfers to transformers. 2 Related Works The attention mechanism is widely used as a possible feature to explain the model decision [ 23 , 28 ] . However, the local explanation is facing an issue that the attention map can be manipulated while keeping the same prediction [ 12 , 31 , 29 ] . While this feature is considered as a weak faithful explanation [ 27 , 1 ] , this enables the selection of a plausible map. Among the few studies on attention supervision, [ 18 ] showed that supervision can harm classification performance in sentiment classification tasks. Regularization was considered to circumvent the issue of a rather flat distribution of attention weights as reported by [ 13 ] . [ 19 ] suggested an additional constraint in the learning objective to force this representation to be sparse. To overcome the lack of human references in many datasets, many contributions offer task-specific solutions, such as [ 22 ] guiding attention based on topic-related vocabulary and [ 14 ] using a WordNet-based heuristic for evidence inference, while [ 20 ] provides an effective heuristic map that is closer to human annotation but only for NLI. While the authors of existing techniques have not fully explored their effects and limitations in different tasks, this study aims to provide a comprehensive view of how different techniques improve attention plausibility. Hard attention, also referred to as rationalized learning in the literature [ 5 ] , is an alternative form of the attention mechanism that comprises two components: a generator function that masks irrelevant input tokens, and a predictor that is trained to make predictions on the remaining inputs. While hard-attention is advantageous with respect to soft-attention in robustness and faithfulness aspects, it introduced a trade-off between sparsity and accuracy because the full context is inaccessible [ 24 ] . Other post-hoc explanation techniques can provide faithful explanations (such as gradient-based methods or feature suppression), they present two main drawbacks: (i) incurring additional computational costs during each inference and (ii) offering benefits only to the model developer, without the flexibility to impose constraints for plausible explanations [ 2 ] while its explanation cannot be guaranteed to be plausible for end-users [ 3 , 21 ] . To the best of our knowledge, no existing study has brought a broad and comprehensive overview of how different techniques improve attention plausibility. Although regularization techniques are independent of human annotation and heuristics can overcome the lack of human annotation, it is still unclear how they improve plausibility compared to supervision. Furthermore, the authors of the existing techniques suggest improvement without questioning their implications and limitations in different tasks, especially in soft-attention models. This study focuses on addressing these fundamental issues and does not include a comparative analysis of hard-attention techniques and post-hoc explanation methods but they are promising for future works. 3 Tasks and Datasets To ensure the generalization of our findings across different tasks, we investigate three different datasets from the ERASER benchmark [ 6 ] and [ 30 ] designed for plausibility studies. The e-SNLI corpus [ 4 ] , a reference dataset in natural language inference (NLI), consists of pairs of sentences, a premise and a hypothesis with a label stating whether the hypothesis entails, contradicts, or is unrelated to the premise. The annotators also answered the question Why is a pair of sentences in a relation of entailment, neutrality, or contradiction? by highlighting the relevant words in both the premise and hypothesis and providing a short explanatory text. The corpus consists of 549,367 sentence pairs for training and 9,842 pairs for the validation and test sets respectively. Note that the SNLI corpus is known to have artifacts [ 10 ] , where some

Read the complete text