CAMAL: Improving Attention Alignment and Faithfulness with Segmentation Masks
Rajdeep Singh Hundal, Yan Xiao, Jin Song Dong, Manuel Rigger · 2026
Read CAMAL: Improving Attention Alignment and Faithfulness with Segmentation Masks by Rajdeep Singh Hundal, Yan Xiao, Jin Song Dong, Manuel Rigger (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2605-08325.txt.
Introduction
Rajdeep Singh Hundal, Yan Xiao, Jin Song Dong, and Manuel Rigger use segmentation masks to make vision models look at the right places for the right reasons. Two qualities matter: attention should land on the truly discriminative regions, and the decision should actually depend on that attention.
Their method, CAMAL, extracts a model's class activation maps during training and compares them against ground-truth regions taken from the segmentation masks that many datasets already provide. Training then pushes the attention toward those regions. Because the supervision comes from existing annotations, the approach stays efficient and scalable while targeting both spatial accuracy and causal faithfulness of attention.
Catalog record
| CREATOR | Rajdeep Singh Hundal, Yan Xiao, Jin Song Dong, Manuel Rigger |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 59 KB |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2605.08325
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
CAMAL: Improving Attention Alignment and Faithfulness with Segmentation Masks Rajdeep Singh Hundal, Yan Xiao, Jin Song Dong, Manuel Rigger Abstract Many vision datasets now provide segmentation masks in addition to annotated images to support a wide range of tasks. In this work, we propose Class Activation Map Attention Learning ( CAMAL ), an efficient and scalable method that utilizes segmentation masks to improve attention alignment and faithfulness in vision models. Specifically, attention alignment refers to the degree to which a model’s attention aligns with ground-truth discriminative regions, while attention faithfulness refers to the degree to which a model’s attention influences its decision. Improving both attention alignment and faithfulness is essential for ensuring that model attention is both spatially accurate and causally meaningful. To improve attention alignment and faithfulness in vision models, CAMAL first extracts the model’s attention for each image during training and then compares the attention to ground-truth discriminative regions obtained from the corresponding segmentation masks. CAMAL then acts as an auxiliary regularizer, encouraging attention that aligns with ground-truth discriminative regions, while suppressing attention elsewhere. We evaluated CAMAL across two learning paradigms— Deep Learning (DL) and Deep Reinforcement Learning (DRL)—and observed consistent, significant improvements in both attention alignment and faithfulness. In particular, CAMAL yields statistically significant gains in attention alignment across all settings, and improves attention faithfulness by over 35% compared to recent work. Moreover, we show that improved attention alignment and faithfulness enhance explainability, while yielding improved or comparable generalization performance without increasing inference cost. These findings demonstrate that the spatial information contained within segmentation masks can be effectively leveraged to guide model attention across learning tasks. 1 Introduction As vision models are increasingly deployed in real-world applications, it is imperative to understand their decisions and correct them when necessary. To this end, Class Activation Maps (CAMs) ( 50 ) are widely used to interpret vision models. They highlight a model’s attention, that is, the model’s perceived discriminative regions in an image. This provides an explanation of which regions the model attends to during predictions. However, as an explainability technique, CAM approaches cannot ensure attention alignment —the degree to which a model’s attention aligns with ground-truth discriminative regions. In fact, recent work by 46 shows that models are still susceptible to shortcut learning, mistaking spurious regions for discriminative ones. Furthermore, as demonstrated by 1 , attention alignment alone does not guarantee attention faithfulness —the degree to which a model’s attention influences its decision. That is to say, attention might itself be inconsequential to the model’s decision-making process. Thus, CAMs offer interpretation, but lack corrective capability. To address this gap in corrective capability, recent work have proposed attention supervision . Notably, 45 , 29 , and 33 proposed auxiliary training objectives that minimize the distance between a model’s attention and reference regions that indicate where the attention should be. A common characteristic of these works is that they all use CLIP-based priors —pretrained zero-shot models originally proposed by OpenAI ( 36 ) —to derive the reference regions. We term these reference regions as pseudo-discriminative regions , a proxy for ground-truth discriminative regions. Crucially, because these priors originate from pretrained zero-shot models optimized for broad generalization rather than precise spatial localization, the reference regions they project may only coarsely approximate true discriminative regions ( 24 ) , a phenomenon we empirically observed in a pilot study (see Appendix B ). Consequently, this leads to attention being corrected with signals that are themselves untrustworthy . Many vision datasets now provide segmentation masks in addition to annotated images to support a wide range of tasks, including those used in Deep Learning (DL) ( 16 ; 20 ; 22 ) and Deep Reinforcement Learning (DRL) ( 23 ; 44 ; 11 ) . Furthermore, modern developments such as SAM 3 by Meta ( 6 ) significantly reduce the cost of obtaining high-quality segmentation masks by enabling prompt-based automatic segmentation. Crucially, these segmentation masks are typically human-verified, and thus serve as a trustworthy signal that reflects ground-truth discriminative regions. This increased accessibility to segmentation masks raises a natural question: can the well-structured, complete, and accurate spatial information provided within these masks be leveraged to reliably correct attention in vision models? In this work, we propose Class Activation Map Attention Learning ( CAMAL ) as an efficient and scalable approach towards attention supervision that leverages segmentation masks to reliably correct attention in vision models. Specifically, CAMAL leverages these masks to identify and encourage attention within ground-truth discriminative regions, while suppressing attention in spurious regions. At its core, CAMAL is based on the premise that attention supervision must (1) be grounded in trustworthy reference regions to ensure reliability, and (2) scale efficiently. We provide our source code at https://doi.org/10.5281/zenodo.20084662 to support reproducibility. Our contributions are: 1. We introduce CAMAL , a principled approach to attention supervision that grounds attention learning in trustworthy reference regions obtained from segmentation masks. 2. We formulate a mask-guided auxiliary regularization objective that jointly encourages attention within ground-truth discriminative regions and suppresses attention in spurious regions. In addition, we propose an efficient batch-level attention extraction method that reduces the typical computational overhead from O(B) to O(1) , enabling scalable supervision. 3. We provide a comprehensive and statistically rigorous evaluation of attention supervision across two learning paradigms, DL and DRL. In our evaluation, we show that CAMAL achieves consistent, statistically significant improvements in attention alignment and over 35% gains in attention faithfulness, while maintaining improved or comparable generalization across 18 DL settings and 4 DRL environments. 2 Related Work Recent work by 45 , 29 , and 33 improve attention through auxiliary regularization objectives that align attention with reference regions (see Appendix B for a more detailed discussion of these works). Beyond auxiliary regularization objectives, several other approaches have been explored for improving attention in vision models. Among these, input augmentation approaches improve attention by modifying training images. For example, 21 randomly masked patches in training images, while 47 swapped patches instead. Architectural approaches in the same vein have also been explored. For example, 49 augmented the network architecture by introducing an additional classifier head that takes masked feature maps as input, while 7 introduced a self-attention mechanism for each feature map. Both input and architectural augmentation approaches encourage the model to discover a broader set of ground-truth discriminative regions instead of relying on the most discriminative regions. CAMAL shares a similar motivation, but instead improves attention directly through an auxiliary regularization objective grounded in trustworthy reference regions, rather than indirectly through input or network augmentation. 3 Method Figure 1 : Architecture diagram for CAMAL . Given an input image I^{c} passed through a model (steps 1–2), CAMAL extracts the model’s attention H^{c} (steps 3–4) and compares it with ground-truth discriminative regions obtained from the corresponding segmentation mask M^{c} (step 5). Lastly, CAMAL regularizes the network by encouraging attention within ground-truth discriminative regions ( \alpha\uparrow ) while suppressing attention in spurious regions ( \beta\downarrow ) (steps 5–6). Importantly, while illustrated with a Convolutional Neural Network (CNN) for clarity, CAMAL can be naturally extended to a wide range of network architectures and tasks. We ground attention supervision in trustworthy reference regions obtained directly from segmentation masks, formulating a mask-guided auxiliary regularization objective that explicitly