Know Your Attention Maps: Class-specific Token Masking for Weakly Supervised Semantic Segmentation
Joëlle Hanna, Damian Borth · 2025
Read Know Your Attention Maps: Class-specific Token Masking for Weakly Supervised Semantic Segmentation by Joëlle Hanna, Damian Borth (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2507-06848.txt.
Introduction
Labeling every pixel of an image by hand is expensive, so weak supervision learns from image-level tags alone. Joëlle Hanna and Damian Borth take a different route in 2025, reading the attention maps inside a vision transformer as the segmentation signal itself.
Their transformer carries one special token per class and trains with random masking, which pushes each token to own its class. At test time the attention maps of the tokens for the predicted classes are merged into draft masks. Those drafts beat rival methods on standard and specialist data sets, and a segmentation model trained on them nearly matched fully supervised results. The method needs no external activation-map machinery.
Catalog record
| CREATOR | Joëlle Hanna, Damian Borth |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 31 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2507-06848
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Know Your Attention Maps: Class-specific Token Masking for Weakly Supervised Semantic Segmentation Joëlle Hanna, Damian Borth Abstract Weakly Supervised Semantic Segmentation (WSSS) is a challenging problem that has been extensively studied in recent years. Traditional approaches often rely on external modules like Class Activation Maps [ 36 ] to highlight regions of interest and generate pseudo segmentation masks. In this work, we propose an end-to-end method that directly utilizes the attention maps learned by a Vision Transformer (ViT) [ 7 ] for WSSS. We propose training a sparse ViT with multiple [CLS] tokens (one for each class), using a random masking strategy to promote [CLS] token - class assignment. At inference time, we aggregate the different self-attention maps of each [CLS] token corresponding to the predicted labels to generate pseudo segmentation masks 1 1 1 Code is available at github.com/HSG-AIML/TokenMasking-WSSS . Our proposed approach enhances the interpretability of self-attention maps and ensures accurate class assignments. Extensive experiments on two standard benchmarks and three specialized datasets demonstrate that our method generates accurate pseudo-masks, outperforming related works. Those pseudo-masks can be used to train a segmentation model which achieves results comparable to fully-supervised models, significantly reducing the need for fine-grained labeled data. 1 Introduction Semantic segmentation is a fundamental task in computer vision, aiming to assign a class label to each pixel in an image. This fine-grained understanding of visual data is essential for various domain-specific applications, such as medical imaging, autonomous driving, and remote sensing. Traditionally, achieving high performance in semantic segmentation has relied heavily on the availability of large annotated datasets. However, acquiring such fine-grained labels is often labor-intensive, expensive or even impractical, especially in specialized domains where expert knowledge is required. Figure 1 : Our method uses multiple [CLS] tokens in ViTs to encourage class-specific self-attention maps. By aggregating these maps, we generate a pseudo-mask (bottom left) comparable to the ground truth (bottom right), without ever using the fine-grained segmentation labels. To address these challenges, weakly supervised semantic segmentation (WSSS) has emerged as a promising alternative. WSSS leverages weaker forms of supervision, such as image-level labels, to reduce the burden of detailed annotations. A common approach in WSSS involves the use of Class Activation Maps [ 35 ] , which highlight the most discriminative regions of an image for a particular class. Despite their effectiveness, CAMs suffer from limitations such as coarse localization and an inability to capture fine details, as they only emphasize the most prominent features. Moreover, generating CAMs requires an additional module on top of the existing model, adding complexity to process. Given these limitations, an alternative approach is to leverage the self-attention maps of Vision Transformers (ViTs, [ 7 ] ). Unlike CAMs, which rely on additional modules, the attention mechanism in ViTs inherently provides interpretability by capturing spatial dependencies across the entire image. Vision Transformers, with their self-attention mechanisms, have shown remarkable performance in various vision tasks. However, a significant challenge in using ViTs for WSSS is the inability to assign specific classes to individual attention heads, making it difficult to interpret and utilize their outputs for segmentation tasks effectively [ 9 ] . In this work, we propose a novel approach for WSSS that leverages the multi-head self-attention mechanism of ViTs in a more structured manner. Instead of relying on a single [CLS] token that aggregates global information, we introduce multiple [CLS] tokens, each designed to correspond to a specific class in the segmentation task. This allows the model to decompose the scene into distinct class-specific representations. Ideally, each [CLS] token would correspond to one class. However, there is no inherent guarantee that each [CLS] token will only capture information relevant to its intended class. To address this, we implement a novel masking mechanism on the [CLS] token output embeddings to help ensure that each [CLS] token is properly assigned to a single class (Figure 1 ). Furthermore, since not all attention heads contribute equally to class-specific feature extraction, we introduce a pruning mechanism to remove redundant or noisy heads. By selectively removing less important heads, we aim to sharpen the attention mechanism, making the remaining heads more interpretable. The ultimate goal is to leverage these refined attention heads to create accurate class-specific segmentation masks, thus enabling the ViT to perform weakly supervised semantic segmentation effectively. This work aims to bridge the gap between the powerful capabilities of ViTs and the need for class-specific feature embeddings in complex vision tasks. The contributions of this work can be summarized as follows: • We propose to use multiple [CLS] tokens instead of one, and develop a masking mechanism for the [CLS] tokens that reliably assigns each [CLS] token to its respective class. • We introduce attention head pruning during training, significantly reducing noise in self-attention maps and improving pseudo-mask quality. • We validate the application of this method in WSSS on two standard datasets and across three distinct domains: remote sensing, medical imaging, and general scene understanding. Each domain presents unique challenges: remote sensing requires accurate parsing of complex landscapes, medical imaging demands precise object-centric segmentation for diagnostic purposes, and general scene understanding involves diverse and sparse labels. • We demonstrate that our work can effectively reduce the need for fine-grained labeled data, benefiting both small-scale and large-scale datasets. 2 Related Works 2.1 Weakly Supervised Semantic Segmentation Weakly supervised semantic segmentation (WSSS) offers an alternative to traditional supervised methods by relying on less precise forms of supervision, such as image-level labels [ 5 , 30 ] , bounding boxes [ 12 , 25 ] , or scribbles [ 15 , 27 ] . WSSS approaches can be broadly categorized into single-stage and multi-stage methods, each with distinct workflows and objectives. Single-stage WSSS methods aim to directly generate segmentation maps from weak labels without intermediate processing. These approaches often leverage end-to-end models that integrate weak supervision signals with advanced architectures. Recent advancements include Vision Transformers (ViTs) specifically tailored for WSSS. Hanna et al . [ 9 ] introduced a multimodal ViT enforcing sparsity in attention heads during training, enabling the direct generation of high-quality segmentation maps. Similarly, Xu et al . [ 33 ] proposed a Multi-Class Token Transformer that employs class tokens to enhance localization for multiple objects in a single forward pass. These single-stage approaches simplify the WSSS pipeline by bypassing the need for pseudo-mask refinement stages, making them computationally efficient and suitable for real-time applications. Multi-stage WSSS methods, on the other hand, involve an iterative process where intermediate representations, such as pseudo segmentation masks (pseudo-masks), are generated and refined before final segmentation. This process typically begins with generating coarse localization maps from weak labels, followed by refinement and re-training in a fully supervised manner. One common approach is based on Class Activation Maps (CAMs), initially proposed by Zhou et al . [ 36 ] , which provide rough localization cues by highlighting discriminative image regions. Building on CAMs, Ahn et al . [ 1 ] introduced AffinityNet, which refines these maps by considering pixel affinities to propagate segmentation information. Wei et al . [ 31 ] extended this idea with Adversarial Erasing, iteratively suppressing confident CAM regions to uncover complete object extents. Further advancements in multi-stage WSSS have integrated various forms of weak