Polyline Path Masked Attention for Vision Transformer

Zhongchen Zhao, Chaodong Xiao, Hui Lin, Qi Xie, Lei Zhang, Deyu Meng · 2025

Read Polyline Path Masked Attention for Vision Transformer by Zhongchen Zhao, Chaodong Xiao, Hui Lin, Qi Xie, Lei Zhang, Deyu Meng (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2506-15940.txt.

Introduction

Zhongchen Zhao and co-authors give vision transformers a sense of place in a 2025 paper. Two core needs drive image nets: long-range links and place sense. Self-attention gives the first but a weak form of the second. A state-space kin, Mamba2, bakes in next-door links through its mask. The team fuse the two: a mask drawn by a polyline scan path sits in the attention step.

The scan walks the image in bent lines that keep next-door ties. A full math read of the mask shape backs the design, with a fast way to build it. Tests span image sorts, object finds, and scene cuts. The new models beat past top forms on all three. On a scene cut bench they top a kin set by up to 1.3 points. Code is free for later work.

Catalog record

CREATORZhongchen Zhao, Chaodong Xiao, Hui Lin, Qi Xie, Lei Zhang, Deyu Meng
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE98 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2506-15940

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Polyline Path Masked Attention for Vision Transformer Zhongchen Zhao, Chaodong Xiao, Hui Lin, Qi Xie, Lei Zhang, Deyu Meng Abstract Global dependency modeling and spatial position modeling are two core issues of the foundational architecture design in current deep learning frameworks. Recently, Vision Transformers (ViTs) have achieved remarkable success in computer vision, leveraging the powerful global dependency modeling capability of the self-attention mechanism. Furthermore, Mamba2 has demonstrated its significant potential in natural language processing tasks by explicitly modeling the spatial adjacency prior through the structured mask. In this paper, we propose Polyline Path Masked Attention ( PPMA ) that integrates the self-attention mechanism of ViTs with an enhanced structured mask of Mamba2, harnessing the complementary strengths of both architectures. Specifically, we first ameliorate the traditional structured mask of Mamba2 by introducing a 2D polyline path scanning strategy and derive its corresponding structured mask, polyline path mask, which better preserves the adjacency relationships among image tokens. Notably, we conduct a thorough theoretical analysis on the structural characteristics of the proposed polyline path mask and design an efficient algorithm for the computation of the polyline path mask. Next, we embed the polyline path mask into the self-attention mechanism of ViTs, enabling explicit modeling of spatial adjacency prior. Extensive experiments on standard benchmarks, including image classification, object detection, and segmentation, demonstrate that our model outperforms previous state-of-the-art approaches based on both state-space models and Transformers. For example, our proposed PPMA-T/S/B models achieve 48.7%/51.1%/52.3% mIoU on the ADE20K semantic segmentation task, surpassing RMT-T/S/B by 0.7%/1.3%/0.3%, respectively. Code is available at https://github.com/zhongchenzhao/PPMA . 1 Introduction The research of foundational models has long been a cornerstone of deep learning. In computer vision, Convolutional Neural Networks (CNNs) huang2017densely ; resnet and Vision Transformers (ViTs) vaswani2017attention ; vit ; SwinTransformer ; huang2020ccnet currently represent the dominant architectures. Notably, ViTs have become the most mainstream architecture in large models through the powerful self-attention mechanism, which can capture the non-local self-similarity within global receptive fields. However, the quadratic complexity of the Transformer, when implementing self-attention, severely limits its application in large image processing models. Moreover, as shown in Fig. 1 (b), classic positional encoding methods vaswani2017attention ; SwinTransformer ; RoPE in ViTs lack the explicit modeling capability of spatial distance between image tokens, and largely ignore the important spatial adjacency priors in texture, shape, semantics, and so on. This increases the learning pressure and limits its capability for fine-grained image feature extraction. Compared to CNNs and Transformers, the recently proposed Mamba mamba achieves linear complexity while maintaining global receptive fields, demonstrating strong potential as the next-generation architecture. Specifically, Mamba follows the State Space Models (SSMs) paradigm and employs the selective scan mechanism with the state transition matrix to recursively propagate dependencies among Figure 1 : (a)-(b) Illustration of the modules in Mamba2 and ViT. (c) Our method adapts the structured mask of Mamba2 to 2D scanning and integrates it with ViT’s self-attention. tokens in a sequence. Building on this foundation, Mamba2 mamba2 further refines the state transition matrix into a lightweight structured mask and introduces a unified theoretical framework, Structured State Space Duality (SSD), to bridge SSMs and attention variants. Under SSD, the core selective scan mechanism of Mamba2 can be reformulated as a form of structured masked attention, i.e. , a Linear Attention katharopoulos2020transformers element-wise multiplied by the structured mask , as illustrated in Fig. 1 (a). Notably, this structured mask explicitly encodes the sequence adjacency of tokens, enabling Mamba2 to match or surpass Transformers across various natural language processing (NLP) tasks. Following its success in NLP, Mamba mamba has been rapidly adapted to various visual domains, including: high-level tasks (classification, object detection, segmentation zhu2024ViM ; Vmamba ; GrootV ; SpatialMamba ), low-level tasks (super-resolution, denoising, deraining MambaIR ; MambaIRv2 ; Freqmamba ), image generation DiM , video analysis Videomamba , point cloud analysis PointMamba ; Voxelmamba , and remote sensing images Rs-mamba . Although Mamba mamba has demonstrated impressive results on certain vision tasks, empirical results on high-level vision tasks demonstrate that even state-of-the-art (SOTA) Mamba-based backbones Vmamba ; GrootV ; SpatialMamba ; mlla still underperform SOTA Transformer-based backbones biformer ; RMT with a substantial performance gap. As shown in Fig. 1 (a), this gap mainly stems from two issues: (I) 1D Scanning Issue. Mamba’s 1D scanning strategy arranges the tokens of a 2D image into a 1D sequence, which inevitably disrupts the inherent spatial adjacency within 2D images and limits the effectiveness of its recursive selective scanning mechanism. (II) Weak Global Dependency Modeling Issue. The linear attention in Mamba2 omits the non-linear softmax layer, leading to a decrease in the precision and stability of global dependency modeling of images. In this paper, we present Polyline Path Masked Attention ( PPMA ), a brand-new method that effectively combines the advantages of ViTs and Mamba2. Specifically, as illustrated in Fig. 1 (c), to address the 1D Scanning Issue when applying current Mamba to 2D images, we propose a novel 2D polyline path scanning strategy and derive an efficient calculation method for its corresponding structured mask, the polyline path mask. Then, we embed the polyline path mask as an explicit positional encoding into the ViT framework. This not only avoids the Weak Global Dependency Modeling Issue of Mamba2, but also alleviates the positional encoding issue of ViTs. As a result, our method fully leverages the powerful global context modeling capability of the self-attention mechanism in ViTs together with the explicit spatial adjacency modeling capability of the polyline path mask in Mamba2, achieving SOTA performance on mainstream high-level vision tasks. To the best of our knowledge, this is the first work to integrate Mamba2’s structured mask mechanism into ViTs. The main contributions of this study are summarized as follows: • We propose a 2D polyline path scanning strategy for visual Mamba, which better preserves the inherent 2D spatial structure of images compared to existing scanning strategies. Building on this, we further derive a novel structured mask, termed polyline path mask, which is more suitable for 2D images than the traditional structured mask used in Mamba2. • We conduct a comprehensive theoretical analysis for the proposed 2D polyline path mask. Specifically, we theoretically prove that it can be decomposed into two 1D structured masks with clear physical meanings (i.e., horizontal and vertical scanning masks). More importantly, by leveraging this decomposability, we derive an efficient algorithm to reduce its computational complexity from \mathcal{O}(N^{2}) in the naive calculation to \mathcal{O}(N^{\frac{3}{2}}) . • The polyline path mask can be seamlessly integrated into various attention variants in a plug-and-play manner without introducing a substantial increase in computational complexity. In this paper, we incorporate it into the vanilla self-attention and criss-cross attention, deriving the Polyline Path Masked Attention (PPMA). • Leveraging PPMA, we construct a hybrid Mamba2-Transformer model. Experimental results demonstrate that our model achieves SOTA performance on standard benchmarks for image classification, object detection, and segmentation. 2 Related Work Vision Transformers. ViTs have become foundational in large-scale vision models such as SAM SAM and Sora brooks2024video , primarily due to their self-attention mechanism that captures long-range dependencies. Moreover, the spatial structural information provided by positional encodings (e.g., APE vaswani2017attention , RPE SwinTransformer , and RoPE RoPE ) is also crucial to ViTs. However, traditional positional encodings

Read the complete text