SF-Mamba: Rethinking State Space Model for Vision
Masakazu Yoshimura, Teruaki Hayashi, Yuki Hoshino, Wei-Yao Wang, Takeshi Ohashi · 2026
Read SF-Mamba: Rethinking State Space Model for Vision by Masakazu Yoshimura, Teruaki Hayashi, Yuki Hoshino, Wei-Yao Wang, Takeshi Ohashi (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2603-16423.txt.
Introduction
Yoshimura and co-authors rethink Mamba models for image tasks. Vision Transformers scale poorly, but Mamba's one-way scan limits patch contact. Added scans in prior work cost speed and shuffle data often. Two fixes anchor SF-Mamba. Patch swapping feeds two-way flow into a single scan. Batch folding with timed state resets boosts GPU parallelism. Tests on classification, detection, and segmentation beat strong baselines. Throughput also rose across model sizes.
Catalog record
| CREATOR | Masakazu Yoshimura, Teruaki Hayashi, Yuki Hoshino, Wei-Yao Wang, Takeshi Ohashi |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 68082 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2603.16423
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
SF-Mamba: Rethinking State Space Model for Vision Masakazu Yoshimura, Teruaki Hayashi, Yuki Hoshino, Wei-Yao Wang, Takeshi Ohashi Abstract The realm of Mamba for vision has been advanced in recent years to strike for the alternatives of Vision Transformers (ViTs) that suffer from the quadratic complexity. While the recurrent scanning mechanism of Mamba offers computational efficiency, it inherently limits non-causal interactions between image patches. Prior works have attempted to address this limitation through various multi-scan strategies; however, these approaches suffer from inefficiencies due to suboptimal scan designs and frequent data rearrangement. Moreover, Mamba exhibits relatively slow computational speed under short token lengths, commonly used in visual tasks. In pursuit of a truly efficient vision encoder, we rethink the scan operation for vision and the computational efficiency of Mamba. To this end, we propose SF-Mamba, a novel visual Mamba with two key proposals: auxiliary patch swapping for encoding bidirectional information flow under an unidirectional scan and batch folding with periodic state reset for advanced GPU parallelism. Extensive experiments on image classification, object detection, and instance and semantic segmentation consistently demonstrate that our proposed SF-Mamba significantly outperforms state-of-the-art baselines while improving throughput across different model sizes. We will release the source code after publication. Figure 1 : Top-1 accuracy and throughput on ImageNet-1K classification. SF-Mamba offers superior accuracy–throughput trade-offs compared to state-of-the-art architectures. 1 Introduction The field of deep learning for vision has undergone several architectural shifts, driving progress across a wide range of applications including classification, segmentation, and detection ( 11 ; 25 ; 22 ; 46 ; 41 ; 45 ; 1 ) . More recently, Vision Transformers (ViTs) ( 13 ) have emerged as the dominant paradigm and offer strong flexibility and generalizability compared to convolution-based models by tokenizing images into patches and applying the self-attention mechanism ( 49 ) . ViT-based vision models have been widely used for multi-modal learning tasks ( 24 ; 16 ) ; however, one of the main limitations of ViTs is the quadratic complexity for computing attention in terms of sequence length, which hinders the scalability to high-resolution inputs and large datasets with limited computational resources. Mamba ( 18 ) introduces a selective state-space model (SSM), which enables data-dependent flexible token scanning in a left-to-right order, thereby achieving powerful but efficient processing with linear-time complexity. Building on its success, Mamba has been extended to the vision domain, achieving higher accuracy while being efficient in terms of memory cost, FLOPs, and the number of parameters ( 63 ; 30 ; 38 ) . In addition, recent studies suggest that visual Mamba has good transfer learning capabilities comparable to or even surpassing those of ViTs ( 57 ; 17 ) , and it has the potential to replace the ViT-based foundation model ecosystem. However, many visual Mamba models suffer from slow processing speeds, especially on low-resolution images, which makes them not truly efficient. One reason is that Mamba adopts a recurrent left-to-right scanning mechanism, which prevents earlier patches from accessing information in future patches. As a result, many visual Mamba methods employ a multi-directional scan strategy, where the input sequence is rearranged and processed from multiple directions (e.g., from top-left to bottom-right, bottom-right to top-left). This allows models to compensate for the inability of standard Mamba to reference future patches and yields strong performance on vision tasks. However, such a rearrangement incurs substantial overhead during both training and inference. In fact, MambaVision ( 20 ) achieves a fast inference by avoiding costly multi-scan and instead rely on attention layers appended after the unidirectional scan. These attention layers enable information to flow backward, allowing earlier tokens to indirectly benefit from later tokens while retaining the efficiency of unidirectional Mamba. Yet, relying solely on attention for backward information flow poses a limitation: backward context can only be injected in deeper layers, leaving shallower layers deprived of future information and potentially restricting the expressiveness of the representation. Another reason why visual Mamba is slow lies in Mamba itself. As reported in the Mamba paper, unless the token length exceeds around 1000–2000, it is slower than Attention. Unless the task involves high-resolution images, the token length of the vision patches typically remains below 1000. In this paper, we rethink visual Mamba from two perspectives in pursuit of a truly efficient image encoder. The first is the data flow. Instead of using the slower multi-directional scan, we adopt a unidirectional scan. However, this approach lacks future-to-past information flow, which is crucial to generate high-quality features. To address this, we propose an auxiliary patch swapping that enables future-to-past information flow within a unidirectional scan (Sec. 3.2 ). It introduces two additional tokens mixing the corresponding directional flow, which does not require significant burden compared with existing multi-scan approaches. The second perspective addresses the inefficiency of Mamba when processing short sequences. We attribute this limitation to suboptimal GPU parallelization, and to mitigate it, we introduce batch folding with periodic state reset (Sec. 3.3 ). This method reshapes batched inputs to maximize GPU thread utilization while preserving independence across sequences, thereby enhancing parallel efficiency. To this end, we propose SF-Mamba , which is equipped with the two key innovations of swapping and folding. Extensive experiments on image classification (Fig. 1 ), object detection, and semantic/instance segmentation demonstrate that SF-Mamba consistently outperforms state-of-the-art baselines while achieving faster inference, paving a new path toward efficient and effective Mamba architectures for vision. In summary, our main contributions are three-fold: • Efficient uni-scan for non-causal ordering. We propose a lightweight mechanism, auxiliary patch swapping, that introduces two learnable auxiliary tokens and a parameter-free swap operation, enabling bidirectional information flow across layers with negligible overhead compared to existing multi-scan approaches. • Efficient GPU parallelism for vision tasks. To address inefficiency in low-resolution vision tasks, we design a batch folding strategy that merges the batch and sequence dimensions, maximizing GPU utilization while preserving the independence of hidden states across sequences. This method can speed up any Mamba-based method especially with short sequence processing. • Empirical validation across various tasks. Experiments on classification, detection, and segmentation show that SF-Mamba outperforms state-of-the-art CNN-, Transformer-, hybrid CNN-Transformer-, and Mamba-based baselines. 2 Related Work CNNs and Vision Transformers. Convolutional Neural Networks (CNNs) first led to breakthroughs in large-scale image classification ( 11 ; 25 ) , with deeper networks such as VGG ( 46 ) and ResNet ( 22 ) extending success to segmentation ( 33 ) and detection ( 42 ) . Vision Transformers (ViTs) ( 13 ) , inspired by self-attention ( 49 ) , have since become the dominant paradigm by effectively modeling long-range dependencies. Follow-up works such as DeiT ( 14 ) , and Swin Transformer ( 31 ) improved efficiency and scalability. Models trained on large-scale data are used as a foundation for a variety of tasks ( 37 ; 45 ; 48 ) . Visual State Space Models. To address the quadratic cost of attention, state-space models (SSMs) have emerged as efficient alternatives. Mamba ( 18 ) introduced selective state spaces, enabling linear-time complexity with strong long-range modeling. Inspired by this success, many visual Mamba variants ( 63 ; 30 ) extended SSMs to visual data. Hybrid Architectures in Vision. Beyond single-paradigm designs, recent studies have demonstrated that