V“Mean”ba: Visual State Space Models only need 1 hidden dimension
Tien-Yu Chi, Hung-Yueh Chiang, Chi-Chih Chang, Ning-Chi Huang, Kai-Chiang Wu · 2024
Read V“Mean”ba: Visual State Space Models only need 1 hidden dimension by Tien-Yu Chi, Hung-Yueh Chiang, Chi-Chih Chang, Ning-Chi Huang, Kai-Chiang Wu (2024) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2412-16602.txt.
Introduction
Chi and co-authors present a computer vision paper, published in 2024, on shrinking a family of image models so they run on small devices. The leading image models are powerful but slow, because their core step grows steeply with image size. A newer family processes images in a line, step by step, which scales far better. On modern chips, though, that line process underuses the hardware and creates a bottleneck.
The team's fix is bold: cut the model's inner width down to a single channel. Their method needs no retraining, so it can be applied to trained models directly. The squeezed model keeps most of its accuracy while running much faster. That trade suits phones, cameras, and other gear with tight power budgets. The paper shows that for these models, a wide inner space is mostly waste.
Catalog record
| CREATOR | Tien-Yu Chi, Hung-Yueh Chiang, Chi-Chih Chang, Ning-Chi Huang, Kai-Chiang Wu |
|---|---|
| YEAR | 2024 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 22 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2412-16602
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
V“Mean”ba: Visual State Space Models only need 1 hidden dimension Tien-Yu Chi, Hung-Yueh Chiang, Chi-Chih Chang, Ning-Chi Huang, Kai-Chiang Wu Abstract Vision transformers dominate image processing tasks due to their superior performance. However, the quadratic complexity of self-attention limits the scalability of these systems and their deployment on resource-constrained devices. State Space Models (SSMs) have emerged as a solution by introducing a linear recurrence mechanism, which reduces the complexity of sequence modeling from quadratic to linear. Recently, SSMs have been extended to high-resolution vision tasks. Nonetheless, the linear recurrence mechanism struggles to fully utilize matrix multiplication units on modern hardware, resulting in a computational bottleneck. We address this issue by introducing VMeanba , a training-free compression method that eliminates the channel dimension in SSMs using mean operations. Our key observation is that the output activations of SSM blocks exhibit low variances across channels. Our VMeanba leverages this property to optimize computation by averaging activation maps across the channel to reduce the computational overhead without compromising accuracy. Evaluations on image classification and semantic segmentation tasks demonstrate that VMeanba achieves up to a 1.12x speedup with less than a 3% accuracy loss. When combined with 40% unstructured pruning, the accuracy drop remains under 3%. 1 Introduction Computer vision has advanced significantly due to deep learning and the availability of large-scale datasets. Convolutional Neural Networks (CNNs) have become foundational for tasks such as image classification [ 11 , 22 , 9 ] and object detection [ 6 , 5 , 21 ] . However, CNNs struggle to capture long-range dependencies. Vision Transformers (ViTs) [ 3 , 18 , 24 ] which utilize self-attention mechanisms, effectively address this limitation but suffer from high computational costs due to quadratic complexity. To mitigate these costs, research has focused on reducing ViT complexity [ 25 , 1 , 18 , 17 , 15 ] , applying model compression techniques [ 19 , 14 , 30 , 27 , 24 , 13 ] , and exploring alternative architectures like RWKV and State Space Models (SSMs) [ 20 , 8 , 4 , 7 ] . State Space Models (SSMs) have recently garnered attention in computer vision as efficient and effective alternatives to Vision Transformers (ViTs), demonstrating competitive performance across various tasks [ 16 , 29 , 12 , 23 ] . For example, VMamba [ 16 ] achievesd 82.6% top-1 accuracy on ImageNet-1k [ 2 ] , surpassing Swin Transformer [ 18 ] by 1.3% with comparable FLOPs. However, despite reducing computational complexity, SSMs still fail to fully utilize matrix multiplication units on GPUs, creating a bottleneck in vision-based SSM models. Figure 1: The GPU kernel time of each operation in a VMamba block. The latency is measured using feature maps with an input resolution of 224\times 224 . We rank the kernels by their latency and highlights the top-5 time-consuming kernels on the bar chart. The selective scan operation is one of the major contributors in the VMamba block. To this end, we first analyze the latency breakdown of VMamba [ 16 ] and identify the selective scan operation [ 7 ] as one of the key bottlenecks in inference. Figure 1 shows that the selective scan accounts for 14.3% of the total kernel time in a VMamba block. While optimizing selective scan operation is critical for enhancing SSM efficiency, few research works address this problem and optimizing the efficiency of SSMs remains unexplored. In this paper, we propose VMeanba , a novel activation compression method designed to optimize the selective scan operation in VMamba blocks. The high-level overview of VMeanba is presented in Figure 2 . The key idea is to reduce the input tensor’s channel dimensions in the associate scan operation by applying a mean operation. Through analysis of the weight and activation distributions in the trained VMamba model, we identified a smooth pattern with small variances that allows for dimensional reduction. Based on this observation, we developed the VMeanba block to exploit this pattern, resulting in a more efficient associate scan operation without compromising accuracy. Experimental results demonstrate that VMeanba achieves up to a 1.12x speedup with less than a 3% accuracy loss. To the best of our knowledge, this is the first work optimizing of the selective scan operation in VMamba. Figure 2: Overview of the VMeanba block. VMeanba reduced the channel dimension of the inputs to the associated scan operation by applying a transform T , thereby simplifying the computation. The proposed VMeanba components are highlighted in red, while the original selective scan components are shown in blue and green, with the green block indicating the main area of optimization. 2 Methods 2.1 Distribution Analysis of VMamba We conduct an in-depth investigation into the characteristics of each layer’s output within the Mamba block of VMamba. The output is denoted as y_{layer}\in\mathbb{R}^{B\times D\times L} , where B is the batch size, D is the inner channel dimension utilized by the scan algorithm within the Mamba block, and L is 4x of the feature map size HW . Our analysis revealed that for each y_{layer} , the distribution of values across the inner channel dimension is remarkably consistent across different data points, as illustrated in figure 3 . This observation raised a critical question: Is the full dimensionality D necessary for each y_{layer} ? Figure 3: The figure illustrates the distribution of inner dimension values of y_{layer} across various data points as a function of sequence length. Notably, the distribution remains remarkably consistent across different data points for identical l values, as indicated by the arrows. The distribution for l=195 , shown on the right, provides further evidence of this concentration. To explore this, we hypothesized a property described by equation ( 1 ): \begin{gathered}y_{\text{layer}}[:,d,:]\sim y_{layer}[:,d^{\prime},:]\quad\\ \forall d,d^{\prime}\in[1,D],d\neq d^{\prime}\end{gathered} (1) Given that the scan algorithm in the Mamba block performs a linear transformation, this unique property of y_{layer} can be attributed to the inputs \bar{A} , \bar{B}u_{t} and C to the SSM system. Consequently, we propose that a reduced set of inputs (\bar{A}_{reduce},(\bar{B}u_{t})_{reduce},C_{reduce}) , referred to collectively as I_{basis} , can effectively represent the original inputs (\bar{A},\bar{B}u_{t},C) . By leveraging these reduced inputs, we can optimize the computational efficiency of each Mamba block. 2.2 VMeanba Building on the findings from section 2.1 , we indroduce a new model inference efficiency optimization method called VMeanba , which computes I_{basis} for each Mamba block using mean operators. We further design a pipeline to select which layers in the model will undergo this optimization. VMeanba block. The I_{basis} is derived by having a transform function T that maps the original inputs (\bar{A},\bar{B}u_{t},C) to reduced dimension inputs. After processing by the original Mamba block, the output is recovered using an inverse transform function T^{-1} . This entire process can be expressed as equation ( 2 ). \begin{gathered}y_{layer}=T^{-1}(Mamba(T(\bar{A},\bar{B}u_{t},C)))\end{gathered} (2) In this process, T is defined as the mean operator applied along the inner channel dimension axis, and T^{-1} is defined as the broadcast operator. While the mean transform may lead to a loss of information, it significantly reduces the dimensionality of the inputs from D to 1 , with our experiments demonstrating that model performance is maintained. The computational complexity analysis is provided in B . Layer Selection. We