Sparsified State-Space Models are Efficient Highway Networks

Woomin Song [email protected], Jihoon Tack [email protected], Sangwoo Mo [email protected], Seunghyuk Oh [email protected], Jinwoo Shin [email protected] · 2025

Read Sparsified State-Space Models are Efficient Highway Networks by Woomin Song [email protected], Jihoon Tack [email protected], Sangwoo Mo [email protected], Seunghyuk Oh [email protected], Jinwoo Shin [email protected] (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2505-20698.txt.

Introduction

Woomin Song and co-authors prune state-space sequence models so their upper layers act like highways for information. State-space models rival attention-based ones by sweeping a recurrence over the input, but each token nudges the state only slightly, so old information can get stuck behind dense updates. The catalog creator field is fused with email addresses, so the clean head byline governs here. Their 2025 method, Simba, cuts tokens where they matter least.

Redundancy grows with depth, they observe, since upper layers hold the global picture. Simba therefore prunes upper layers hardest, scoring each token by its total effect on the final output, gathered through the recurrence itself. At equal computing cost it beat the Mamba baseline across language tasks. The authors show the highway effect directly: pruned models carried information across long spans better than the dense original.

Catalog record

CREATORWoomin Song [email protected], Jihoon Tack [email protected], Sangwoo Mo [email protected], Seunghyuk Oh [email protected], Jinwoo Shin [email protected]
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE46 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2505-20698

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Sparsified State-Space Models are Efficient Highway Networks Woomin Song [email protected], Jihoon Tack [email protected], Sangwoo Mo [email protected], Seunghyuk Oh [email protected], Jinwoo Shin [email protected] Abstract State-space models (SSMs) offer a promising architecture for sequence modeling, providing an alternative to Transformers by replacing expensive self-attention with linear recurrences. In this paper, we propose a simple yet effective trick to enhance SSMs within given computational budgets by sparsifying them. Our intuition is that tokens in SSMs are highly redundant due to gradual recurrent updates, and dense recurrence operations block the delivery of past information. In particular, we observe that upper layers of SSMs tend to be more redundant as they encode global information, while lower layers encode local information. Motivated by this, we introduce Simba, a hierarchical sparsification method for SSMs based on token pruning. Simba sparsifies upper layers more than lower layers, encouraging the upper layers to behave like highways. To achieve this, we propose a novel token pruning criterion for SSMs, measuring the global impact of tokens on the final output by accumulating local recurrences. We demonstrate that Simba outperforms the baseline model, Mamba, with the same FLOPS in various natural language tasks. Moreover, we illustrate the effect of highways, showing that Simba not only enhances efficiency but also improves the information flow across long sequences. Code is available at https://github.com/woominsong/Simba . Reviewed on OpenReview: https://openreview.net/forum?id=G1p0YwrX8X 1 Introduction State-space models (SSMs) ( Gu et al., 2022b ; Gu & Dao, 2023 ) offer a promising architecture for sequence modeling, efficiently handling sequences using linear recurrence structures. Thanks to this efficiency, SSMs have shown potential as an alternative to Transformers ( Vaswani et al., 2017 ) , which use the self-attention mechanism, incurring high computational costs for long sequences. In particular, Mamba ( Gu & Dao, 2023 ) has recently demonstrated that SSMs can scale up to billions of parameters and show comparable performance with Transformers in various domains ( Zhu et al., 2024 ; Li et al., 2024a ; Li et al., 2024b ) . After their success, numerous works have aimed to enhance SSMs and Mamba further. One popular approach involves hybrid models combining Transformers and SSMs ( Lieber et al., 2024 ; Poli et al., 2024 ) . This approach assists the models in retaining past information through the global memory of Transformers ( Vardasbi et al., 2023 ; Jelassi et al., 2024 ) . However, this compromises the efficiency of SSMs by reintroducing expensive self-attention. Instead of sacrificing efficiency, we explore an alternative direction to improve SSMs by comparing models on a fixed computational budget. Specifically, we investigate sparsification of large pre-trained SSMs, known to yield better models than training small ones from scratch ( Frankle & Carbin, 20189 ) . To this end, we first analyze the behavior of tokens in pre-trained SSMs. We observe that tokens in SSMs are highly redundant, as they are gradually updated over sequences. This redundancy is particularly noticeable in the upper layers. Furthermore, dense recurrence operations over the redundant tokens block the delivery of past information, potentially harming the contextual understanding of SSMs. (a) Conceptual illustration of Simba (b) Performance highlights Figure 1: Simba: hierarchical sparsification of SSMs via token pruning. (a) We found that tokens in state-space models (SSMs) are highly redundant, especially in the upper layers. Motivated by this observation, we propose hierarchically sparsifying pre-trained SSMs by progressively pruning tokens across layers. This results in models with a trapezoidal shape, featuring sparse upper layers that act like highways, enhancing efficiency and information flow of the original SSM. (b) We highlight results comparing Simba-2.8b with Mamba and Pythia, all with the same number of FLOPS. We report the mean accuracy over 6 NLP benchmarks and perplexity on the PG-19 dataset with 2k context, following the setups in Section 4 . Simba outperforms both models in accuracy and perplexity. Inspired by this, we propose Simba, a simple yet effective method to sparsify SSMs through token pruning (i.e., it is training-free). Our core idea is to sparsify SSMs into a hierarchical form, enforcing more sparsity in upper layers than in lower layers. As a result, the upper layers behave as highways to transmit past information, enabling efficient inference and facilitating the information flow across long sequences. Figure 1(a) illustrates the visual concept of our approach, obtaining a trapezoidal-shaped sparsified network. To implement this, we propose a novel token pruning criterion for SSMs. Specifically, our score measures the global influence of each token on the final output by reformulating SSM equations to accumulate the effect from local recurrences. This approach can also be viewed as an SSM extension of attention-based token pruning criteria used for Transformers ( Goyal et al., 2020 ) . We found that our criterion outperforms intuitive baselines, such as uniform pruning of tokens with even intervals. Our experiments show that Simba, obtained by sparsifying Mamba without any fine-tuning, significantly outperforms Mamba using the same number of FLOPS in various tasks. For instance, Simba consistently achieves better FLOPS-accuracy curves on 6 NLP benchmarks, including Lambada ( Paperno et al., 2016 ) , HellaSwag ( Zellers et al., 2019 ) , PIQA ( Bisk et al., 2020 ) , ARC-Challenge ( Clark et al., 2018 ) , ARC-Easy ( Clark et al., 2018 ) , and WinoGrande ( Sakaguchi et al., 2021 ) . Here, Simba obtained from Mamba-2.8b performs on par with the original Mamba-2.8b, despite using similar FLOPS to Mamba-1.4b, as highlighted in Figure 1(b) . For example, it achieves an average accuracy of 62.5% for 6 downstream NLP tasks, improving 58.8% of Mamba-1.4b. We also demonstrate the language modeling ability of Simba by measuring perplexity on the PG-19 dataset ( Rae et al., 2019 ) across different context lengths. Like the NLP benchmarks, Simba achieves better perplexity than Mamba using the same number of FLOPS. More importantly, Simba performs robustly over long sequences exceeding the pre-trained context length, such as twice longer than the trained length, unlike Mamba, which significantly deteriorates with length extrapolation. This supports the idea that the highway structures in Simba facilitate long sequence modeling. We further investigate the effect of highways in Simba. Somewhat unexpectedly, we found that Simba performed better than its original unpruned Mamba in some of our experiments, potentially benefiting from the highways created at the upper layers. To further investigate the positive effect of highways, we examine the information flow across layers by assessing the influence of the sequence tokens on the final output. We observe that Mamba relies on tokens near the end across all layers, while Simba also focuses on earlier tokens at the upper layers, showcasing the role of highways. 2 Related work State-space models (SSMs) are a powerful architecture for sequence modeling, integrating concepts from classic control theory ( Kalman, 1960 ) with recurrent neural networks ( Elman, 1990 ) . The key idea of SSMs is to employ linear recurrence ( Katharopoulos et al., 2020 ; Gu et al., 2022b ; Gu et al., 2022a ; Mehta et al., 2023 ; Smith et al., 2023 ; Fu et al., 2023 ; Orvieto et al., 2023 ; Poli et al., 2023 ; Peng et al., 2023 ; Sun et al., 2023 ; De et al., 2024 )

Read the complete text