On Pruning State-Space LLMs

Tamer Ghattas, Michael Hassid, Roy Schwartz · 2025

Read On Pruning State-Space LLMs by Tamer Ghattas, Michael Hassid, Roy Schwartz (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2502-18886.txt.

Introduction

Tamer Ghattas, Michael Hassid, and Roy Schwartz test whether state-space language models can be pruned like transformers, in a 2025 paper. These models are a cheaper alternative to transformers. Pruning, deleting weights, could cut their cost further. The team adapted several pruning methods to the state-space structure and applied them to four models across multiple tasks. The models proved robust to some methods, WANDA among them. Other methods broke them quickly. The paper maps which pruning tools are safe for this newer architecture.

Catalog record

CREATORTamer Ghattas, Michael Hassid, Roy Schwartz
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE33 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2502-18886

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

On Pruning State-Space LLMs Tamer Ghattas, Michael Hassid, Roy Schwartz Abstract Recent work proposed state-space models (SSMs) as an efficient alternative to transformer-based LLMs. Can these models be pruned to further reduce their computation costs? We adapt several pruning methods to the SSM structure, and apply them to four SSM-based LLMs across multiple tasks. We find that such models are quite robust to some pruning methods (e.g., WANDA), while using other methods lead to fast performance degradation. 1 1 1 www.github.com/schwartz-lab-NLP/SSM-Pruner Figure 1: Pruning SSM-based LLMs. Right : the Mamba SSM block: the input is linearly projected using five projection matrices ( W_{Z} , W_{X} , W_{B} , W_{C} , W_{\Delta} ), to be used in later parts of the block. Every SSM head is represented using two vectors (two rows for InProj and two columns for OutProj). Left : our different structure pruning methods. Each yellow cell represents a pruned element in the corresponding head. (1) State pruning : head extraction from W_{B} and W_{C} tensors then pruning the corresponding conv1d filters; (2) Head dimension pruning : head extraction from W_{X} , W_{Z} , W_{A} , W_{D} and W_{\Delta} , and pruning the corresponding conv1d filters and OutProj rows; (3) Head Merging : mean-pooling every two BC-heads and all corresponding components; (4) SSM-FLAP : adapting FLAP to SSMs, which prunes whole heads on all InProj sub-components heads and their correspondingly conv1d and OutProj. 1 Introduction Selective-State Space models (SSMs, Gu et al., 2022 ) have recently gained attention as an appealing alternative to transformers Gu and Dao (2024) ; Dao and Gu (2024) . SSMs leverage both selective memory capabilities and RNN Elman (1990) properties, showing comparable results against transformer-based peers. However, SSM-based LLMs are still parameter-heavy, which raises the question of how well they can be compressed. In this work, we focus on one of the key compression methods— pruning LeCun et al. (1989) . Modern LLM pruning methods have been developed and tested mostly for transformer components such as self-attention and feed-forward. Here we study how well SSM-based LLMs can be pruned. We adapt several structured pruning methods to SSMs, e.g., pruning different SSM heads using different criteria, or merging existing heads ( Fig. 1 ). We apply these methods to four SSM-based LLMs, along with WANDA Sun et al. (2024) , an unstructured pruning method that requires no adaptation. We compare all methods across six different tasks. Our results show that all models are robust to unstructured pruning with WANDA, even when reducing up to 50% of the SSM parameter count. We also observe that pruning SSM states leads to small degradation in almost all cases. In contrast, pruning the SSM heads leads to a sharp drop in performance in all cases. We also show that the output projection in SSM-based LLMs is much more sensitive to pruning than the input projection. Our results hint that SSM-based LLMs can indeed be made more efficient, but the choice of pruning method has a large effect on the pruning quality. 2 Background In this section, we discuss recent SSM architectures and some of the best performing pruning methods commonly used with transformer-based models. 2.1 SSM Architectures State Space Models ( Gu et al., 2022 ) are a class of seq2seq models that represent inputs via hidden states evolving over time. Unlike attention-based architectures, SSMs leverage structured representations to achieve sub-quadratic complexity. Recent work has shown that SSM-based LLMs can be trained and reach competitive performance to transformers-based LLMs. Mamba ( Gu and Dao, 2024 ) uses a “selective” SSM variant that adaptively focuses on certain parts of the input at each time step. Mamba-2 Dao and Gu (2024) builds upon structured space duality framework, which bridges the gap between recurrent-style processing and attention-like operations. This enables the use of hardware optimizations made for attention. By producing SSM parameters in parallel and simplifying its core layer, Mamba-2 is 2–8× faster than its predecessor while maintaining performance on par with transformers across various benchmarks. The SSM component in Mamba-2 is composed of several sub-components ( Fig. 1 ). The input is first passed through a linear layer (InProj), composed of the following tensors: W_{X} , W_{Z} \in\mathbb{R}^{D,H,P} ; W_{B} , W_{C} \in\mathbb{R}^{D,H,N} ; W_{\Delta} \in\mathbb{R}^{D,H} . This results in five matrices: X,B,C,\Delta, and Z . 2 2 2 Where D is the model dimension, H is the number of heads, P is the head dimension, and N is the state dimension. Then, X,B , and C are passed through a 1-D convolution layer. Its output, along with W_{\Delta} and two learned parameter matrices W_{A} , W_{D} \in\mathbb{R}^{1,H} , are passed to the SSD algorithm Dao and Gu (2024) . Its output is then joint with W_{Z} and normalized using RMS Norm Zhang and Sennrich (2019) , and projected back to D —the model dimension. Multi-head patterns in SSM-based LLMs Similarly to group-query attention in transformers (GQA; Ainslie et al., 2023 ), SSM-based LLMs allow grouping some of the SSM subcomponents. The choice of which components to merge is referred to as the multi-head pattern Dao and Gu (2024) . In contrast to the commonly used GQA pattern in transformers, different SSM-based LLMs use different patterns. For example, Mamba-2 Dao and Gu (2024) groups the W_{B} , W_{C} matrices, while Hybrid-Llama3-Mamba2-3B (HLM-3B; Wang et al., 2024 ) groups W_{X} and W_{B} . 2.2 Pruning Methods Pruning is frequently used to compress LLMs. Below we describe common pruning methods. Unstructured pruning induces sparsity in the model weights. Such methods reduce model size, though it is hard to translate this sparsity to runtime gains. A common implementation of this approach is magnitude pruning , which prunes unimportant parameters based on their absolute values Frankle and Carbin (2019) . A recent highly effective variant of magnitude pruning is WANDA Sun et al. (2024) , which also takes activations into account. Importantly, WANDA does not require fine-tuning, making it highly efficient for compressing LLMs. Structured pruning removes entire sections of model weights rather than individual elements Ma et al. (2023) ; Molchanov et al. (2019) ; Fan et al. (2019) . One such method is FLAP An et al. (2023) , which prunes based on the fluctuation of each input channel. It determines whether the output feature map can be recovered after pruning a column from the weight matrix. FLAP avoids the need for retraining and requires only a single forward pass for both pruning and bias compensation. Finally, group-query attention (GQA), which merges distinct attention KV heads using mean-pooling, can also be thought of as a structured pruning method. 3 Pruning SSM-based LLMs We aim to study the effect of different pruning methods on SSM-based LLMs. Below we describe how we adapt different pruning methods, developed for transformers, to SSMs. We note that within the Mamba-2 SSM component ( Fig. 1 ), the layer dimension dictates the shapes for the matrices within the input projection (InProj), which comprise approximately 67% of all parameters in the SSM component. The output projection (OutProj) and the 1d convlution layer (conv1d) are roughly 32% and < 1%, respectively. WANDA is an unstructured pruning method, which can be applied to SSM layers with minimal adjustments, as it originally operates

Read the complete text