SeRpEnt: Selective Resampling for Expressive State Space Models
Stefano Rando, Luca Romani, Matteo Migliarini, Luca Franco, Denis Gudovskiy, Fabio Galasso · 2025
Read SeRpEnt: Selective Resampling for Expressive State Space Models by Stefano Rando, Luca Romani, Matteo Migliarini, Luca Franco, Denis Gudovskiy, Fabio Galasso (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2501-11729.txt.
Introduction
Rando and co-authors probe choice in state space tools and build SeRpEnt on that base. State space tools, SSMs, rose of late as a swap for Transformers. They dodge two flaws of care tools: square cost and long tie loss. The Mamba form matches Transformers with no care at all. Its edge rests on a choice step for state marks in the form. Choice was scored by tests lone, and its cause stayed dim.
Their read ties choice time spans to facts flow in runs. Choice spans act as straight near-forms of facts in the flow. SeRpEnt then uses choice to squeeze runs by facts load. A re-pick step groups bits by their facts load in the run. Tests on the Long Range set and word tasks show gains. The re-pick step drives those gains in the score reads.
Catalog record
| CREATOR | Stefano Rando, Luca Romani, Matteo Migliarini, Luca Franco, Denis Gudovskiy, Fabio Galasso |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 30 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2501-11729
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
SeRpEnt: Selective Resampling for Expressive State Space Models Stefano Rando, Luca Romani, Matteo Migliarini, Luca Franco, Denis Gudovskiy, Fabio Galasso Abstract State Space Models (SSMs) have recently enjoyed a rise to prominence in the field of deep learning for sequence modeling, especially as an alternative to Transformers. Their success stems from avoiding two well-known drawbacks of attention-based models: quadratic complexity with respect to the sequence length and inability to model long-range dependencies. The SSM variant Mamba has demonstrated performance comparable to Transformers without any form of attention, thanks to the use of a selective mechanism for the state parameters. Selectivity, however, is only evaluated empirically and the reasons of its effectiveness remain unclear. In this work, we show how selectivity is related to the sequence processing. Our analysis shows that selective time intervals in Mamba act as linear approximators of information. Then, we propose our SeRpEnt architecture, a SSM that further exploits selectivity to compress sequences in an information-aware fashion. It employs a resampling mechanism that aggregates elements based on their information content. Our empirical results in the Long Range Arena benchmark and other language modeling tasks show benefits of the SeRpEnt’s resampling mechanism. keywords Sequence Modeling, Long Range Dependencies, State Space Models, Resampling, Pooling 1 Introduction Sequence models, and more specifically language models, have gained the spotlight in the deep learning research due to their role as the backbone of foundational models (FMs) ( 1 , 2 , 3 ) . Overwhelmingly, their architecture is based on the Transformer ( 4 ) with its attention mechanism ( 5 ) . This approach is justified by the impressive body of empirical results in numerous applications ( 6 ) . The Transformer, however, does not come without its own drawbacks. The attention layer enables the dense processing of elements in a sequence at the cost of a computational burden that scales quadratically with the context length. Besides the computational limitations, Transformers and sparse attention models ( 7 ) are inefficient at modeling long-range dependencies, both empirically ( 8 ) and in ad-hoc benchmarks ( 9 ) . State Space Models (SSMs) ( 10 ) are a class of models thoroughly studied in many established scientific areas. Recently, the HiPPO ( 11 ) has paved the way in deep learning for sequence modeling by showing how SSMs can be leveraged for long-range dependencies with a careful initialization of the state parameters. Then, a class of SSM variants has been built on top of HiPPO, starting from the structured SSM (S4) ( 12 ) to, eventually, Mamba ( 13 ) . The latter is the first SSM to match Transformers’ performance without resorting to any form of attention. The main Mamba novelty is the selectivity, where state space parameters are themselves outputs of time-dependent neural networks ( 14 ) . While the idea is supported by compelling experiments, however, the source of its added expressivity is left unexplored with a room for further analytical and architectural exploration. Figure 1 : ( Top Left ) A linear time-invariant SSM with uniform sampling assumes a sequence \{x_{l}\} is sampled from an underlying continuous function which incurs a loss of precision due to the constant time interval \Delta . ( Top Right ) The Mamba computes time-variable intervals \Delta_{l} that are dependent on the sequence elements: the sequence is assumed to be non-uniformly sampled from the underlying function. ( Bottom ) Our SeRpEnt assumes time-dependent intervals \Delta_{l} and, additionally, resamples the sequence through interpolation by aggregating together elements whose information is related. Stemming from a renewed analysis of the selectivity mechanism in Mamba, we propose SeRpEnt , a selective resampling procedure for sequence compression . We show that selectivity learns to discriminate sequence elements based on amount of information in them. Using this analysis, SeRpEnt compresses sequences in an information-aware fashion: an overall reduction of the sequence length improves the global processing, while local elements are aggregated based on their information content. Figure 1 visualizes the main SeRpEnt novelty when compared to previous works. Our main contributions are summarized as follows: • We show that selectivity is rooted in information processing capabilities, in particular the selective time intervals learned by Mamba act as linear approximators of information. • We develop a selective resampling strategy for SSMs and propose our SeRpEnt , a model that compresses sequences in an information-aware fashion and, hence, enables more efficient global processing by aggregating together elements based on their information content. • We showcase SeRpEnt’s performance in the Long Range Arena and other language modeling tasks to advance the development of SSMs as an alternative to Transformers. 2 Related Work Despite progress in the field of machine learning for sequence processing ( 15 ) , capturing long-range dependencies has proven a difficult problem to tackle ( 16 ) . Particularly, this is evident in scenarios characterized by large contextual windows in the domain of Natural Language Processing (NLP) ( 17 ) , where the retrieval of specific information in large textual corpora presents a challenging task ( 18 , 19 ) . In this section, we review recent mainstream approaches for the problem of modeling long-range dependencies. We discuss their properties and rationale, as well as their shortcomings. 2.1 Attention Mechanism in Transformers The attention mechanism [ 5 , 4 ] has become ubiquitous in sequence modeling, thanks to its capability of relating tokens in a sequence. That justifies approaches for long-sequence modeling built upon its cornerstone architecture: the Transformer. However, the Transformer presents a dual problem in the specific context of long-range dependencies: the computational complexity of inferring a token is quadratic in the length of the sequence, and its expressivity is negatively affected when applied to long sequences ( 8 , 20 ) . To overcome the first shortcoming, alternative forms of attention have been proposed ( 21 , 22 , 7 ) , which rely on sparse representations and computations. More recently, [ 23 ] develop a fast implementation which exploits the GPU hardware to minimize data exchange. While sparse attention successfully deals with the computational side of the problem, their second drawback, i.e. the expressivity for modeling long-range sequences, remains a major obstacle in many practical applications ( 24 ) . 2.2 State Space Models State space models are an established family of transformations in traditional statistics. The authors of HiPPO ( 11 ) propose the use of SSMs as an alternative backbone for long sequence modeling by leveraging a theory for parameters initialization. Unlike transformers, this approach unlocks inference with linear complexity. [ 12 ] further suggests an efficient convolutional approximation for SSMs which enables concurrent computations at training time. The initial success of SSMs spurred a research in related areas e.g. audio generation ( 25 ) , reinforcement learning ( 26 ) , and spatiotemporal modeling ( 27 ) . [ 28 ] are the first to explore the use of SSMs for language modeling. More recently, Mamba ( 13 ) proposes the first SSM capable of matching Transformer’s performance in the NLP tasks. Mamba does so by removing the linear time-invariant constraint in previous works and by using a time-dependent parametric selectivity mechanism. This leads to a fully recurrent model that generalizes prior gated