Flash PD-SSM : Memory-Optimized Structured Sparse State-Space Models

Aleksandar Terzić [email protected], Francesco Carzaniga [email protected], Nicolas Menet [email protected], Yannick Biehl [email protected], Michael Hersche [email protected], Thomas Hofmann [email protected], Abbas Rahimi [email protected] · 2026

Read Flash PD-SSM : Memory-Optimized Structured Sparse State-Space Models by Aleksandar Terzić [email protected], Francesco Carzaniga [email protected], Nicolas Menet [email protected], Yannick Biehl [email protected], Michael Hersche [email protected], Thomas Hofmann [email protected], Abbas Rahimi [email protected] (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2605-19150.txt.

Introduction

Aleksandar Terzić and six colleagues propose Flash PD-SSM, a state-space model matching unstructured expressivity at structured efficiency. It uses a trainable set of structured sparse matrices, one discretely chosen per step, giving finite-state automaton expressiveness at scale. On synthetic mechanistic and state-tracking tasks its expressivity holds in practice. On time-series tasks beyond 17,000 steps it sets a best accuracy among state-space methods. In hybrid language models it improves state tracking and language modelling, raising throughput and cutting memory use.

Catalog record

CREATORAleksandar Terzić [email protected], Francesco Carzaniga [email protected], Nicolas Menet [email protected], Yannick Biehl [email protected], Michael Hersche [email protected], Thomas Hofmann [email protected], Abbas Rahimi [email protected]
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE64288
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2605.19150

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Flash PD-SSM : Memory-Optimized Structured Sparse State-Space Models Aleksandar Terzić [email protected], Francesco Carzaniga [email protected], Nicolas Menet [email protected], Yannick Biehl [email protected], Michael Hersche [email protected], Thomas Hofmann [email protected], Abbas Rahimi [email protected] Abstract State-space models (SSMs) face a fundamental trade-off between efficiency and expressivity that is mainly dictated by the structure of the model’s transition matrix. Unstructured transition matrices enable maximal expressivity, as measured by their ability to model finite-state automaton (FSA) transitions, but come at a prohibitively high compute and memory cost. In contrast, most structured transition matrix forms are highly efficient both in runtime and memory consumption, but suffer from limited expressivity. Building on recent work on structured sparse SSMs, we propose Flash PD-SSM , a novel SSM that achieves comparable throughput to widely-used structured SSMs with significantly better expressivity guarantees. Flash PD-SSM maintains a trainable set of structured sparse matrices, a single one of which is discretely selected at each time-step, enabling FSA expressiveness at the level of unstructured matrices while maintaining the efficiency required for training models at scale. First, we validate Flash PD-SSM against a suite of alternative models on synthetic mechanistic and state-tracking tasks, finding that its theoretical expressivity is achieved in practice. Second, on multivariate time-series tasks involving sequences of length over 17 000, we find that Flash PD-SSM defines a new state-of-the-art (SoTA) accuracy among competing SSM methods. Finally, we demonstrate that Flash PD-SSM is an effective drop-in replacement for hybrid LLMs, yielding improvements both in natural language state-tracking and in common language modeling scenarios. The model exhibits increased throughput and decreased memory consumption compared to SSMs widely used in frontier language models. 1 Introduction State-space models (SSMs) [ 21 , 23 , 15 , 50 , 36 , 19 , 11 , 60 , 49 ] offer a scalable alternative to Transformers [ 54 ] , providing distinct advantages such as linear compute scaling relative to sequence length and constant memory consumption during inference. Modern LLMs increasingly employ hybrid architectures that combine the strengths of Transformers and SSMs, often utilizing a higher proportion of SSM layers in order to effectively reap their computational benefits [ 60 , 43 , 12 , 55 , 28 , 58 ] . Figure 1 : Flash PD-SSM is expressive, fast, and memory-efficient. Synthetic state-tracking accuracy on a collection of FSA emulation tasks [ 13 , 57 ] . The runtime (measured relative to PD-SSM [ 53 ] ) and memory consumption of Flash PD-SSM and other popular SSMs are also reported. The maximum sequence length is 2048 and all models have hidden dimension 1024. The circle’s size indicates peak memory consumption during training. Flash PD-SSM is the only SSM in the upper right quadrant, i.e., both highly expressive and fast. Initially, the computational costs associated with training large models have required the use of diagonal transition matrices, which impose significant limitations on model expressivity [ 19 , 33 , 9 , 46 ] . Consequently, the design of SSM transition matrices that better balance the expressivity-efficiency trade-off has become an active area of inquiry, spurring a range of works investigating alternative parametrizations [ 47 , 60 , 17 , 49 , 40 , 57 , 27 ] . Here, we build upon a recent development that utilizes structured sparse transition matrices, namely the PD-SSM model [ 53 ] . The PD-SSM matrices provide optimal state-tracking expressivity guarantees, at the level of unstructured SSMs, while enabling efficient parallel computation through linear-complexity matrix-matrix multiplication of the highly sparse structured transition matrices. However, while the structured sparse matrices utilized by PD-SSM offer linear asymptotic complexity in the parallel evaluation of the sequence of hidden states, the generation of the input-dependent matrices required for this computation incurs a significant overhead. If L is the sequence length and N the SSM hidden state size, generating the sparse transition matrices in PD-SSM incurs an \mathcal{O}(LN^{2}) memory cost. In practice, a parameter-matched Mamba2 [ 11 ] is more than 40\times faster (see Figure 1 ), making PD-SSM unsuitable for large-scale deployment. In this work, we address this bottleneck by introducing Flash PD-SSM , a highly efficient and effective structured sparse SSM. Flash PD-SSM provides state-of-the-art (SoTA) results in various state-tracking and pure language modeling tasks, while maintaining a superior throughput and reduced memory usage to competing SSMs widely used in frontier language models, including both Mamba2 [ 11 ] and DeltaNet [ 48 , 60 ] . Our main contributions are as follows. Novel Selection Mechanism and Hardware-Aware Design. First, we design a novel generation mechanism for structured sparse transition matrices that maintains optimal theoretical expressivity guarantees, yet reduces the memory complexity from \mathcal{O}(LN^{2}) to \mathcal{O}(LN) . We combine this principled algorithmic improvement with our custom hardware-aware Triton/CUDA implementation of the structured linear recurrence of PD-SSM. At a sequence length of 5120 , Flash PD-SSM achieves 18\% faster training than DeltaNet and 4\% faster than Mamba2 (see Figure 5 ). At the same time, it results in a more than 12\% memory reduction with respect to both DeltaNet and Mamba2 (see Figure 5 ). We fully characterize the performance of Flash PD-SSM in Section 4.1 . Synthetic State-Tracking, Mechanistic Capabilities, and Time-Series Modeling. Second, we show that Flash PD-SSM can achieve the theoretical guarantees of structured sparse SSMs in practice, by exhibiting excellent length generalization accuracy ( 96.3\% avg.) on synthetic state-tracking tasks [ 13 , 57 ] , substantially outperforming both diagonal (Mamba2 [ 11 ] with 58.9\% avg. and Mamba3 [ 27 ] with 90.7\% avg.) and DPLR SSMs (Gated DeltaProduct [ 49 ] with 80.7\% avg.). These results confirm the superior state-tracking capabilities of Flash PD-SSM , placing it at the frontier of expressiveness, throughput, and memory usage. Probing a different type of fundamental model capabilities, on mechanistic capability tests (MAD [ 41 ] ), Flash PD-SSM achieves scores comparable to Mamba [ 19 ] ( 69.2\% vs. 69.3\% ). Since high performance on this task has been linked to strong language modeling capabilities, we use these results to inform our choice of hybridization for the language modeling tasks that follow. Furthermore, Flash PD-SSM achieves SoTA results on time-series tasks such as the UEA multivariate time-series classification task [ 2 ] , containing time series data of length over 17 000, and the PPG-Daglia time-series regression [ 42 ] task. Natural Language State-Tracking and Language Modeling. Finally, we show that Flash PD-SSM is effective in hybrid language models through two classes of tasks: natural language state-tracking and general language modeling. We extend the Box [ 26 ] state-tracking dataset to more than twice its original length (up to 26 state transitions) and use it to benchmark frontier hybrid models, Granite4 [ 24 ] and NemotronH [ 6 ] , combining self-attention and Mamba2. Here, we demonstrate that Flash PD-SSM surpasses the out-of-distribution (length generalization) accuracy of Mamba2 by up to 10\% in an equal setting while maintaining comparable throughput. We then consider the pure language modeling capabilities of Flash PD-SSM to evaluate the feasibility of its inclusion in frontier models. Informed by the results on MAD, we train Delta-PD , a hybrid language model consisting of Flash PD-SSM and DeltaNet layers. We perform full pretraining on the FineWeb100B [ 39 ] dataset

Read the complete text