Uncovering the Spectral Bias in Diagonal State Space Models

Ruben Solozabal, Velibor Bojkovic, Hilal AlQuabeh, Kentaro Inui, Martin Takáč · 2025

Read Uncovering the Spectral Bias in Diagonal State Space Models by Ruben Solozabal, Velibor Bojkovic, Hilal AlQuabeh, Kentaro Inui, Martin Takáč (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2508-20441.txt.

Introduction

Ruben Solozabal and four co-authors study hidden biases in state space models in a paper from two thousand twenty-five. These sequence models are usually seeded by a framework built on polynomial approximations. Simpler diagonal versions run faster, but how their starting values bias learning was unclear.

Looking through a frequency lens, the team shows how pole placement at the start steers what the model learns. They propose a new diagonal seeding scheme defined in the Fourier domain. The insight lets them scale the models up with confidence. On a long-range benchmark, the approach reached state-of-the-art results, training from scratch even on very long sequences. The authors argue initialization deserves the same care as architecture.

Catalog record

CREATORRuben Solozabal, Velibor Bojkovic, Hilal AlQuabeh, Kentaro Inui, Martin Takáč
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE55 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2508-20441

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Uncovering the Spectral Bias in Diagonal State Space Models Ruben Solozabal, Velibor Bojkovic, Hilal AlQuabeh, Kentaro Inui, Martin Takáč Abstract Current methods for initializing state space models (SSMs) parameters mainly rely on the HiPPO framework , which is based on an online approximation of orthogonal polynomials. Recently, diagonal alternatives have shown to reach a similar level of performance while being significantly more efficient due to the simplification in the kernel computation. However, the HiPPO framework does not explicitly study the role of its diagonal variants. In this paper, we take a further step to investigate the role of diagonal SSM initialization schemes from the frequency perspective. Our work seeks to systematically understand how to parameterize these models and uncover the learning biases inherent in such diagonal state-space models. Based on our observations, we propose a diagonal initialization on the discrete Fourier domain S4D-DFouT . The insights in the role of pole placing in the initialization enable us to further scale them and achieve state-of-the-art results on the Long Range Arena benchmark, allowing us to train from scratch on very large datasets as PathX-256. 1 Introduction Figure 1 : The kernels learned by S4D on sCIFAR present a —“local attention” profile—. When unrolled, these kernels align with positions corresponding to the vicinities of the pixel being attended. State space models (SSMs) have recently emerged as a principled and scalable means of modeling long sequences across diverse domains—including image processing DBLP:conf/icml/ZhuL0W0W24 ; DBLP:journals/corr/abs-2405-19450 ; DBLP:conf/nips/NguyenGGDSDBR22 , time-series forecasting DBLP:conf/iclr/ZhangSPDGR23 , and natural language understanding DBLP:conf/iclr/FuDSTRR23 ; DBLP:journals/corr/abs-2312-00752 ; DBLP:conf/icml/DaoG24 . At their core, SSMs perform a sequence-to-sequence mapping via a long-range convolution kernel that is parameterized under a continuous-time linear dynamical system. This formulation captures dependencies at multiple timescales and admits strong stability guarantees. Crucially, the “HiPPO matrix” DBLP:conf/nips/GuDERR20 provides a principled method to structure state transition matrices that performs online compression of the input stream by projecting onto an orthogonal polynomial basis. However, naively propagating this dense state through a sequence of length L incurs in \mathcal{O}(N^{2}L) cost —where N is the state dimension— making direct implementation impractical for very long sequences. To overcome this bottleneck, the Structured State Space Sequence (S4) DBLP:conf/iclr/GuGR22 model introduces a normal plus low-rank decomposition of the transition matrix, which reduces the computational complexity while maintaining the expressive capacity DBLP:conf/iclr/GuJTRR23 . Building on the foundations of S4, recent diagonal variants DBLP:conf/nips/0001GB22 ; DBLP:conf/nips/GuG0R22 ; DBLP:conf/icml/OrvietoSGFGPD23 ; DBLP:conf/iclr/SmithWL23 , have shown that restricting the state matrix to be diagonal still preserves the performance. By introducing complex-valued initialization schemes DBLP:conf/nips/GuG0R22 , these diagonal SSMs achieve performance comparable to the original S4 architecture while being more computationally efficient, as the computation of the kernel simplifies to a Vandermonde matrix multiplication. Nevertheless, while the S4 framework has a mathematical interpretation for addressing long-range dependencies, the efficacy of its diagonal variants remains theoretically unexplained. In this work, we take a step further by investigating the initialization of diagonal SSMs from a frequency perspective DBLP:conf/icml/ParnichkunMMSHL24 . We aim to uncover how the initialization of the state matrix, particularly the distribution of the imaginary part of its eigenvalues, directly influences the system’s ability to capture long-range dependencies. Our analysis reveals that current schemes highly depend on the input sequence exhibiting a well-defined inherent timescale that aligns with the model’s discretization step \Delta DBLP:conf/iclr/GuJTRR23 . Since such alignment is unknown a priori, existing models often compensate by spreading the poles across a wide frequency range, resulting in over-parameterization DBLP:conf/nips/GwakMKP24 , non-uniform spectral sensitivity, and aliasing artifacts. Building on this analysis, we introduce S4D-DFouT, an initialization in the discrete domain that explicitly ensures a uniform coverage of the frequency spectrum independently of the selection of the discretization step \Delta . By placing poles directly in the discrete domain, we can reduce the sensitivity these models present to the frequency response without compromising its ability to capture both short- and long-range effects. To summarize, the key contributions of this paper are threefold: • Frequency-aware analysis of Diagonal SSMs: The theoretical understanding of the roles played by the initialization schemes on diagonal SSMs is still lacking. Therefore, we systematically describe diagonal SSMs from a frequency perspective. We analyze how different initialization schemes affect the model’s ability to represent temporal dependencies (Section 3 ). • Proposal of S4D-DFouT initialization: Our observations indicate that current models are highly sensitive to the discretization. We propose a novel approach, S4D-DFouT, directly in the discrete domain that explicitly designs the pole distribution to optimize spectral coverage. This strategy eliminates the existing coupling between the decay and frequency selection, resulting in a more robust and easy-to-scale initialization. (Section 4 ). • SSMs initialization synchronization enhances model efficiency: Finally, we propose a synchronized initialization across all the SSMs in a given layer. By coordinating their pole placements, the entire layer can be jointly initialize. Our experiments demonstrate that such frequency-aware synchronization enables diagonal SSMs to scale more effectively and outperform previous methods on Long Range Arena (LRA) DBLP:conf/iclr/Tay0ASBPRYRM21 enabling to train from scratch on PathX-256 (Section 5 ). 2 Related literature Initial approaches to sequence modeling—such as recurrent neural networks (RNNs) and long short-term memory (LSTM) hochreiter1997long models—achieved notable success but encountered significant limitations when modeling long-range dependencies, primarily due to vanishing and exploding gradients DBLP:conf/icml/PascanuMB13 ; DBLP:conf/nips/MikhaeilMD22 . These challenges motivated the search for efficient alternative architectures. Legendre Memory Units (LMUs) DBLP:conf/nips/VoelkerKE19 then introduced a non-trainable, a continuous-time SSM parameterized using Legendre polynomial bases to encode long-term memory. Building on this, the HiPPO framework provided a principled method to construct state matrices with trainable state matrices, leading to a successful family of long-range convolution models DBLP:conf/iclr/GuGR22 ; DBLP:conf/iclr/SmithWL23 ; DBLP:conf/icml/PoliMNFDBBER23 ; DBLP:conf/iclr/HasaniLWCAR23 . Diagonal variants of SSMs have recently emerged as a simplification of the DPLR-based S4 architecture DBLP:conf/nips/0001GB22 ; DBLP:conf/nips/GuG0R22 ; DBLP:conf/icml/OrvietoSGFGPD23 . The Diagonal State Spaces (DSS) DBLP:conf/nips/0001GB22 method empirically demonstrated that simply removing the low-rank term from the HiPPO DPLR representation to form a diagonal HiPPO matrix preserves performance. Building on this insight, the S4D framework DBLP:conf/nips/GuG0R22 systematically explores diagonal initialization. Nevertheless, while the S4 framework has a mathematical interpretation for addressing long-range dependencies, the efficacy of its diagonal variants remains theoretically unexplained. Most recently, DBLP:journals/corr/abs-2411-19455 provided a data-centric perspective on SSM initialization. Their work emphasizes the importance of aligning model initialization with the autocorrelation structure of the input data. We push beyond this data-driven criterion by investigating the initialization of diagonal SSMs from a frequency perspective. We observe that current initialization schemes—whether based on HiPPO inverse-frequency laws, or linear grids—drive the model to learn convolutional kernels whose resonance aligns with the principal frequential components of the data. As illustrated in Fig. 1 , the kernels learned by SSMs act as localized, image-sized filters, effectively aggregating spatially local features rather than capturing truly global, long-range interactions. Previous works DBLP:conf/iclr/MaZKHGNMZ23 have leveraged exactly this effect on Transformers DBLP:conf/nips/VaswaniSPUJGKP17 ; DBLP:conf/nips/LiJXZCWY19 to perform well on LRA. By combining attention with an exponential moving average, it injects a built-in position-aware encoding mechanism. Our findings show that current initialization schemes rely heavily on coincidental alignment between the dominant timescales in the input and the poles in the system. Since such alignment is rarely known a priori, existing models often compensate by spreading poles across

Read the complete text