Let SSMs be ConvNets: State-space Modeling with Optimal Tensor Contractions

Yan Ru Pei · 2025

Read Let SSMs be ConvNets: State-space Modeling with Optimal Tensor Contractions by Yan Ru Pei (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2501-13230.txt.

Introduction

Yan Ru Pei rewrites state-space sequence models as convolutional networks to train them faster in a 2025 study. State-space blocks process long sequences efficiently at run time, but their training is usually coded in a fixed, depthwise style. Viewing each block's operations as tensor contractions opens a choice: the order of contraction changes the cost, and the best order can be found systematically for every block.

The resulting network family, called Centaurus, gains design freedom beyond the standard configuration. Faster training of long-sequence models matters for audio, genomics, and any data where context stretches far.

Catalog record

CREATORYan Ru Pei
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE78 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2501-13230

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Let SSMs be ConvNets: State-space Modeling with Optimal Tensor Contractions Yan Ru Pei Abstract We introduce Centaurus, a class of networks composed of generalized state-space model (SSM) blocks, where the SSM operations can be treated as tensor contractions during training. The optimal order of tensor contractions can then be systematically determined for every SSM block to maximize training efficiency. This allows more flexibility in designing SSM blocks beyond the depthwise-separable configuration commonly implemented. The new design choices will take inspiration from classical convolutional blocks including group convolutions, full convolutions, and bottleneck blocks. We architect the Centaurus network with a mixture of these blocks, to balance between network size and performance, as well as memory and computational efficiency during both training and inference. We show that this heterogeneous network design outperforms its homogeneous counterparts in raw audio processing tasks including keyword spotting, speech denoising, and automatic speech recognition (ASR). For ASR, Centaurus is the first network with competitive performance that can be made fully state-space based, without using any nonlinear recurrence (LSTMs), explicit convolutions (CNNs), or (surrogate) attention mechanism. 1 Introduction Sequence or temporal modeling encompasses a wide range of tasks from audio processing to language modeling. Traditionally, there have been many (related) statistical methods employed ( Box et al., 2015 ) . In the age of deep learning, neural networks have been predominantly used ( LeCun et al., 2015 ) , including recurrent neural networks (RNNs), convolutional neural networks (CNNs), transformers ( Vaswani, 2017 ) , and neural ODEs ( Chen et al., 2018 ) . In many cases, the model will inevitably suffer from one of two drawbacks: 1) cannot be efficiently trained (or fitted) in parallel due to the sequential nature of the model, 2) cannot be efficiently configured for online inference due to its large memory and computational requirement. To address this, deep state-space models (SSMs) were adapted for sequence modeling, and have shown incredible potential across a wide range of tasks ( Gu et al., 2021 ; Goel et al., 2022 ; Gu & Dao, 2023 ) . Due to the linearity of the SSM layers, they can not only be configured for efficient online inference with small memory and computational resources, but also configured for efficient training using parallel hardware with unrolling strategies ( Gu et al., 2022 ; Smith et al., 2022 ; Dao & Gu, 2024 ; Heinsen, 2023 ) . Currently, most deep SSM networks (along with most neural networks in general) follow the architectural recipe of transformers, where they are composed of uniform “SSM blocks” throughout the network, containing little to no variations in the shapes of the intermediate features or weights. This simplifies the designs of deep SSM networks, but may sacrifice performance and efficiency in practice. To explore the opposite direction, we go “back to the future” to classical CNN designs instead, where a much more heterogeneous design principle is followed. More specifically, as we go deeper in the network, we will gradually downsample the temporal dimension, increase the channel dimension, and increase connective sparsity ( Tan & Le, 2021 ) , to balance between size and efficiency ( Pei & Coenen, 2024 ) . In order to allow for the possibility of such a heterogeneous design, we make two novel contributions in this work. First, we express SSM blocks as tensor networks (or with einsum expressions), where we can easily observe the connective structure of existing SSM blocks as being mostly depthwise-separable. This motivates us to design new SSM blocks using this tensor network formalism, giving us connective structures such as full and bottleneck SSM blocks, as inspired by classical CNN blocks. Then, we optimize the contraction order of each SSM block dynamically based on the type of the SSM block, the shape of the input features, and the shapes of the SSM system matrices. This enables significant speedups during training for all SSM blocks (both old and new ones). We name this class of efficient CNN-like networks Centaurus 1 1 1 As a quick explanation of the name, we are exploring Tensorized Temporal Networks, or TenTeNs, from which we get a hundred, or CENTaurus. Admittedly, Centaurus is not actually composed of the word root “cent”, so this wordplay is not too sensible. However, Centaurus is a constellation, usually visualized as stars being linked; this is visually similar to what a tensor network looks like, so some naming sense is recovered. . The model source code is available as supplementary material on the OpenReview page . 2 Related Work 2.1 Deep State-space Modeling The seminal work proposing a memory encoding using orthogonal Legendre polynomials in a recurrent state-space model is the Legendre Memory Unit (LMU) ( Voelker et al., 2019 ) , where Legendre polynomials (a special case of Jacobi polynomials) are used. The HiPPO formalism ( Gu et al., 2020 ) then generalized this to other common orthogonal functions. Later, this sparked a cornucopia of works interfacing with deep state-space models including S4 ( Gu et al., 2021 ) , H3 ( Fu et al., 2022 ) , and Mamba ( Gu & Dao, 2023 ) , achieving impressive results on a wide range of tasks from audio generation Goel et al. (2022) to language modeling. Besides a few recent exceptions ( Smith et al., 2022 ; Dao & Gu, 2024 ) , These networks mostly use an underlying depthwise structure, which may limit the network capacity, albeit reducing the compute requirement of the network. An important focal point of this work is to generalize such SSM models to enable more connective flexibility, such that design choices of classical convolutional blocks can be carried over to the Centaurus model. 2.2 Classical convolutional blocks Here, we look at the variants besides the standard full convolutional layer (where every pair of input and output channels is connected). First, the simplest variant is the depthwise convolutional layer, made popular by the MobileNetV1 architecture ( Howard et al., 2017 ) . This variant connects the input and output channels one-to-one, which drastically reduces the connectivity of the architecture, hence also its parameters and computational complexity. This architecture is prevalent for both computer vision and speech domains ( Kriman et al., 2020 ; Hannun et al., 2019 ) . Next, a variant with moderate connectivity is the grouped convolutional layer, appearing as early as the AlexNet work ( Krizhevsky et al., 2012 ) . In this structure, the input/output channels are divided into groups, and each input-channel group is associated with an output-channel group one-to-one. The input and output channels are then only intra-connected within each group, but there are no inter-connections between groups. Finally, there is a class of convolutional blocks known as bottlenecks, typically containing a sequence of three convolutional layers. This structure is used prominently in the ResNet model ( He et al., 2016 ) , and also in MobileNetV2 ( Sandler et al., 2018 ) onwards. Typically, the order of convolutional layers can be pointwise-depthwise-pointwise, resulting in an output feature of the same tensor shape as the input feature. 2.3 Tensor Networks Tensor networks were originally developed as a tool for approximating the wavefunctions of many-body quantum systems ( Orús, 2019 ) . It can be considered as

Read the complete text