On the Generalization Properties of Selective State-Space Models for Filtering Tasks for Unknown Systems
Alex Tang, M. Emrullah Ildiz, Batin Kurt, Samet Oymak, Necmiye Ozay · 2026
Read On the Generalization Properties of Selective State-Space Models for Filtering Tasks for Unknown Systems by Alex Tang, M. Emrullah Ildiz, Batin Kurt, Samet Oymak, Necmiye Ozay (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2604-23818.txt.
Introduction
Tang and colleagues ask whether selective state-space models such as Mamba can filter unknown systems as well as transformers do. Transformers do this through in-context learning. The team trains the model on trajectories drawn from several systems. At run time they feed it outputs of one unseen system from that group. The model must predict the next output online. Under stated assumptions they derive generalisation bounds explaining the models' success. They confirm performance numerically and weigh pros and cons against transformers.
Catalog record
| CREATOR | Alex Tang, M. Emrullah Ildiz, Batin Kurt, Samet Oymak, Necmiye Ozay |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 43809 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2604.23818
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
On the Generalization Properties of Selective State-Space Models for Filtering Tasks for Unknown Systems Alex Tang, M. Emrullah Ildiz, Batin Kurt, Samet Oymak, Necmiye Ozay Abstract Selective State-Space Models (SSMs) such as Mamba have emerged as an alternative architecture to self-attention based transformers in sequence modeling tasks. Recent works have demonstrated the use of transformers in some filtering and output prediction tasks via in-context learning. In this paper, we analyze whether structured SSMs can work equally well for filtering of unknown systems. In particular, we train the SSM on trajectory samples from a set of systems. At run-time, the SSM is given the outputs of an unknown system from the same set and is expected to predict the next output online. Theoretically, under appropriate assumptions, we derive generalization bounds as to why SSMs succeed in such tasks. Empirically, we demonstrate the performance via several numerical examples. We also discuss the advantages and disadvantages of SSMs versus transformers for this task. I INTRODUCTION The problem of predicting future trajectories from noisy observations is central to many control applications. When an accurate system model is available a priori and the noise satisfies standard assumptions, e.g., Gaussianity, a broad class of estimation methods is available. These include the Kalman filter (KF) for linear systems [ 12 ] , the extended Kalman filter (EKF) for nonlinear systems [ 3 ] , and sampling-based methods such as particle filters for nonlinear and non-Gaussian settings [ 6 ] . Since classical filtering methods require an explicit model of the system dynamics, their direct application is limited when such a model is unavailable or difficult to obtain. In recent years, learned sequence models have become increasingly prominent for prediction and inference tasks [ 8 , 1 ] , especially with the widespread adoption of transformer architectures [ 17 ] . A key advantage of these models is their ability to perform in-context learning [ 5 ] , namely, to infer useful task-specific structure directly from the input at run-time. This makes sequence models particularly useful for online prediction problems in which the underlying system is unknown and must be inferred implicitly from past observations. Recent work has shown that transformer-based architectures can indeed be trained in this manner for filtering and output prediction of unknown systems [ 7 ] . While transformer models have been gaining traction in filtering applications, they are neither the only nor the undisputed best solution for this problem. Interest has grown for an alternative architecture, the State Space Model, which we shall refer to as the SSM. The quadratic-time self-attention modules of the transformer are replaced in SSMs by linear state-space models which have linear time complexity at prediction time [ 2 ] . A major improvement to SSMs came with the advent of the Mamba Selective SSM architecture [ 9 ] , which uses a linear time-varying (LTV) state-space model instead of earlier time-invariant architectures. While technically a subset of general State-Space Models, further instances of the initialism “SSM” will refer exclusively to the Selective SSM, as it is the architecture of interest in this paper. In addition to run time advantages, state-space models have other potential strengths over transformers. Transformer models are trained and operate with a specific, finite context window which dictates the longest sequence length that it is able to consider. While this context window can be, in theory, made arbitrarily large, the onerous computational and memory requirements of transformers mean that in practice, it is possible for the context window to be shorter than the desired prediction horizon. There are workarounds to this, such as the sliding window [ 4 ] , but these methods are even more hardware-demanding. More importantly in the context of control systems, theoretical guarantees of transformer performance [ 7 , 13 ] do not immediately extend to the sliding window implementation. This makes them problematic in applications where prediction accuracy or safety cannot be exposed to risk. SSMs, however, do not suffer from this drawback. This is due to the fact that they have no context window, since they simply apply a recurrent map to the input at each time step. Therefore, no architectural limitations prevent them from working for arbitrarily long prediction horizons, for which any theoretical guarantees on SSM performance are automatically valid. In practice, SSMs have already been experimentally applied to filtering problems. Empirical results [ 14 ] show that the performance of SSMs in this setting matches, or sometimes even exceeds, the performance of transformer models in a variety of systems exhibiting complex phenomena such as chaotic dynamics. This, combined with prior experimental work on transformer filters, which showed that they already can approach the performance of Kalman filters [ 7 ] indicates the potential of SSMs in this domain. In this paper, we focus on generalization performance guarantees for SSMs. While such analyses for transformers have been published in the literature [ 7 ] , no such bounds currently exist for SSMs. This paper introduces the — to our knowledge — first generalization guarantees for SSMs specific to filtering applications. In Section III , we will establish and prove such a bound, and in Section IV , we will present a number of experimental results which showcase the generalization ability of SSMs. Specific Notation: We will use \mathbb{R}_{+} to mean the non-negative real numbers. \mathbb{R}_{++} correspondingly denotes \mathbb{R}_{+}\setminus\{0\} . The symbol \otimes will denote the Kronecker product, while \times denotes multiplication. The d -dimensional identity matrix will be referred to as \mathbf{I}_{d} , but we will drop the subscript in situations where the dimension is clear from context. Finally, \mathrm{softplus}(x)\triangleq\ln(1+e^{x}) . II PRELIMINARIES AND PROBLEM SETUP II-A The SSM Architecture In this section, we summarize the SSM architecture in [ 10 , 9 ] . The selective SSM with n_{i} input and n_{o} output channels is derived from a continuous-time LTV state-space model \displaystyle\dot{\mathbf{h}}(\bar{t}) \displaystyle=\mathbf{A}{\mathbf{h}}(\bar{t})+\mathbf{B}(\bar{t})\mathbf{y}(\bar{t}) (1) \displaystyle\hat{\mathbf{y}}(\bar{t}) \displaystyle=\mathbf{C}(\bar{t})\mathbf{h}(\bar{t}), where \mathbf{h}(\bar{t})\in\mathbb{R}^{l} is the hidden state of the SSM, \mathbf{y}(\bar{t})\in\mathbb{R}^{n_{i}} is the input, and \hat{\mathbf{y}}(\bar{t})\in\mathbb{R}^{n_{o}} is the output. The diagonal matrix \mathbf{A}\in\mathbb{R}^{n_{o}l\times n_{o}l} and the weight matrices \mathbf{B}(\bar{t})\triangleq\mathbf{I}_{n_{o}}\otimes\mathbf{W}_{\mathbf{B}}\mathbf{y}(\bar{t}) and \mathbf{C}(\bar{t})\triangleq\mathbf{I}_{n_{o}}\otimes\mathbf{y}(\bar{t})^{\intercal}\mathbf{W}_{\mathbf{C}}^{\intercal} are learnable parameters. This continuous-time model is discretized with a non-uniform sampling time defined by \Delta_{\bar{t}}\triangleq\mathrm{softplus}(p+\mathbf{q}^{\intercal}\mathbf{y}(\bar{t})) , again with learnable parameters p and \mathbf{q} . Assuming a zero-order hold, this gives a time-varying discrete-time linear system. 1 1 1 The time \bar{t} of the continuous-time system ( 1 ) is just a fictitious construct that does not correspond to real time, whereas time-index t in the discrete-time model will be treated as the actual time from now on. When taking \mathbf{h}(0)=0 , the unrolled form of this discrete-time dynamics is written as [ 11 ] : \hat{\mathbf{y}}_{t}=\left(\mathbf{I}\otimes\mathbf{y}_{t-1}^{\intercal}\mathbf{W}_{\mathbf{C}}^{\intercal}\right)\sum_{i=0}^{t-1}\left[\mathbf{A}_{\Delta}^{t-1-i}\Delta_{i}\left(\mathbf{I}\otimes\mathbf{W}_{\mathbf{B}}\mathbf{y}_{i}\right)\mathbf{y}_{i}\right], (2) where \mathbf{A}_{\Delta}^{i}=\exp\left[(\Delta_{t-1}+\cdots+\Delta_{t-i})\mathbf{A}\right] for all i>0 and \mathbf{A}_{\Delta}^{i}=\mathbf{I} if i=0 . At each time t , then, the SSM maps the input sequence \mathcal{Y}_{t-1}\triangleq\{\mathbf{y}_{i}\}_{i=0}^{t-1} to the output \hat{\mathbf{y}}_{t} . As shorthand for ( 2 ), we write this relation as \hat{\mathbf{y}}_{t}\triangleq\widehat{\mathrm{SSM}}(\mathcal{Y}_{t-1}) . Finally, we denote by \mathcal{A} the set of all possible SSMs with this architecture, with parameters defined by the parameter set \Theta\triangleq\{\mathbf{A},\mathbf{W_{B}},\mathbf{W_{C}},p,\mathbf{q}\} . All elements of \Theta are assumed to be bounded in the 2 -norm such that \|\cdot\|_{2}<\infty . II-B Filtering Problem Let us specify the filtering problem of