Time-Varying Deep State Space Models for Sequences with Switching Dynamics
Sanja Karilanova, Subhrakanti Dey, Ayça Özçelikkale · 2026
Read Time-Varying Deep State Space Models for Sequences with Switching Dynamics by Sanja Karilanova, Subhrakanti Dey, Ayça Özçelikkale (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2605-15311.txt.
Introduction
Sanja Karilanova, Subhrakanti Dey, and Ayça Özçelikkale propose time-varying deep state-space models for sequences with switching dynamics. Neurons follow time-varying dynamics supplied by a dictionary of basis functions, each evolving differently over time. Tests on synthetic switching systems and on speech denoising under switching noise show the time-varying model consistently beats time-invariant counterparts at comparable computational cost. The study also maps which dynamic aspects matter most and how freedom should be allocated across model components.
Catalog record
| CREATOR | Sanja Karilanova, Subhrakanti Dey, Ayça Özçelikkale |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 27998 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2605.15311
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Time-Varying Deep State Space Models for Sequences with Switching Dynamics footnoteinfo S. Karilanova acknowledges the support of Center for Interdisciplinary Mathematics (CIM), Uppsala University. A. Özçelikkale acknowledges the support from the Swedish Research Council through grant agreement no. 2024-05194. The computations were enabled by resources provided by the National Academic Infrastructure for Supercomputing in Sweden (NAISS), partially funded by the Swedish Research Council through grant agreement no. 2022-06725. Sanja Karilanova, Subhrakanti Dey, Ayça Özçelikkale Abstract The identification and modeling of time-varying systems is a fundamental challenge in signal processing and system identification. To address this challenge, we propose a class of time-varying state-space model (SSM) based neural networks in which the neurons’ states are governed by time-varying dynamics. The proposed model provides the learnable time-varying dynamics through a dictionary of basis functions, where each basis function evolves differently over time. We evaluate the proposed approach on both synthetic data from switching systems and a speech denoising task where real audio is corrupted with switching dynamics noise. The results show that the proposed time-varying model consistently outperforms its time-invariant counterparts while maintaining comparable computational complexity. Our investigations also reveal which aspects of the time-varying dynamics of the data most need to be captured by the proposed time-invariant models, how the additional freedom provided by time-varying basis functions should be allocated across model components, and to what extent larger models can compensate for time-invariant limitations. keywords System identification, Time-varying, Deep learning, Switching State-Space Models, Deep State-Space Models 1 Introduction Figure 1 : Left: Example network architecture with an input layer with a single channel, two hidden layers with SSM neurons and an output layer of two channels. Middle: An SSM neuron with time-varying dynamics. Right: Illustration of the proposed basis function expansion for a single time-varying element of the SSM state transition matrix. Many real-world processes are inherently time-varying, meaning that their underlying dynamics evolve over time. This time-varying behavior appears in various domains, such as financial markets, weather forecasting, energy consumption, and neural activity analysis. A central challenge in system identification is therefore to build models that are flexible enough to capture such evolving dynamics 7 ; 9 . State Space Models (SSMs) provide a principled and interpretable framework for system identification. In their classical form with linear state transitions, often referred to as Linear Dynamical Systems (LDS) 8 , they assume fixed time-invariant transition dynamics. A common extension of LDSs for handling time-varying data is the Switching Linear Dynamical System (SLDS) 18 , where the dynamics transition between multiple LDS modes. While effective in capturing regime changes, learning SLDSs remains challenging, as it requires estimating the number of modes, the model order of each mode, and the corresponding parameters 14 . Another extension of LDS, distinct from SLDS, is deep-SSMs. These models stack LDS blocks into layers with nonlinear transformations and have recently achieved performance comparable to transformers on long-sequence modeling tasks 6 16 , as well as have been applied to nonlinear system identification 1 . Extensions to Deep-SSMs include input-dependent mechanism 4 and input timescale invariant formulation 3 . Traditionally, system identification research has long explored time-varying and non-stationary behavior through methods such as adaptive filtering, basis function expansions 19 ; 2 ; 13 ; 23 , and change point detection 20 . More recently, deep learning has also become a powerful tool for system identification 10 , including approaches based on time-varying neural networks, such as dynamic neural networks 7 and non-stationary transformer architectures 9 . Our work brings the time-varying basis function expansion methodology into contemporary deep SSM architectures and illustrates the performance for system identification tasks. Unlike explicit modeling through SLDSs, our framework requires no explicit specification of switching modes and instead models smooth, continuously time-varying dynamics, enabling efficient training through standard backpropagation through time (BPTT). In contrast to modeling time variation in deep SSMs through input-dependence, our framework explicitly models time-dependent dynamics independent of the input. Figure 1 provides a visualization of our proposed framework. The key contributions of this paper are as follows: • We propose a novel time-varying deep SSM framework using learnable basis functions to parameterize state transition, input, and output matrices over time. • We provide experimental validation on both synthetic and real-world data on audio denoising scenarios with non-stationary noise dynamics. • We explore the trade-offs in the proposed model architecture, including effects of different parts of SSM dynamics and basis function allocation, and compare it with its time-invariant counterpart. Overall, our results show that the proposed time-varying model consistently outperforms its time-invariant counterparts under switching dynamics data while retaining comparable computational complexity. 2 Methods 2.1 Preliminaries 2.1.1 Time-invariant SSM model: A time-invariant linear discrete-time SSM is given by 11 \displaystyle\bm{x}[t] \displaystyle=\bm{A}\bm{x}[t-1]+\bm{B}\bm{u}[t-1] (1a) \displaystyle\bm{y}[t] \displaystyle=\bm{C}\bm{x}[t], (1b) with possibly learnable matrix parameters \bm{A}\in\mathbb{R}^{n\times n} , \bm{B}\in\mathbb{R}^{n\times n_{in}},\bm{C}\in\mathbb{R}^{n_{out}\times n},\bm{D}\in\mathbb{R}^{n_{out}\times n_{in}} , input \bm{u}[t]\in\mathbb{R}^{n_{in}\times 1} , state variable \bm{x}[t]\in\mathbb{R}^{n\times 1} , and output \bm{y}[t]\in\mathbb{R}^{n_{out}\times 1} . We refer to the SSM in ( 1 ) as a neuron in the context of a SSM based neural network. Multiple neurons with such SSM dynamics may be used to form a SSM layer, see Figure 1 . Layers with SSM dynamics are combined with possibly non-linear activation function layers and mixing layers to form a deep SSM network 6 16 . 2.1.2 Basis function expansion: Basis function expansion is a way of representing a function, f(t) , as a linear combination of simpler functions, \phi^{(k)}(\cdot) , called basis functions, as follows: \displaystyle f(t)=\sum^{K}_{k=1}\alpha^{(k)}\times\phi^{(k)}(t), (2) where \alpha_{k}\in\mathbb{R} are the coefficients. Basis functions may assume various forms, such as with Fourier series, Gaussian functions, polynomial functions, rational orthogonal functions, and radial basis functions. For example, we can have \phi^{(k)}=\mathcal{N}(t|\mu_{k},\sigma_{k}^{2}) for a dictionary of Gaussian functions with \mathcal{N}(\cdot) representing the shape of a Gaussian function and \phi^{(k)}=\sin(w_{k}t+\phi_{k}) for a dictionary of sine functions, where \mu_{k},\sigma_{k},w_{k},\phi_{k} are fixed pre-defined scalars. 2.2 Proposed time-varying SSM model We propose the following time-varying SSM model \displaystyle\bm{x}[t] \displaystyle=\bm{A}[t]\bm{x}[t-1]+\bm{B}[t]\bm{u}[t-1] (3a) \displaystyle\bm{y}[t] \displaystyle=\bm{C}[t]\bm{x}[t], (3b) where \bm{A}[t],\bm{B}[t],\bm{C}[t] have the same dimensions as the time-invariant model in ( 1 ). However, each element of these matrices is now a linear combination of basis functions. In particular, for \bm{A}[t] , we have \displaystyle[\bm{A}[t]]_{ij}=a_{i,j}[t]=\sum^{K_{A}}_{k=1}a_{i,j}^{(k)}\times\phi_{A,i,j}^{(k)}[t] (4) where [\bm{A}[t]]_{ij}=a_{i,j}[t] denotes the i th row j th column element of the matrix \bm{A}[t] . An illustration of the expansion in ( 4 ) is provided in Figure 1 . We note that the number of basis functions K_{A} is independent of any other dimensions in the model. Here we present the general case where the basis function \phi_{A,i,j}^{(k)} may differ across SSM matrices (A,B,C) , indices (i,j) , and expansion terms (k) . In practice, this heterogeneity can be reduced by using a smaller shared dictionary of basis functions. Similar to \bm{A}[t] , we have the following: \displaystyle[\bm{B}[t]]_{ij} \displaystyle=b_{i,j}[t]=\sum^{K_{B}}_{k=1}b_{i,j}^{(k)}\times\phi_{B,i,j}^{(k)}[t] (5) \displaystyle[\bm{C}[t]]_{ij} \displaystyle=c_{i,j}[t]=\sum^{K_{C}}_{k=1}c_{i,j}^{(k)}\times\phi_{C,i,j}^{(k)}[t]. (6) We note that K_{A} , K_{B} , K_{C} can be chosen independently of each other. Hence, a subset of \bm{A}[t],\bm{B}[t],\bm{C}[t] can remain time-invariant as in the time-invariant model ( 1 ) independently of the others. 2.3 Stability A central question in deep SSM development is to ensure stability of the SSM dynamics. The stability of the dynamics in ( 3 ) is controlled by \bm{A}[t] . For simplicity