A Theoretical Analysis of Mamba’s Training Dynamics: Filtering Relevant Features for Generalization in State Space Models
Mugunthan Shandirasegaran, Hongkang Li, Songyang Zhang, Meng Wang, Shuai Zhang · 2026
Read A Theoretical Analysis of Mamba’s Training Dynamics: Filtering Relevant Features for Generalization in State Space Models by Mugunthan Shandirasegaran, Hongkang Li, Songyang Zhang, Meng Wang, Shuai Zhang (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2602-12499.txt.
Description
Mamba and other selective state space models have worked well in tests without attention, yet their theory lags behind. Mugunthan Shandirasegaran led this work with four co-authors, looking at a simple but typical Mamba block. It is a single-layer, single-head picking state space model with input-tied gating, then a two-layer MLP trained by gradient descent. Data include key and off-topic token patterns under token-level noise.
Two cases are studied, most-vote and nearness-ordered series, and the model is proven to extend with limit sample cost and settling bounds that improve as signal rises and noise falls. The gating vector lines up with key traits while skipping off-topic ones, stating a choice role similar to focus but through picking loops. Made-up tests support the theory, showing when and why Mamba-style models learn well.
Catalog record
| CREATOR | Mugunthan Shandirasegaran, Hongkang Li, Songyang Zhang, Meng Wang, Shuai Zhang |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 210 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2602-12499