A Theoretical Analysis of Mamba’s Training Dynamics: Filtering Relevant Features for Generalization in State Space Models

Mugunthan Shandirasegaran, Hongkang Li, Songyang Zhang, Meng Wang, Shuai Zhang · 2026

Read A Theoretical Analysis of Mamba’s Training Dynamics: Filtering Relevant Features for Generalization in State Space Models by Mugunthan Shandirasegaran, Hongkang Li, Songyang Zhang, Meng Wang, Shuai Zhang (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2602-12499.txt.

Description

Mamba and other selective state space models have worked well in tests without attention, yet their theory lags behind. Mugunthan Shandirasegaran led this work with four co-authors, looking at a simple but typical Mamba block. It is a single-layer, single-head picking state space model with input-tied gating, then a two-layer MLP trained by gradient descent. Data include key and off-topic token patterns under token-level noise.

Two cases are studied, most-vote and nearness-ordered series, and the model is proven to extend with limit sample cost and settling bounds that improve as signal rises and noise falls. The gating vector lines up with key traits while skipping off-topic ones, stating a choice role similar to focus but through picking loops. Made-up tests support the theory, showing when and why Mamba-style models learn well.

Catalog record

CREATORMugunthan Shandirasegaran, Hongkang Li, Songyang Zhang, Meng Wang, Shuai Zhang
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE210 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2602-12499

Browse the full library · About the Tower of Babel