Trained Mamba Emulates Online Gradient Descent in In-Context Linear Regression
Jiarui Jiang, Wei Huang, Miao Zhang, Taiji Suzuki, Liqiang Nie · 2025
Read Trained Mamba Emulates Online Gradient Descent in In-Context Linear Regression by Jiarui Jiang, Wei Huang, Miao Zhang, Taiji Suzuki, Liqiang Nie (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2509-23779.txt.
Introduction
Mamba, a state-space sequence model, rivals transformers at in-context learning, but the theory of how it learns in context lagged behind. Jiarui Jiang and 4 co-authors study Mamba's training dynamics on in-context linear regression, a standard test bed. They develop tools for the non-convex training problem shaped by Mamba's structure. The results give an exponential convergence rate to the in-context solution and a loss bound comparable to the transformer's.
The mechanism they uncover differs from the transformer's usual story. A trained Mamba turns out to emulate a variant of online gradient descent on the hidden function. Simulations confirm the theory.
Catalog record
| CREATOR | Jiarui Jiang, Wei Huang, Miao Zhang, Taiji Suzuki, Liqiang Nie |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 183 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2509-23779
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Trained Mamba Emulates Online Gradient Descent in In-Context Linear Regression Jiarui Jiang, Wei Huang, Miao Zhang, Taiji Suzuki, Liqiang Nie Abstract State-space models (SSMs), particularly Mamba, emerge as an efficient Transformer alternative with linear complexity for long-sequence modeling. Recent empirical works demonstrate Mamba’s in-context learning (ICL) capabilities competitive with Transformers, a critical capacity for large foundation models. However, theoretical understanding of Mamba’s ICL remains limited, restricting deeper insights into its underlying mechanisms. Even fundamental tasks such as linear regression ICL, widely studied as a standard theoretical benchmark for Transformers, have not been thoroughly analyzed in the context of Mamba. To address this gap, we study the training dynamics of Mamba on the linear regression ICL task. By developing novel techniques tackling non-convex optimization with gradient descent related to Mamba’s structure, we establish an exponential convergence rate to ICL solution, and derive a loss bound that is comparable to Transformer’s. Importantly, our results reveal that Mamba can perform a variant of online gradient descent to learn the latent function in context. This mechanism is different from that of Transformer, which is typically understood to achieve ICL through gradient descent emulation. The theoretical results are verified by experimental simulation. 1 Introduction State-space models (SSMs), notably Mamba ( Gu and Dao, 2024 ) , have recently emerged as compelling alternatives to Transformer-based architectures ( Vaswani et al., 2017 ) . Mamba integrates gating, convolutions, and state-space modeling with selection mechanisms, enabling linear-time complexity. This effectively addresses the quadratic computational costs typically associated with self-attention mechanisms in Transformers. Consequently, Mamba demonstrates superior efficiency in processing long sequences while maintaining or even surpassing Transformer performance across diverse benchmarks ( Gu and Dao, 2024 ; Dao and Gu, 2024 ; Patro and Agneeswaran, 2024 ; Liu et al., 2024 ; Ahamed and Cheng, 2024 ; Li et al., 2024a ; Li et al., 2024b ) . In-context learning (ICL) ( Brown et al., 2020 ) is a powerful paradigm that enables models to generalize to unseen tasks by dynamically leveraging contextual examples (such as input-output pairs) without task-specific fine-tuning. This capability has become a defining characteristic of large foundation models, significantly enhancing their flexibility and adaptability. While extensive research has provided substantial insights into Transformer-based ICL mechanisms ( Garg et al., 2022 ; Gatmiry et al., 2024 ; Sander et al., 2024 ; Zheng et al., 2024 ; Zhang et al., 2025 ) , the principles underlying Mamba’s ability to perform in-context learning remain largely unexplored, highlighting a compelling research gap. Recent empirical studies have examined Mamba’s (ICL) capabilities, showing it matches Transformers on many standard ICL tasks, while surpassing them in specialized scenarios like sparse parity ( Park et al., 2024 ; Grazzi et al., 2024 ) . Bondaschi et al., 2025 theoretically analyzed its representational capacity for in-context learning of Markov chains, and Li et al., 2025a investigated binary classification tasks with outliers. ( Yang et al., 2024 ; Yang et al., 2025 ; Behrouz et al., 2025b ; Behrouz et al., 2025a ) leverage the connection between SSMs and online learning to design new architectures. However, even the linear regression model, a canonical setting widely used for theoretical analysis of Transformer-based ICL mechanisms, remains theoretically underexplored in the context of Mamba. To fill this gap, we analyze Mamba’s training dynamics on in-context linear regression tasks. More precisely, following the previous ICL analysis in Transformers ( Garg et al., 2022 ; Zhang et al., 2024 ; Ahn et al., 2023 ) , this paper focuses on a data generative model with N input-output pairs ( \{\bm{x}_{i},y_{i}\}_{i=1}^{N} ) and a query input ( \bm{x}_{q} ) satisfying y=f(\bm{x})=\bm{w}^{\top}\bm{x} , where \bm{x} denotes the input and y denotes the output, and \bm{w} is randomly sampled from Gaussian distribution, termed the context . In this work, we develop a rigorous theoretical framework to analyze how randomly initialized Mamba models, when trained through gradient descent, evolve to implement in-context learning. We demonstrate that the trained Mamba architecture dynamically leverages the input context to perform implicit estimation of the vector \bm{w} . This estimation is achieved through hidden state updates that mimic online gradient descent steps, finally implementing prediction for y_{q}=f(\bm{x}_{q})=\bm{w}^{\top}\bm{x}_{q} . We also provide a loss bound that is comparable to Transformers’. Our contributions are summarized as follows: • We construct a Mamba architecture (S6: S4 with selection) capable of ICL, establishing its exponential convergence rate to ICL solution, and further derive the loss bound after convergence. The loss matches that of Transformers. • Technically, we develop novel techniques to address optimization challenges induced by random initialization and gradient descent, rigorously characterizing Mamba’s training dynamics when trained from scratch. • We reveal how trained Mamba achieves in-context linear regression by progressively aligning its hidden states with the context through sequential token processing. This finding provides a new perspective for understanding Mamba’s ICL mechanism, distinct from Transformer-based approaches. All the above results are verified by experiments. 2 Related Work In-Context Learning The seminal work of Brown et al., 2020 demonstrated the in-context learning capability in Transformers, showing their ability to infer functional mappings from input-output exemplars without weight updates. Garg et al., 2022 initiated the investigation of ICL from the perspective of learning particular function classes. Following these, a line of research analyze this phenomenon through the lens of algorithm imitation: Transformers can be trained to implement various learning algorithms that can mimic the latent functions in context, including: a single step of gradient descent ( Von Oswald et al., 2023 ; Akyürek et al., 2023 ) , statistical algorithms ( Bai et al., 2023 ) , reinforcement learning algorithm ( Lin et al., 2024 ) , multi-step gradient descent ( Gatmiry et al., 2024 ) , mesa-optimization ( Zheng et al., 2024 ) , Newton’s method ( Giannou et al., 2025 ) , weighted preconditioned gradient descent ( Li et al., 2025b ) , in context classification ( Bu et al., 2024 ; Shen et al., 2024 ; Bu et al., 2025 ) among others. Recent work extends ICL analysis beyond Transformers: ( Lee et al., 2024 ; Park et al., 2024 ) empirically compared popular architectures (e.g., RNNs, CNNs, SSMs, Transformers) on synthetic ICL tasks, identifying capability variations across model types and task demands. Tong and Pehlevan, 2024 demonstrate that MLPs can learn in-context a series of classical tasks such as regression and classification with less computation than Transformers. Sushma et al., 2024 show that state space models augmented with local self-attention can learn linear regression in-context. Unlike existing research on ICL, this work focuses on the ICL mechanism of Mamba (specifically S4 with selection) and its training dynamics. Theoretical Understanding of SSMs As Gu et al., 2022 introduce structured state spaces models in modeling long sequence and further be extended to Mamba ( Gu and Dao, 2024 ) , which gained significant attention as alternatives to Transformers, extensive research has sought to theoretically understand the mechanisms and capabilities of state-spaces models (SSMs). Dao and Gu, 2024 propose the framework of state space duality, which establishes a connection between SSMs and attention variants through the lens of structured matrices. Vankadara et al., 2024 provide a scaling analysis of signal propagation in SSMs through the lens of feature learning.