Block-Biased Mamba for Long-Range Sequence Processing
Annan Yu, N. Benjamin Erichson · 2025
Read Block-Biased Mamba for Long-Range Sequence Processing by Annan Yu, N. Benjamin Erichson (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2505-09022.txt.
Introduction
Annan Yu and Benjamin Erichson probe in 2025 why Mamba, a sequence model built for long range links, still falters on long range tasks. The model extends state space methods with input driven dynamics and shines in language and vision. Yet on tasks built to test long memory, it lags older designs.
The pair inspect three angles: what the model can express, its built in bias, and how steady its training is. On each, they show a gap against a classic rival. Their fix adds block wise dynamics and a per channel bias to the core unit. The new unit beat classic rivals on long range tests while holding language scores.
Catalog record
| CREATOR | Annan Yu, N. Benjamin Erichson |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 96 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2505-09022
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Block-Biased Mamba for Long-Range Sequence Processing Annan Yu, N. Benjamin Erichson Abstract Mamba extends earlier state space models (SSMs) by introducing input-dependent dynamics, and has demonstrated strong empirical performance across a range of domains, including language modeling, computer vision, and foundation models. However, a surprising weakness remains: despite being built on architectures designed for long-range dependencies, Mamba performs poorly on long-range sequential tasks. Understanding and addressing this gap is important for improving Mamba’s universality and versatility. In this work, we analyze Mamba’s limitations through three perspectives: expressiveness, inductive bias, and training stability. Our theoretical results show how Mamba falls short in each of these aspects compared to earlier SSMs such as S4D. To address these issues, we propose B 2 S 6 \text{B}_{2}\text{S}_{6} , a simple extension of Mamba’s S6 unit that combines block-wise selective dynamics with a channel-specific bias. We prove that these changes equip the model with a better-suited inductive bias and improve its expressiveness and stability. Empirically, B 2 S 6 \text{B}_{2}\text{S}_{6} outperforms S4 and S4D on Long-Range Arena (LRA) tasks while maintaining Mamba’s performance on language modeling benchmarks. 1 Introduction Mamba [ 26 ] has recently emerged as an exciting alternative to Transformers, demonstrating strong empirical performance not only in language tasks [ 84 , 45 ] , but also in a variety of domains, including vision [ 109 , 105 ] , audio [ 43 ] , time series forecasting [ 43 ] , and operator learning [ 33 ] . Based on the state-space model (SSM) framework of S4 and S4D, Mamba replaces attention with a selective state space mechanism. Unlike attention, which computes pairwise interactions across the sequence, this approach uses data-dependent weights in a recurrent structure, aligning with the temporal nature of sequential data. Despite its success as a general-purpose sequence modeling tool [ 51 ] , Mamba consistently underperforms on benchmarks such as the Long-Range Arena (LRA) [ 3 ] . This limitation is surprising, given that Mamba is derived from architectures specifically designed for modeling long-range dependencies. This behavior raises an important question: Why does Mamba struggle with long-range sequence modeling? To address this question, we conduct a theoretical analysis of the S6 state-space unit that underlies the Mamba architecture. Our goal is twofold: first, to explain why Mamba underperforms on long-range sequence tasks, and second, to provide a principled remedy. Through a rigorous mathematical analysis, we identify three key factors that limit Mamba’s ability to model long-range dependencies: • Expressiveness. Unlike S4D, which allows each internal channel to learn an independent recurrent unit, Mamba shares parameters across all channels. This design reduces the “effective width” of the model and, in turn, its capacity. We show that Mamba does not have universal approximation properties and cannot approximate a wide range of sequence transformations (see Theorem 1 ). • Inductive Bias. Unlike S4D, which uses a position-based bias that helps preserve long-range memory, Mamba adaptively decides what information to remember or forget based on the input. This input-dependent mechanism is too extreme and can cause the model to discard useful long-term information too quickly, leading to poor retention in long-range tasks (see Theorem 2 ). • Training Stability. Unlike S4D, which tends to train reliably on long sequences, Mamba empirically exhibits unstable training behavior on the LRA benchmark tasks. We theoretically show that this instability comes from the input-dependent selection mechanism, which becomes increasingly difficult to optimize as the length of the sequence increases (see Theorem 3 ). S4D S6 (Mamba) B 2 S 6 [formula] Figure 1 : Comparison of S4D, S6, and \text{B}_{2}\text{S}_{6} units. S4D uses independent linear SSM units for each channel, giving it high capacity (or width) but no input-dependent selectivity. S6 introduces a selective mechanism that modulates its internal dynamics based on the input, but shares parameters across channels, limiting its effective width and expressiveness. Our proposed \text{B}_{2}\text{S}_{6} unit partitions the input into smaller blocks, enabling selective behavior across multiple subspaces (see also [ 20 ] ). Additionally, it includes a channel-specific, input-independent bias term that further increases model capacity. These design choices improve the performance on long-range sequence tasks. Motivated by these findings, we propose the Block-Biased-S6 unit ( \text{B}_{2}\text{S}_{6} ), a simple yet principled extension of the S6 architecture that addresses its limitations in modeling long-range dependencies (see Figure 1 ). Like S6, \text{B}_{2}\text{S}_{6} retains the selective mechanism that modulates internal dynamics based on the input. However, instead of computing the selection weights from the full input \mathbf{u}_{k} , we divide the model into smaller blocks, with each block operating on a subset of \mathbf{u}_{k} . This multihead structure was already suggested in [ 20 ] , but in addition to that, we introduce an input-independent, channel-specific bias term to the recurrent unit. We prove that these modifications enhance \text{B}_{2}\text{S}_{6} ’s expressiveness and assign it a more appropriate inductive bias for handling long-range dependencies. We also apply a reduced learning rate to the parameters governing the input-dependent sampling intervals, which improves training stability. Together, these modifications enable \text{B}_{2}\text{S}_{6} to model long-range dependencies more effectively. On the LRA benchmark, \text{B}_{2}\text{S}_{6} resurrects Mamba from failure and even outperforms S4 and S4D. Furthermore, we show that \text{B}_{2}\text{S}_{6} achieves comparable perplexity to Mamba when trained on language tasks, demonstrating its versatility across domains. Contributions. Our main contributions are as follows: 1. We analyze the limitations of Mamba in modeling long-range dependencies and provide theoretical results that characterize the failure modes from three key aspects: (i) expressiveness (see Theorem 1 ), (ii) inductive bias (see Theorem 2 ), and (iii) training stability (see Theorem 3 ). 2. We introduce the \text{B}_{2}\text{S}_{6} unit, a principled extension of the S6 architecture. We show that \text{B}_{2}\text{S}_{6} has universal approximation capabilities (see Theorem 4 ) and is thus more expressive, while also equipped with a more suitable inductive bias for long-range sequence processing (see Theorem 5 ). 3. We evaluate \text{B}_{2}\text{S}_{6} on the Long-Range Arena benchmark, where it achieves state-of-the-art performance. In addition, a preliminary examination on the SlimPajama dataset [ 78 ] shows that \text{B}_{2}\text{S}_{6} matches Mamba’s performance on language modeling tasks, demonstrating its versatility. We refer interested readers to Appendix A for an extended discussion of related works. 2 The S4D and S6 recurrent units Introduced in [ 28 ] and [ 27 ] , the S4 and S4D recurrent units process sequences of d -dimensional inputs \mathbf{u}=(\mathbf{u}_{1},\ldots,\mathbf{u}_{L}) to produce corresponding d -dimensional outputs \Gamma_{\text{S4/S4D}}(\mathbf{u})=\mathbf{y}=(\mathbf{y}_{1},\ldots,\mathbf{y}_{L}) . Throughout this paper, we use subscripts to index positions in a sequence and superscripts to index channels. For example, \mathbf{u}_{k}^{(i)} denotes the i th entry of the k th input vector \mathbf{u}_{k} . Each channel of the S4/S4D model is computed independently with a linear system. For the i th channel, the update rule is \mathbf{x}^{(i)}_{k}=\overline{\mathbf{A}}^{(i)}\mathbf{x}^{(i)}_{k-1}+\overline{\mathbf{B}}^{(i)}\mathbf{u}_{k}^{(i)},\quad\mathbf{y}_{k}^{(i)}=\overline{\mathbf{C}}^{(i)}\mathbf{x}^{(i)}_{k},\qquad 1\leq i\leq d, (1) where \mathbf{x}^{(i)}_{k}\in\mathbb{C}^{n\times 1} is the hidden state with initial condition \mathbf{x}^{(i)}_{0}=\mathbf{0} , and \mathbf{u}_{k}^{(i)}\in\mathbb{R} . The output \mathbf{y}_{k}^{(i)} is technically a complex number but we usually take its real part only. The matrices \overline{\mathbf{A}}^{(i)}\in\mathbb{C}^{n\times n} , \overline{\mathbf{B}}^{(i)}\in\mathbb{C}^{n\times 1} , and \overline{\mathbf{C}}^{(i)}=\mathbb{C}^{1\times n} are derived from a continuous-time system. While there are many different discretization rules, we focus on the Zero-Order Hold (ZOH) discretization: \overline{\mathbf{A}}^{(i)}=\exp(\Delta^{(i)}\mathbf{A}),\quad\Delta^{(i)}=\text{exp}(b^{(i)}),\qquad\overline{\mathbf{B}}^{(i)}=\mathbf{A}^{-1}(\overline{\mathbf{A}}^{(i)}-\mathbf{I})\mathbf{B}^{(i)},\quad\overline{\mathbf{C}}^{(i)}=\mathbf{C}^{(i)}. (2) Here, \mathbf{A}\in\mathbb{C}^{n\times n} , \mathbf{B}^{(i)}\in\mathbb{C}^{n\times 1} , \mathbf{C}^{(i)}\in\mathbb{C}^{1\times