Swimba: Switch Mamba Model Scales State Space Models
Zhixu Du 1, Krishna Teja Chitty-Venkata 2, Murali Emani 3, Venkatram Vishwanath 3 Hai Helen Li 1, Yiran Chen 1 1 Duke University 2 Red Hat, Inc. 3 Argonne National Laboratory · 2026
Read Swimba: Switch Mamba Model Scales State Space Models by Zhixu Du 1, Krishna Teja Chitty-Venkata 2, Murali Emani 3, Venkatram Vishwanath 3 Hai Helen Li 1, Yiran Chen 1 1 Duke University 2 Red Hat, Inc. 3 Argonne National Laboratory (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2603-06938.txt.
Introduction
Du and colleagues present Swimba, a Switch Mamba model that adds expert specialization to state space models. Mixture-of-experts raises capacity, but for state space mixers it can multiply recurrent update cost. One design runs experts over separate models, keeping many paths, so cost grows with expert count. A second design mixes experts in parameter space, keeping one state path and running the recurrence once. Swimba takes the second route, routing over expert-produced streams, with proofs of well-definedness and stability and tests of speed and latency.
Catalog record
| CREATOR | Zhixu Du 1, Krishna Teja Chitty-Venkata 2, Murali Emani 3, Venkatram Vishwanath 3 Hai Helen Li 1, Yiran Chen 1 1 Duke University 2 Red Hat, Inc. 3 Argonne National Laboratory |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 41237 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2603.06938
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Swimba: Switch Mamba Model Scales State Space Models Zhixu Du 1, Krishna Teja Chitty-Venkata 2, Murali Emani 3, Venkatram Vishwanath 3 Hai Helen Li 1, Yiran Chen 1 1 Duke University 2 Red Hat, Inc. 3 Argonne National Laboratory Abstract Mixture-of-experts (MoE) is a common approach for increasing parameter capacity, but applying MoE to state space model (SSM) token mixers can multiply the cost of the recurrent state update. We study how to introduce expert specialization into selective SSMs while preserving computational efficiency. We show that MoE–SSM can refer to two designs: (1) MoE over separated SSMs, which maintains multiple state trajectories and thus scales compute with the number of experts; and (2) MoE-parameterized SSM, which mixes experts in parameter space, maintains a single state trajectory, and evaluates the recurrence once. Our method, Switch Mamba (Swimba), follows the second design by routing over expert-produced SSM streams. Theoretically, we establish well-definedness and stability for MoE-parameterized SSMs and characterize the relationship between the two designs. Empirically, we evaluate Swimba on standard benchmark tasks and measure real-time throughput and latency. Under matched FLOPs, Swimba achieves slightly better average performance than the baseline, with a small slowdown in real-time latency and throughput. Overall, these results suggest that parameter-space MoE can increase SSM capacity while keeping the dominant recurrence cost fixed. 1 Introduction State space models (SSMs) have become a practical alternative to attention for long-sequence modeling, combining recurrence with modern accelerator-friendly implementations. Recent selective SSM architectures such as Mamba [ 20 ] and its SSD reformulation in Mamba-2 [ 11 ] show that linear-time token mixing can be competitive with full attention. At the same time, scaling language models often relies on mixture-of-experts (MoE) to increase parameter count without proportionally increasing computational overhead [ 34 , 13 ] . Most MoE work targets feed-forward blocks, where expert activation does not replicate a costly recurrent state update, with only a few works applying MoE to attention [ 39 , 9 ] . Extending MoE to SSM token mixers can retain the O(L) computational complexity while boosting layer capacity through MoE scaling. However, it introduces a different challenge. The core recurrence is the dominant cost in SSMs, so naive implementation can scale compute with the number of experts. Figure 1 : Compute scaling view for token mixing. The x-axis is sequence length L and the y-axis is per-token computational overhead. Attention scales as O(L^{2}) while SSMs scale as O(L) . Bubble area indicates parameter count, illustrating that MoE can increase parameters with only a small increase in per-token compute, motivating Swimba (MoE-SSM) designs. MoE-SSM can refer to two different designs, and the distinction matters for both efficiency and modeling behavior. MoE of separated SSMs maintains an expert-specific state trajectory and advances multiple recurrences in parallel. This resembles switching dynamics in state-space models [ 16 ] : each expert can store its own memory, but compute and state storage grow with the number of experts. In contrast, MoE-parameterized SSM keeps a single state trajectory and mixes experts in parameter space. Instead of running multiple recurrences and combining outputs, it forms one effective selective SSM by aggregating expert-dependent injection and readout streams, then evaluates the recurrence once with an SSD-style computation [ 11 ] . Much of the prior MoE–SSM hybrid literature increases capacity by interleaving MoE MLP blocks with dense SSM mixers [ 26 , 30 , 1 ] . For approaches that apply MoE within the SSM block, existing work is largely engineering-driven and often does not distinguish between the two MoE–SSM designs [ 38 ] . Without a clear taxonomy, it is easy to conflate these designs and their scaling properties. This paper focuses on MoE-parameterized SSMs. We present Switch Mamba (Swimba) , an MoE-parameterized SSM layer that preserves a single-pass recurrence. Swimba builds on Mamba-2 [ 11 ] : each expert produces candidate selective SSM streams, and a token-level router computes mixture weights. We then mix the expert-dependent SSM streams in parameter space and run a single SSM evaluation to update the recurrence. In our implementation, specialization enters through routing-weighted SSM streams, while state evolution remains a single trajectory. This design targets the central efficiency goal of SSMs: it avoids replicating the expensive recurrence across experts, while scaling the number of model parameters. We also provide formal theoretical statements that motivate the MoE-parameterized SSM design choice. Specifically, Theorem 3.1 shows that mixing in parameter space preserves the single-SSM structure required by Mamba-2 for efficiency. Theorem 3.2 shows that the recurrence cost does not scale with the number of experts. Under a contractive transition, Theorem 3.3 shows that stability can be controlled through bounds on the mixed streams. Finally, we relate MoE-parameterized SSMs to the MoE of separated SSMs baseline: Theorem 3.4 identifies regimes where the two agree, and Theorem 3.5 characterizes the mismatch introduced when routing varies over time, while still allowing increased expressivity from input-dependent mixtures. Empirically, we build Swimba on the Nemotron-H-8B [ 4 ] hybrid backbone, replacing each Mamba-2 token-mixing layer with our MoE-parameterized SSM layer while keeping the rest of the architecture unchanged. We evaluate against the Nemotron-H-8B baseline on standard benchmark tasks. We compare model FLOPs to the baseline and benchmark end-to-end decoding with vLLM [ 24 ] under matched settings. Across these evaluations, Swimba achieves slightly better average performance than the baseline at comparable total FLOPs, while showing a small slowdown in real-time latency and throughput. Prior results [ 6 ] indicate that increasing the number of experts while keeping the number of active experts fixed has limited impact on latency and throughput, suggesting that Swimba retains favorable scaling behavior. Our contributions can be summarized as follows: • We distinguish two MoE–SSM designs, MoE of separated SSMs and MoE-parameterized SSMs , and explain how this distinction determines scaling in compute and memory. We provide theoretical results on well-definedness, stability, and the relationship between the two designs. • We introduce Swimba, an MoE-parameterized SSM layer that mixes expert-dependent streams in parameter space and preserves SSM computation as a single recurrence evaluation. • We evaluate Swimba on standard benchmarks and report compute- and decoding-oriented measurements (FLOPs analysis and vLLM throughput/latency) under matched settings. Swimba achieves slightly better average performance than the baseline at comparable runtime cost, suggesting strong potential for scaling. 2 Background Notation. The input sequence has length T , with X\in\mathbb{R}^{T\times P} and output Y\in\mathbb{R}^{T\times P} . We write X_{t}\in\mathbb{R}^{P} and Y_{t}\in\mathbb{R}^{P} for the t -th token. We use h_{t}\in\mathbb{R}^{N\times P} for the latent state, which can be viewed as P independent states \{h_{t,p}\in\mathbb{R}^{N}\}_{p=1}^{P} broadcast across channels. Parameters A , B , and C have shapes \mathbb{R}^{N\times N} , \mathbb{R}^{N\times P} , and \mathbb{R}^{N\times P} , respectively. Parameters A_{t} , B_{t} , and C_{t} refers to the time-varying parameters at time t , sharing the same shape as A , B , and C . We use E to denote the total number of experts in an MoE model and E_{e}(\cdot) to denote the e -th expert function. 2.1 State Space Models State Space Models (SSMs) provide a framework for modeling sequences by discretizing a continuous-time system. The system maps a 1-D input function x(t)\in\mathbb{R}\to y(t)\in\mathbb{R} via a latent state h(t)\in\mathbb{R}^{N} evolving according to the differential equation h^{\prime}(t)={A}h(t)+{B}x(t)