Routing Mamba: Scaling State Space Models with Mixture-of-Experts Projection

Zheng Zhan, Liliang Ren, Shuohang Wang, Liyuan Liu, Yang Liu, Yeyun Gong, Yanzhi Wang, Yelong Shen · 2025

Read Routing Mamba: Scaling State Space Models with Mixture-of-Experts Projection by Zheng Zhan, Liliang Ren, Shuohang Wang, Liyuan Liu, Yang Liu, Yeyun Gong, Yanzhi Wang, Yelong Shen (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2506-18145.txt.

Introduction

Zheng Zhan and co-authors scale Mamba models with routed experts in a 2025 paper. State space models read long sequences at flat cost, and Mamba leads that pack. Growing their skill with mixture of experts tricks had mostly failed or backfired.

Routing Mamba spreads sparse expert sets across its projection layers and shares routing calls among them. Light sub modules inside Mamba join the same routing, which ties the system together. At a set size, the routed model matched a dense one with over twice the active weights. Hybrid builds saved near a quarter of compute as well.

Catalog record

CREATORZheng Zhan, Liliang Ren, Shuohang Wang, Liyuan Liu, Yang Liu, Yeyun Gong, Yanzhi Wang, Yelong Shen
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE50 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2506-18145

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Routing Mamba: Scaling State Space Models with Mixture-of-Experts Projection Zheng Zhan, Liliang Ren, Shuohang Wang, Liyuan Liu, Yang Liu, Yeyun Gong, Yanzhi Wang, Yelong Shen Abstract Linear State Space Models (SSMs) offer remarkable performance gains in efficient sequence modeling, with constant inference-time computation and memory complexity. Recent advances, such as Mamba, further enhance SSMs with input-dependent gating and hardware-aware implementations, positioning them as strong alternatives to Transformers for long sequence modeling. However, efficiently scaling the expressive power of SSMs, particularly with Mixture of Experts (MoE), remains challenging, as naive integration attempts often falter or degrade performance. In this work, we introduce Routing Mamba (RoM), a novel approach that scales SSM parameters using sparse mixtures of linear projection experts. By sharing routing decisions between projection layers and lightweight sub-modules within Mamba across experts, RoM leverages synergies among linear projection experts for effective and efficient sparse scaling of Mamba layers. At a scale of 1.3B active parameters (10B total) and 16K training sequence length, RoM achieves language modeling performance equivalent to a dense Mamba model requiring over 2.3 × \times more active parameters, and demonstrates consistent perplexity across context lengths. Experimental results further show RoM effectively scales hybrid language models, yielding a 23% FLOPS saving compared to dense Mamba scaling for similar performance. 1 Introduction State Space Models (SSMs) have recently gained substantial attention for their efficiency in addressing long-range dependencies and their ability to support parallel training. Originally from the Kalman filter model [ 24 ] , modern SSMs [ 18 , 19 , 17 , 20 , 6 ] have been successfully applied to diverse sequence modeling tasks across various modalities. Among these, Mamba [ 15 ] has emerged as a notable advancement by introducing time-varying parameters into SSMs, enabling selective memorization of input information. It further integrates a hardware-aware selective scan algorithm that accelerates both training and inference, offering a subquadratic-time architecture that scales efficiently with sequence length. Mamba’s scalability allows it to be a competitive alternative to Transformer-based models [ 44 ] for generative language modeling in low-resource scenarios. In parallel to these advancements, Mixture of Experts (MoE) [ 22 , 25 , 13 ] has become a cornerstone for efficiently scaling the number of parameters of large language models [ 34 , 9 , 32 ] . Most contemporary approaches focus on integrating MoE into the feed-forward layers, treating each expert as an independent Feed-forward Network (FFN). Previous works like SwitchHead [ 5 ] and Mixture of Attention Heads (MoA) [ 50 ] extend this paradigm through leveraging attention projections as experts. These works highlight the potential for introducing sparsely activated experts to token-mixing layers for improved performance and efficiency in large language models. Despite these advancements in MoE for Transformers and attention mechanisms, its application to SSMs remains largely unexplored. Specifically, while large-scale SSM models like Mamba [ 16 ] , Jamba [ 27 ] , and Samba [ 39 ] have demonstrated their efficacy, no prior work has successfully investigated the use of Mamba layers as MoE experts to further scale the number of total model parameters. To bridge this gap, our study investigates the feasibility of scaling Mamba layers using the sparse mixture of linear projection experts. By leveraging Mamba’s intrinsic synergies across sub-modules, we successfully improve model quality and scalability with sparse activation of linear projection layers. This work represents a pioneering effort toward developing large-scale models that utilize Mamba layers as scalable and efficient experts within the sparse MoE framework. We first conduct comprehensive experiments to reveal that applying a straightforward MoE scaling strategy to Mamba layers not only fails to enhance performance but also leads to performance degradation and increased latency overhead, without delivering meaningful improvements. These findings highlight the inherent complexities and challenges of effectively scaling Mamba layers with sparse mixture of projection experts, emphasizing the need for a more tailored approach. Unlike conventional MoE implementations that primarily focus on feedforward layers, our work systematically studies the sparse scaling of Mamba’s unique projection layers, consisting of Convolution (Conv), Gate, and Output projections, and transforms them into sparsely activated experts of linear layers. Through selective scaling of sub-modules and a shared routing strategy, our design significantly improves computational efficiency while expanding model capacity. Beyond addressing the challenges of integrating MoE with SSMs, we also study the effectiveness of RoM in hybrid SSMs-Attention model architectures with various sparse scaling strategies, broadening the horizons for large-scale sequence modeling with SSMs. With our tailored design, RoM achieves significant improvements in both performance and efficiency. Specifically, across a scale from 115M up to 10B total parameters, RoM demonstrates superior scaling behavior compared to dense Mamba models, achieving equivalent perplexity performance to its dense counterparts that require up to 2.3 \times more active parameters. RoM also achieves a 23% FLOPS reduction when applied to Samba, a typical Mamba-attention hybrid model. We summarize our contributions as follows: • We propose the first framework for integrating sparse mixture of projection experts into SSMs, demonstrating their applicability beyond traditional feed-forward networks, shedding light on future research on sparse scaling of SSMs and hybrid models. • Our selective scaling and shared routing mechanism enable scalable and efficient training for large-scale sequence modeling tasks, achieving model performance that is comparable to up to 2.3 \times larger dense models. • We validate the effectiveness of RoM across various configurations, demonstrating its adaptability and generalization. Notably, RoM achieves a 23% reduction in FLOPS for hybrid model scaling, highlighting its potential and setting a new direction for efficient sequence modeling. 2 Related Work State Space Models. State Space Models [ 16 , 31 , 45 ] have emerged as a compelling architectural paradigm for sequence-to-sequence transformations. These models are well-suited for capturing complex dynamics in sequential data. Mamba-2 [ 6 ] extends this foundation by leveraging state space duality, introducing a refined architecture based on selective SSMs that demonstrates both versatility and scalability. Recently, SSM-based Small Language Models (SLM) such as Gated DeltaNet [ 47 ] , Jamba [ 27 ] , Samba [ 39 ] , and Hymba [ 10 ] have shown significant scalability and effectiveness. These models adhere to scaling laws and have proven particularly promising for small-scale language modeling, offering a balance of performance and cost-efficiency for diverse and complex tasks. Despite these advancements, previous works [ 27 , 37 ] fail to successfully integrate MoE with the SSMs. Considering that SSM is the fundamental component in these architectures, incorporating MoE with SSMs presents an intuitive and innovative way to enhance performance while maintaining computational efficiency. This integration has the potential to combine the strengths of both paradigms, enabling more scalable and resource-efficient solutions. Mixture of Expert. Mixture of Experts models have emerged as a powerful technique in LLMs. The original concept of MoE [ 22 ] involves partitioning the input space into regions and training specialized experts for each region, with a gating network to dynamically select the most suitable expert for a given input. Recent advancements [ 42 , 23 , 11 , 13 ] have leveraged the MoE framework to address the increasing complexity and scale of modern datasets. Extensions like GShard [ 25 ] have adapted MoE to multilingual settings, enabling them to scale effectively across

Read the complete text