Characterizing the Behavior of Training Mamba-based State Space Models on GPUs
Trinayan Baruah 1 Kaustubh Shivdikar 1 Sara Prescott 2 David Kaeli 3 · 2025
Read Characterizing the Behavior of Training Mamba-based State Space Models on GPUs by Trinayan Baruah 1 Kaustubh Shivdikar 1 Sara Prescott 2 David Kaeli 3 (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2508-17679.txt.
Introduction
Trinayan Baruah, Kaustubh Shivdikar, Sara Prescott, and David Kaeli published a computing paper in 2025 profiling how Mamba-style sequence models behave while training on graphics processors. These state space models dodge the quadratic cost of attention, suiting long sequences in text, video, and graphs. The team built a suite of representative models spanning several architectures. Profiling tools then measured where time and resources go.
Their analysis exposes the demands these workloads place on current hardware design. Findings point to concrete optimizations for keeping performance scaling as models grow. The authors frame the suite as a reference for architects designing chips for the post-transformer wave of sequence models.
Catalog record
| CREATOR | Trinayan Baruah 1 Kaustubh Shivdikar 1 Sara Prescott 2 David Kaeli 3 |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 21 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2508-17679
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Characterizing the Behavior of Training Mamba-based State Space Models on GPUs Trinayan Baruah 1 Kaustubh Shivdikar 1 Sara Prescott 2 David Kaeli 3 Abstract Mamba-based State Space Models (SSM) have emerged as a promising alternative to the ubiquitous transformers. Despite the expressive power of transformers, the quadratic complexity of computing attention is a major impediment to scaling performance as we increase the sequence length. SSMs provide an alternative path that addresses this problem, reducing the computational complexity requirements of self-attention with novel model architectures for different domains and fields such as video, text generation and graphs. Thus, it is important to characterize the behavior of these emerging workloads on GPUs and understand their requirements during GPU microarchitectural design. In this work we evaluate Mamba-based SSMs and characterize their behavior during training on GPUs. We construct a workload suite that offers representative models that span different model architectures. We then use this suite to analyze the architectural implications of running Mamba-based SSMs on GPUs. Our work sheds new light on potential optimizations to continue scaling the performance for such models. I Introduction The past few years have seen explosive growth in the field of Deep Learning (DL). Early DL models were primarily used for tasks such as product recommendations, search and social media recommendations. In contrast, DL models today directly interact with the users via services such as chatbots, coding-assistants and video generation. This major transformation over the past few years has been driven by the development of Generative AI models, such as Large Language Models (LLMs) and vision-based LLMs. Despite the advances made in recent years by LLM, scaling training of these models has become a major challenge due to the performance of the self-attention layer [ 1 ] . One of the primary challenges in scaling LLMs for training is the quadratic dependence of the compute time on the sequence length. The problem is further exacerbated by the fact that the softmax operation is usually done at higher precisions (e.g., FP16 and FP32) using the vector and exponential units on the GPUs. Today’s GPUs are optimized to run matrix-multiplication (GEMM) on tensor cores, which are optimized for computing GEMMs using lower precision such (e.g., FP4 and FP8). If we consider the FP8 tensor compute throughput on current GPU, such as NVIDIA Hopper H100, the ratio between that and the exponential throughputs is 512. As identified by Shah et al. [ 2 ] , the softmax operator is a major bottleneck and limits efficient performance scaling as a function of sequence length. Machine learning researchers have come up with different methods to address this issue. One approach is to focus only on a given window of tokens [ 3 ] , as used in models such as Mistral7B and LLAMA4. The other approach is to develop foundational models that overcome the quadratic computational scaling barriers, as offered by State Space Models (SSMs), which are the subject of this paper. Fig. 1 : Mamba Operator Breakdown SSMs have emerged as a promising alternative to tackle this problem in recent years. The core idea is borrowed from Recurrent Neural Networks (RNNs) where the output only depends on the previous token instead of attention, where for each output token the model needs to consider all previous tokens. SSMs instead have a hidden state that holds a compressed history of all previous tokens. The sequential nature of the original SSMs, such as S4 [ 4 ] , impact their scalability on parallel hardware such as a GPU. Additionally, the original SSM models did not perform well on various tasks such as copying and induction heads. Mamba [ 5 ] , [ 6 ] was proposed as a selective SSM which was designed to solve both issues. As a result, Mamba has been adopted in popular libraries such as NVIDIA-Megatron [ 7 ] . Given the ubiquity of GPUs for training and inference for such Mamba based models, we need to better understand the performance implications of training Mamba-based models. If we can characterize the common patterns and operators present in SSM models, this information can help guide future hardware design. Identifying architecture bottlenecks present on the GPU when running SSMs can help guide software developers to better optimize their code. In this paper we characterize the behavior of Mamba-based SSM models on GPUs. We provide detailed characteristics of the key SSM operators in Mamba, dissecting key computational and memory behavior, as well as identifying the key bottlenecks so that we can effectively scale the performance of such models on future systems. II Background of SSMs and Mamba In this section, we describe of State Space Models (SSMs), which serve as the foundational building block of Mamba. We refer readers to the work of Gu et al. [ 4 , 5 ] and Dao et al. [ 6 ] for a deeper dive into the mathematics underlying these models. Fundamentally, SSMs can be defined using four matrices A,B,C,D , a hidden state h , an input x , and an output y . The general formulation of an SSM is given by Equations ( 1 ) and ( 2 ). \displaystyle x_{k+1} \displaystyle=Ax_{k}+Bu_{k} (1) \displaystyle y_{k} \displaystyle=Cx_{k}+Du_{k} (2) The state equation of an SSM captures how the hidden state is updated using matrix A, as well as the influence of the input x on matrix B. The output equation captures how the output evolves based on the impact of the state with matrix C, and includes the impact of the input with matrix D. In SSMs such as S4 [ 4 ] , the state-space matrices are independent of the input, which makes them ineffective for certain tasks such as selective copying and induction heads [ 5 ] . To address these limitations, Mamba [ 5 ] introduced modifications in how the state matrices are handled. While the A matrix remains constant, the B and C matrices are now dynamically computed based on the input, making the model content-aware. To enable efficient execution on parallel hardware, such as GPUs, Mamba employs a parallel scan algorithm [ 8 ] . Modern GPUs are heavily optimized for matrix multiplication, with specialized acceleration capabilities such as tensor cores. To further accelerate computation, the developers of Mamba have proposed imposing structural constraints on the A matrix so that the operations can be reformulated as a matrix-multiplication algorithm [ 6 ] instead of using a parallel scan. In this paper, we refer to this implementation as Mamba2. In a Mamba-based SSM model, three of the matrices (namely, X , B and C ) undergo an input projection layer which is basically a fully-connected layer. This is then followed by a one-dimensional convolution and a nonlinear activation function (typically a softplus) applied to the matrices. Matrix A is passed through the input projection layer without going through a convolution and non-linear activation layer. All four matrices are then fed into the SSM block, which is the core innovation of Mamba’s based models. This step performs the operations shown in Equations ( 1 ) and ( 2 ). III Workload Selection and Evaluation Our goal in this work is twofold. We want to understand the variations at the machine learning model level