Differential Mamba

Nadav Schneider, Itamar Zimerman, Eliya Nachmani · 2025

Read Differential Mamba by Nadav Schneider, Itamar Zimerman, Eliya Nachmani (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2507-06204.txt.

Introduction

Language models built on the Mamba design read sequences fast, yet they still waste focus on irrelevant context. Nadav Schneider, Itamar Zimerman, and Eliya Nachmani adapt a fix from a rival design in this 2025 paper. The fix is differential design: subtract one attention map from another to cancel noise. In transformer models that trick cut hallucinations and sharpened recall.

A straight copy of the trick into Mamba did not work, the authors found. Mamba's layers use selective state-space steps rather than attention, so the subtraction needed a careful rebuild. Their new differential mechanism fits that structure. On language modeling tests, the rebuilt model recalled facts better and beat the plain Mamba design. Extra probe tests confirmed where the gains come from. Cleaner focus could lift long-document work.

Catalog record

CREATORNadav Schneider, Itamar Zimerman, Eliya Nachmani
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE40 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2507-06204

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Differential Mamba Nadav Schneider, Itamar Zimerman, Eliya Nachmani Abstract Sequence models like Transformers and RNNs often overallocate attention to irrelevant context, leading to noisy intermediate representations. This degrades LLM capabilities by promoting hallucinations, weakening long-range and retrieval abilities, and reducing robustness. Recent work has shown that differential design can mitigate this issue in Transformers, improving their effectiveness across various applications. In this paper, we explore whether these techniques, originally developed for Transformers, can be applied to Mamba, a recent architecture based on selective state-space layers that achieves Transformer-level performance with greater efficiency. We show that a naive adaptation of differential design to Mamba is insufficient and requires careful architectural modifications. To address this, we introduce a novel differential mechanism for Mamba, empirically validated on language modeling benchmarks, demonstrating improved retrieval capabilities and superior performance over vanilla Mamba. Finally, we conduct extensive ablation studies and empirical analyses to justify our design choices and provide evidence that our approach effectively mitigates the overallocation problem in Mamba-based models. Our code is publicly available. https://github.com/NadavSc/Diff-Mamba 1 Introduction Designing enhanced architectures for deep sequence modeling is a pivotal task in the ML community, as sequence models drive breakthroughs such as ChatGPT [ 1 ] in NLP and Stable diffusion [ 32 ] in computer vision, which are fundamental to modern generative models. However, these models face two major challenges as: efficiency, constrained by the quadratic time complexity in sequence length of the Transformer architecture [ 36 ] , and robustness, which is hindered by inconsistency, reliability issues, and hallucinations, leading to sub-optimal performance. Our work addresses these challenges by enhancing the robustness of recent efficient architectures with sub-quadratic complexity such as Mamba [ 16 ] , making them more reliable and robust. To address this robustness issue, we aim to reduce the over-allocation of attention to irrelevant context across hidden layers, which often leads to noisy representations. Our approach is inspired by Ye et al. 2024 , who mitigated this problem in transformers through differential design. This architectural modification was the core of the Diff-Transformer, a transformer variant that demonstrated improved empirical performance, including greater robustness, enhanced retrieval and long-context capabilities compared to the original model. Our focus is on improving the recently introduced Mamba architecture, which builds upon selective state-space layers (S6). This architecture is known for its efficiency, introducing sub-quadratic complexity in sequence length while also enabling auto-regressive decoding with a complexity that remains independent of sequence length. Beyond these efficiency advantages, recent studies have demonstrated that Mamba-based architectures can match or even surpass the SoTA performance of transformers, even at relatively large scales. For example, Falcon-Mamba 7B [ 46 ] , a pure Mamba model, matches the performance of LLaMA- 3-8B on several language tasks. Additional notable examples of Mamba’s integration in LLMs include Jamba [ 23 ] , Zamba [ 14 ] , Hymba [ 11 ] , and others [ 39 , 37 , 31 ] . More importantly, recent Mamba-based models have demonstrated remarkable performance as reasoning models [ 28 , 40 ] , underscoring their central role in the ongoing test-time scaling revolution, as exemplified by models such as OpenAI’s O1 [ 20 ] and DeepSeek’s R1 [ 19 ] . While the over-allocation problem is a general issue not specific to any architecture, we hypothesize that Mamba-based LLMs have a stronger tendency toward over-allocation compared to transformers. This is primarily due to two factors: (i) Mamba is a softmax-free architecture, meaning it lacks the exponential scaling effect that helps suppress irrelevant attention weights, and (ii) as a state-based model, Mamba operates locally and cannot directly process distant tokens without considering all intermediate tokens, resulting in the dispersion of important tokens among irrelevant ones. This leads to our central research question: Can differential design be leveraged to improve the robustness of Mamba models? where we hypothesize that improving robustness by mitigating the over-allocation problem can enhance the model’s general capabilities, improve retrieval and long-context processing, and potentially reduce issues related to consistency and hallucinations in LLMs. We provide a positive answer to this question by showing that, although a naive implementation of differential mechanisms does not improve Mamba architectures, a more carefully designed mechanism does. Through a systematic evaluation on language tasks, ablation studies, and empirical analysis, we conclude that our variant is favorable compared to the vanilla Mamba. Our main contributions are as follows: (i) We present Diff-Mamba, a modification of the Mamba architecture inspired by Diff-Transformer, which mitigates the problem of over-allocating attention scores to irrelevant context and improves the general language modeling abilities of the model. (ii) Through a series of ablation studies and empirical analysis using mechanistic interpretability tools, we justify our design choices and demonstrate that the intermediate representations obtained from our method are indeed less noisy. (iii) Finally, we show that Diff-Mamba demonstrates improved retrieval and long-context capabilities compared to Mamba. This is particularly important, as recurrent LLMs such as Mamba are primarily designed to address the inefficiencies caused by the quadratic complexity of transformers, which becomes especially critical in long-context regimes. Accordingly, Diff-Mamba achieves improvements over Mamba precisely in the domain where such architectures are most needed [ 6 ] . 2 Background Here we describe the scientific context and introduce the key terminology for discussing our method. 2.1 Differential Transformer Self-Attention Self-Attention is a fundamental component of Transformer architectures [ 36 ] , has significantly shaped recent advances in both NLP and computer vision. This mechanism enables dynamic focus allocation by capturing pairwise token dependencies, allowing the model to determine the relative importance of each token within a sequence. Mathematically, it is expressed as: Attention(Q,K,V)=\alpha V,\quad\alpha=\text{softmax}\left(\frac{QK^{T}}{\sqrt{d_{k}}}\right) (1) In this formulation, Q , K , and V represent the queries, keys, and values, respectively, while d_{k} denotes the key dimension. Transformers extend this mechanism by employing H parallel attention heads, enabling the model to capture a broader spectrum of dependencies. Differential Attention To address the problem of over-allocation of attention to irrelevant tokens, Ye et al. 2024 , introduced Diff Transformer, a mechanism that reduces attention noise through differential denoising by splitting each attention head into two and subtracting one attention map from the other. This mechanism can be described as follows: DiffAttention(Q_{1},K_{1},Q_{2},K_{2},V)=(\alpha_{1}-\lambda\alpha_{2}),\quad\alpha_{i}=\text{softmax}\left(\frac{Q_{i}{K_{i}}^{T}}{\sqrt{d_{k}}}\right) (2) where \lambda is is a learnable scalar. To better improve the training dynamics, \lambda is re-parameterized and Group Normalization [ 42 ] is applied at the end of each head (post-subtraction). 2.2 State-Space Layers State-space layers were first introduced in Gu et al. 2021 and were later substantially improved by the S4 model [ 18 ] . Since then, they have demonstrated strong performance across a variety of domains, including NLP [ 12 , 26 ] , audio generation [ 15 ] , image modeling [ 3 , 27 ] , long-horizon video understanding [ 38 ] , reinforcement learning [ 9 , 25 ] , and speech recognition [ 33 ] . These models implement linear recurrent update rules derived from time-invariant state-space formulations, which can be efficiently computed in parallel using convolutions and with sub-quadratic complexity. 2.3 Mamba and Selective State-Space Layers A Mamba block processes a signal U\in\mathbb{R}^{L\times D} where D is the hidden dimension and L is the number of tokens. Its core

Read the complete text