MambaVoiceCloning: Efficient and Expressive Text-to-Speech via State-Space Modeling and Diffusion Control
Sahil Kumar, Namrataben Patel, Honggang Wang, Youshan Zhang · 2026
Read MambaVoiceCloning: Efficient and Expressive Text-to-Speech via State-Space Modeling and Diffusion Control by Sahil Kumar, Namrataben Patel, Honggang Wang, Youshan Zhang (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2604-00292.txt.
Introduction
Kumar and three colleagues build MambaVoiceCloning, a text-to-speech system whose conditioning runs entirely on state-space models at inference, with no attention or recurrent layers. A gated two-way Mamba text encoder, a Temporal Bi-Mamba guided by a throwaway alignment teacher, and an Expressive Mamba with modulation give linear-time conditioning, bounded memory, and streaming. Against StyleTTS2, VITS, and hybrid rivals, it wins modest but reliable gains on quality, pitch, and word error, while cutting encoder size to 21 million parameters and raising throughput 1.6 times.
Catalog record
| CREATOR | Sahil Kumar, Namrataben Patel, Honggang Wang, Youshan Zhang |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 74375 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2604.00292
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
MambaVoiceCloning: Efficient and Expressive Text-to-Speech via State-Space Modeling and Diffusion Control Sahil Kumar, Namrataben Patel, Honggang Wang, Youshan Zhang Abstract MambaVoiceCloning (MVC) asks whether the conditioning path of diffusion-based TTS can be made fully SSM-only at inference—removing all attention and explicit RNN-style recurrence layers across text, rhythm, and prosody—while preserving or improving quality under controlled conditions. MVC combines a gated bidirectional Mamba text encoder, a Temporal Bi-Mamba supervised by a lightweight alignment teacher discarded after training, and an Expressive Mamba with AdaLN modulation, yielding linear-time 𝒪 ( T ) \mathcal{O}(T) conditioning with bounded activation memory and practical finite look-ahead streaming. Unlike prior Mamba–TTS systems that remain hybrid at inference, MVC removes attention-based duration and style modules under a fixed StyleTTS2 mel–diffusion–vocoder backbone. Trained on LJSpeech/LibriTTS and evaluated on VCTK, CSS10 (ES/DE/FR), and long-form Gutenberg passages, MVC achieves modest but statistically reliable gains over StyleTTS2, VITS, and Mamba–attention hybrids in MOS/CMOS, F 0 RMSE, MCD, and WER, while reducing encoder parameters to 21M and improving throughput by 1.6 × 1.6\times . Diffusion remains the dominant latency source, but SSM-only conditioning improves memory footprint, stability, and deployability. Code available at: https://github.com/sahilkumar15/MVC . 1 Introduction Text-to-Speech (TTS) systems continue to improve in naturalness and expressive control 17 ; 13 ; 21 ; 26 , yet most conditioning stacks rely on transformer attention 27 ; 30 or recurrent modules. Attention introduces quadratic computational and memory complexity and global context mixing, while recurrent architectures exhibit long-range drift and unstable memory dynamics. Linear attention variants 3 reduce asymptotic cost but preserve global interactions that complicate streaming. Meanwhile, diffusion decoders 23 ; 7 ; 19 ; 14 ; 32 dominate inference runtime, making encoder efficiency central to deployment. Why Mamba vs. Transformer/RNN. State-space models (SSMs), particularly Mamba 6 , provide bounded activations, linear-time sequence scans, and state-persistent streaming. These properties reduce memory pressure relative to attention-based models and mitigate drift in recurrent architectures, supporting stable conditioning over multi-sentence inputs. However, existing Mamba–TTS systems 11 ; 33 remain hybrid at inference, retaining attention-based duration or style modules that limit streaming robustness. This work investigates whether diffusion-based TTS can adopt a fully SSM-only conditioning stack at inference for text, rhythm, and prosody under a strictly matched mel–diffusion–vocoder pipeline. The StyleTTS2 decoder and vocoder remain fixed; only the conditioning path is redesigned. MVC introduces three selective SSM modules: a gated bidirectional Mamba text encoder; a Temporal Bi-Mamba aligned using a lightweight monotonic teacher during training only; and an Expressive Mamba with AdaLN modulation. A gated forward–backward fusion mechanism replaces concat-only bi-Mamba fusion used in prior work. Why NaturalSpeech 3, CosyVoice 3, and HiggsAudio-V2 are not direct baselines. Industrial-scale TTS systems such as NaturalSpeech 3 12 , CosyVoice 3 4 , and HiggsAudio-V2 2 rely on multi-hundred-thousand– to million-hour proprietary multilingual corpora, LLM-scale semantic encoders, and multi-stage pipelines. Their performance is driven primarily by scale rather than conditioning-architecture design, making them unsuitable as decoder-matched baselines for a controlled architectural study. For fairness, comparisons are restricted to open-data systems trained under identical preprocessing, mel front-end, vocoder, and optimization schedules; a detailed contextual comparison is provided in Appendix F . Scope of evaluation. The evaluation covers in-distribution speech (LJSpeech 8 , LibriTTS 31 ), zero-shot speakers (VCTK 28 ), cross-lingual CSS10 (ES/DE/FR) 22 , and 2–6 minute Gutenberg passages for long-form testing. MVC yields consistent improvements over StyleTTS2 17 , VITS 13 , and capacity-matched Mamba hybrids 11 ; 33 in MOS, CMOS, F_{0} RMSE, MCD, and WER, while reducing encoder parameters to 21M and improving throughput by a factor of 1.6. Streaming with a finite look-ahead of 0.5–2.0 seconds preserves non-streaming quality, consistent with prior monotonic and streaming sequence modeling approaches, and the diffusion decoder remains the primary latency source 23 ; 9 . Additional runtime, memory, and SSM-sensitivity analyses appear in Appendix A.1 . Contributions. (1) A diffusion-based TTS system with a fully SSM-only inference-time conditioning path spanning text, rhythm, and prosody under a fixed decoder. (2) A gated bidirectional Mamba fusion with AdaLN that improves long-range prosody stability and reduces drift on multi-sentence and out-of-distribution text. (3) Protocol- and capacity-matched baselines that isolate the architectural impact of removing inference-time attention. (4) A deployment-oriented analysis covering memory usage, throughput, SSM hyperparameter sensitivity, long-form behavior, and finite look-ahead streaming, demonstrating predictable linear-time characteristics. 2 Related Work TTS conditioning spans attention-based encoders, diffusion decoders, zero-shot systems, and recent state-space models. MVC examines how these paradigms affect efficiency, memory usage, and long-form stability. Attention-based TTS. Transformer-based pipelines such as Tacotron, Tacotron2, JETS, StyleTTS, and StyleTTS2 30 ; 25 ; 18 ; 17 ; 27 provide strong alignment and style modeling but rely on quadratic attention complexity. Even linear attention variants 29 ; 3 maintain global interactions that couple text, duration, and prosody, making streaming synthesis sensitive to memory usage. These limitations motivate conditioning stacks with linear-time behavior and bounded activations. Zero-shot and large-scale systems. NaturalSpeech 3 12 , CosyVoice 3 4 , and HiggsAudio-V2 2 leverage multi-hundred-thousand– to million-hour proprietary corpora, multilingual pipelines, and LLM-scale encoders integrating text understanding and expressive control. Their performance is primarily driven by scale rather than conditioning design and cannot be reproduced under academic budgets or open-data constraints. MVC instead evaluates conditioning architecture under a fixed mel–diffusion–vocoder pipeline and controlled data scale. A contextual comparison appears in Appendix F . SSMs and Mamba hybrids. Mamba introduces input-gated selective scans for linear-time modeling with bounded activations 6 . SSMs have been explored in speech enhancement, ASR, and hybrid TTS encoders 20 ; 11 ; 15 ; 33 . However, existing Mamba–TTS systems remain hybrid at inference, reintroducing attention or recurrence for duration and style modeling and limiting streaming stability. Runtime and finite look-ahead behavior under diffusion decoders are rarely analyzed. Positioning of MVC. MVC eliminates attention and recurrence across the entire inference-time conditioning stack for text, rhythm, and prosody, retaining a lightweight aligner only during training. It replaces concat-only bi-Mamba fusion 11 ; 33 with gated forward–backward fusion and AdaLN modulation. Protocol- and capacity-matched baselines isolate the architectural effects of removing inference-time attention and introducing gated AdaLN fusion (Table 12 , Appendix A.2 ). Figure 1: Overview of MambaVoiceCloning (MVC). The framework uses Bi-Mamba Text Encoders for phoneme modeling, a Temporal Bi-Mamba for rhythmic alignment, and an Expressive Mamba for prosodic control. A lightweight aligner (dotted box) provides phoneme–frame supervision only during training, ensuring an SSM-only encoder at inference. Conditioning features drive a diffusion decoder and vocoder for waveform synthesis. 3 Methodology MVC replaces all inference-time attention and recurrence with selective state-space models (SSMs) for text, rhythm/duration, and prosody. A lightweight attention-based aligner provides phoneme–frame supervision during training and is discarded at inference. This yields an SSM-only conditioning stack with linear-time scans and bounded activations. Unlike prior bi-Mamba encoders 11 ; 33 , MVC employs a gated bidirectional Mamba text encoder, a Temporal Bi-Mamba, and an Expressive Mamba with AdaLN conditioning. All Mamba blocks use a state dimension of 96, depthwise convolution kernel size 5, and gating temperature \tau=1.0 unless otherwise specified. High-level overview. Figure 1 summarizes MVC. From phonemized text and reference audio, MVC produces three conditioning streams: a gated Bi-Mamba text encoder, a Temporal Bi-Mamba for rhythm/duration, and an Expressive Mamba operating on mel spectrograms with AdaLN.