T-Mimi: A Transformer-based Mimi Decoder for Real-Time On-Phone TTS

Haibin Wu, Bach Viet Do, Naveen Suda, Julian Chan, Madhavan C R, Gene-Ping Yang Yi-Chiao Wu, Naoyuki Kanda, Yossef Adi, Xin Lei, Yue Liu, Florian Metze, Yuzong Liu · 2026

Read T-Mimi: A Transformer-based Mimi Decoder for Real-Time On-Phone TTS by Haibin Wu, Bach Viet Do, Naveen Suda, Julian Chan, Madhavan C R, Gene-Ping Yang Yi-Chiao Wu, Naoyuki Kanda, Yossef Adi, Xin Lei, Yue Liu, Florian Metze, Yuzong Liu (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2601-20094.txt.

Introduction

Haibin Wu and a large team of colleagues build the decoder behind a phone-based speech synthesiser. Neural audio codecs compress speech into acoustic features; the Mimi codec streams them for real-time text-to-speech. A decoder must turn those features back into natural audio on the phone itself. Their system, T-Mimi, is a transformer-based decoder for the Mimi codec.

It is designed for real-time, on-device use. Phones impose tight limits on latency and compute. A transformer decoder that fits those limits can give apps natural voices without the cloud. On-device synthesis also keeps voice data private. The work pushes high-quality speech generation onto everyday hardware.

Catalog record

CREATORHaibin Wu, Bach Viet Do, Naveen Suda, Julian Chan, Madhavan C R, Gene-Ping Yang Yi-Chiao Wu, Naoyuki Kanda, Yossef Adi, Xin Lei, Yue Liu, Florian Metze, Yuzong Liu
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE19 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2601-20094

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

T-Mimi: A Transformer-based Mimi Decoder for Real-Time On-Phone TTS Haibin Wu, Bach Viet Do, Naveen Suda, Julian Chan, Madhavan C R, Gene-Ping Yang Yi-Chiao Wu, Naoyuki Kanda, Yossef Adi, Xin Lei, Yue Liu, Florian Metze, Yuzong Liu Abstract Neural audio codecs provide promising acoustic features for speech synthesis, with representative streaming codecs like Mimi providing high-quality acoustic features for real-time Text-to-Speech (TTS) applications. However, Mimi’s decoder, which employs a hybrid transformer and convolution architecture, introduces significant latency bottlenecks on edge devices due to the the compute intensive nature of deconvolution layers which are not friendly for mobile-CPUs, such as the most representative framework XNNPACK [ 5 ] . This paper introduces T-Mimi, a novel modification of the Mimi codec decoder that replaces its convolutional components with a purely transformer-based decoder, inspired by the TS3-Codec architecture. This change dramatically reduces on-device TTS latency from 42.1ms to just 4.4ms. Furthermore, we conduct quantization aware training and derive a crucial finding: the final two transformer layers and the concluding linear layers of the decoder, which are close to the waveform, are highly sensitive to quantization and must be preserved at full precision to maintain audio quality. † † address: Meta, USA 1 Introduction Recent advances in neural audio codecs [ 6 , 1 , 14 , 9 ] have fundamentally transformed the intermediate representations used in modern text-to-speech (TTS) systems. While mel-spectrograms have long served as the de facto standard acoustic feature, representations derived from models such as EnCodec [ 3 ] , DAC [ 8 ] and Mimi [ 4 ] have emerged as a superior alternative. These modern codec features, whether in the form of discrete tokens or continuous vectors, offer exceptional compression ratios while preserving perceptually vital information, sometimes leading to higher audio quality than traditional mel-spectrogram-based approaches [ 9 ] . This paradigm shift effectively redefines the TTS pipeline: the new synthesis model now predicts codec features instead of mel-spectrograms, and the neural codec’s own high-fidelity decoder replaces the traditional vocoder, serving as the final stage of generating waveforms from acoustic features. Among the many NACs, the Mimi codec stands out as an excellent choice for building streaming TTS systems. Mimi offers several key advantages: First, it skillfully disentangles the audio signal into separate “semantic” and “acoustic” feature streams. This separation allows a TTS model to exert fine-grained control over the synthesized content (what is said) and the speech style (how it is said). Second, its decoder can reconstruct high-quality, natural-sounding audio from these tokens under a really low frame rate (12.5HZ). Most importantly, Mimi supports streaming, enabling real-time encoding and decoding of audio segments. The streaming decoder is crucial for streaming TTS applications that require immediate responses. However, a significant challenge emerges when deploying advanced models like Mimi onto mobile or edge devices. Mimi’s decoder relies on de-convolution (or transposed convolution) layers for upsampling to reconstruct the final audio waveform. While convolutions are highly parameter-efficient due to weight sharing, they can still be computationally intensive 1 1 1 With the same parameter amount, the transformer codec requires 1/7 computation compared with CNN based codec [ 15 ] . . While these layers can be optimized efficient in theory, they are often computationally inefficient in mobile inference frameworks like XNNPACK 2 2 2 https://github.com/google/XNNPACK , leading to significant latency. These frameworks often have better optimization for transformers than de-convolution layers. This latency issue becomes the primary bottleneck for achieving a smooth, real-time TTS experience on phones. TS3-Codec [ 15 ] , being purely Transformer-based and convolution-free, establishes a new benchmark for efficiency. It not only greatly outperforms CNN-based NACs when matched for parameter size, but also delivers comparable performance at a fraction of the computational cost—requiring a mere 13% of the resources. Inspired by TS3-Codec, this paper introduces T-Mimi, a modified version of the Mimi codec featuring a Transformer-only decoder architecture. By replacing the slow de-convolution layers with a mobile-friendly architecture composed of Transformers and linear layers, we directly address the on-device latency bottleneck. We also applied quantization aware training (QAT) to further reduce the model’s storage footprint and computational requirements, making it better suited for resource-constrained devices. Our primary contributions are as follows: (1). We designed T-Mimi, drastically reducing on-device TTS decoding latency from 42.1ms to just 4.4ms, enabling true real-time TTS performance. A latency of 42.1 ms to generate a single 80ms audio frame is a significant bottleneck to achieve real-time processing. When factoring in the processing time required for other essential TTS components, such as the front-end and acoustic models, along with potential resource contention on the device, this delay makes achieving a seamless, real-time audio experience highly improbable. (2). Through quantization aware training, we discovered a crucial rule: the closer a layer is to the final waveform, the worse the audio quality becomes after it is quantized. Consequently, we found that the final layers of the decoder, specifically the last two Transformer layers and the two linear layers important for audio generation. They should be preserved in full precision to maintain audio fidelity. We take Mimi codec as a representative case study. Our methodology is not limited to Mimi codec, and provides a generalizable framework for optimizing other convolution-based neural audio codecs for efficient on-device deployment. 2 Background 2.1 Codec feature as a good medium for TTS A variety of state-of-the-art TTS models demonstrate the promise of neural audio codec features as acoustic representations. This is exemplified by a variety of state-of-the-art models, such as VALLE [ 11 ] , VoiceStar [ 10 ] , which leverage Encodec features, AudioLM [ 2 ] and SpearTTS [ 7 ] , which utilize SoundStream tokens [ 17 ] , and Spark-TTS [ 12 ] , builts upon BiCodec features. To generate these acoustic features from text or semantic tokens, these systems employ a range of acoustic modeling techniques, including diffusion models [ 13 ] , Tacotron-based architectures [ 9 ] , and depth-transformer-based methods [ 16 ] . Ultimately, regardless of the specific acoustic models and codec features, every system converges on the codec’s decoder as the final, essential step to transform the predicted codec features into the output waveform. Our focus in this paper is to develop an on-phone streaming and low-latency codec decoder. 2.2 On-phone TTS requirements On-phone streaming TTS systems demand a delicate balance of competing requirements: low latency on constrained hardware, a streaming-capable architecture, and high-fidelity audio reconstruction. Additionally, a minimal storage footprint is essential, making quantization a necessity. To this end, quantization aware training is employed to preserve audio quality during model compression. The Mimi codec is a compelling foundation as it already provides a streaming architecture and high-quality reconstruction, making it an ideal candidate for further on-device optimization. 2.3 Quantization aware training Quantization is an important technique for optimizing the efficiency of deep learning models, enabling significant reductions in both memory footprint and inference latency. To mitigate the performance degradation typically associated with post-training quantization, we employ quantization aware training, which simulates the effects of low-precision arithmetic during the training process. 3 Method Figure 1 : Model architectures for the original Mimi decoder (upper) and the T-Mimi decoder (bottom). 3.1 Architecture The model architectures of the original Mimi decoder and the

Read the complete text