MagiCodec: Simple Masked Gaussian-Injected Codec for High-Fidelity Reconstruction and Generation
Yakun Song, Jiawei Chen, Xiaobin Zhuang, Chenpeng Du, Ziyang Ma, Jian Wu, Jian Cong, Dongya Jia, Zhuo Chen, Yuping Wang, Yuxuan Wang, Xie Chen · 2025
Read MagiCodec: Simple Masked Gaussian-Injected Codec for High-Fidelity Reconstruction and Generation by Yakun Song, Jiawei Chen, Xiaobin Zhuang, Chenpeng Du, Ziyang Ma, Jian Wu, Jian Cong, Dongya Jia, Zhuo Chen, Yuping Wang, Yuxuan Wang, Xie Chen (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2506-00385.txt.
Introduction
Yakun Song and a team of co-authors present a 2025 paper on MagiCodec, a tool that turns raw sound into short codes for AI models. Such codes are the base for modern sound generators. Most past tools were tuned only for clean rebuild of the sound, which can leave codes that models find hard to use.
MagiCodec uses a single layer design that can run as sound streams in. Training adds mild noise and rules that push the codes to carry more meaning. Tests show cleaner rebuilds than top rivals and better scores on later tasks. Its codes also spread in a pattern like word use in natural language. Code and trained models are shared in public.
Catalog record
| CREATOR | Yakun Song, Jiawei Chen, Xiaobin Zhuang, Chenpeng Du, Ziyang Ma, Jian Wu, Jian Cong, Dongya Jia, Zhuo Chen, Yuping Wang, Yuxuan Wang, Xie Chen |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 52 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2506-00385
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
MagiCodec: Simple Masked Gaussian-Injected Codec for High-Fidelity Reconstruction and Generation Yakun Song, Jiawei Chen, Xiaobin Zhuang, Chenpeng Du, Ziyang Ma, Jian Wu, Jian Cong, Dongya Jia, Zhuo Chen, Yuping Wang, Yuxuan Wang, Xie Chen Abstract Neural audio codecs have made significant strides in efficiently mapping raw audio waveforms into discrete token representations, which are foundational for contemporary audio generative models. However, most existing codecs are optimized primarily for reconstruction quality, often at the expense of the downstream modelability of the encoded tokens. Motivated by the need to overcome this bottleneck, we introduce MagiCodec , a novel single-layer, streaming Transformer-based audio codec. MagiCodec is designed with a multistage training pipeline that incorporates Gaussian noise injection and latent regularization, explicitly targeting the enhancement of semantic expressiveness in the generated codes while preserving high reconstruction fidelity. We analytically derive the effect of noise injection in the frequency domain, demonstrating its efficacy in attenuating high-frequency components and fostering robust tokenization. Extensive experimental evaluations show that MagiCodec surpasses state-of-the-art codecs in both reconstruction quality and downstream tasks. Notably, the tokens produced by MagiCodec exhibit Zipf-like distributions, as observed in natural languages, thereby improving compatibility with language-model-based generative architectures. The code and pre-trained models are available at https://github.com/Ereboas/MagiCodec . 1 Introduction Recently, large language models (LLMs) have made transformative progress in natural language processing ( Achiam et al. 2023 ; Liu et al. 2024a ; Yang et al. 2024 ) and audio generation ( Anastassiou et al. 2024 ; Borsos et al. 2023 ) , exhibiting a strong capability to model long sequences of discrete tokens. Recent studies have widely adopted audio codec models, such as SoundStream ( Zeghidour et al. 2021 ) and EnCodec ( Défossez et al. 2022 ) , as audio tokenizers within audio language modeling frameworks. However, these methods continue to focus on fidelity and computational efficiency in reconstruction as their primary objectives, and often overlook the semantic modelability of discrete representations. As research in representation learning progresses, many researchers have increasingly recognized the optimization dilemma between generative capacity and reconstruction quality. Specifically, improving reconstruction quality can compromise generation performance and require larger models and more training resources. Conversely, limiting reconstruction capabilities can reduce the upper limit of generation quality ( Yao et al. 2025 ) . Therefore, overemphasis on reconstruction objectives tends to substantially complicate the training of generative models. To enhance generative performance, several studies have introduced explicit semantic supervision to strengthen the encoding of low-frequency semantic content. For example, SemantiCodec ( Liu et al. 2024b ) employs a dual-encoder architecture, combining high-level semantic features extracted by a self-supervised, pretrained AudioMAE ( Huang et al. 2022 ) with an acoustic encoder that captures fine details, and utilizes a diffusion-based decoder to achieve high-quality reconstruction at ultra-low bitrates. X-Codec ( Ye et al. 2025 ) integrates self-supervised semantic representations directly into the quantization process, thereby enhancing the generative capability of audio language models. Although these strategies yield improved semantic retention, they also result in the loss of high-frequency texture details and the introduction of minor artifacts. In addition, they depend on external models, such as diffusion models ( Anastassiou et al. 2024 ) , and thus fail to provide an efficient or fundamental solution. The question of how to achieve high-fidelity reconstruction and improved modelability of discrete codes through intrinsic frequency-domain constraints or regularization mechanisms, without resorting to additional annotation or complex pretraining, remains a central issue in contemporary audio codec research. Another major challenge for neural audio codecs arises from their underlying architecture. A typical neural codec can be divided into three components: an encoder that projects the raw waveform into a continuous latent vector space; a vector quantization (VQ) module that discretizes the latent vectors into tokens drawn from a finite codebook ( Van Den Oord et al. 2017 ) ; and a decoder that reconstructs the audio signal from these discrete tokens ( Wu et al. 2024 ) . The VQ module is the central element in achieving discretization. During training, straight-through gradient estimation is typically employed; although this simplifies backpropagation, it also amplifies the error introduced by quantization. Whether using vanilla vector quantization or residual vector quantization (RVQ) ( Zeghidour et al. 2021 ; Défossez et al. 2022 ) , training commonly suffers from codebook collapse, in which many codebook entries go underutilized or only a small subset of vectors is frequently activated. This phenomenon leads to poor coverage of the encoding space and reduced token diversity, thereby undermining the expressive capacity and efficiency of downstream generative models. In response to these challenges, we propose MagiCodec, a simple Ma sked G aussian I njected Codec for high-fidelity reconstruction and generation. Largely inspired by TS3Codec ( Wu et al. 2024 ) , MagiCodec also adopts an efficient, single-layer streaming codec built on a Transformer backbone. Through a multi-stage training procedure with clearly articulated motivations, it implicitly suppresses high-frequency noise and strengthens low-frequency structure modeling, thereby achieving joint optimization of reconstruction quality and downstream generative performance. Our core idea is to dispense with external labels and rely solely on intrinsic Gaussian noise injection, so that the codec learns to allocate modeling capacity appropriately across different frequency bands, achieving both high-fidelity reconstruction and efficient, modelable discrete codes at constrained bitrates. In addition, we decompose traditional codec training into three phases: autoencoder, vector quantization, and vocoder. Training in the first two stages involves only the generator, thereby preventing audio phase information from being incorporated into the intermediate representations. This staged training approach effectively avoids the issue of codebook collapse and maximizes codebook utilization. Our main contributions can be summarized as follows: • Theoretical contributions. We analytically derive the frequency-domain effect of Gaussian noise injection, showing that it is equivalent to mixing the original signal with a low-pass–filtered version, thereby implicitly regularizing high-frequency components. • Methodological contributions. We design a multi-stage training framework that effectively mitigates codebook collapse and enhances overall token performance, incorporating Gaussian noise injection and latent regularization. These techniques require no external models or labels, yet encourage the codec to learn low-frequency semantic representations and avoid overfitting to high-frequency noise. • Experimental contributions. We conduct comprehensive evaluations, demonstrating that MagiCodec achieves state-of-the-art reconstruction quality across multiple bitrates and metrics. In downstream tasks such as text-to-speech, automatic speech recognition, and semantic information extraction, MagiCodec also significantly outperforms baseline methods, confirming its strong modelability. Analysis of the code distribution further reveals its close adherence to the Zipf distribution observed in natural language, which facilitates downstream model training. 2 Related Work Neural audio codecs aim to encode continuous audio signals into discrete latent representations, which is a form of discrete audio tokenization. In recent years, neural audio codecs have become a research focus to achieve high-quality audio reconstruction at low bitrates ( Kumar et al. 2023 ; Ai et al. 2024 ) . These methods leverage vector quantization to learn compact codebooks that effectively compress audio information while preserving essential details. To further improve reconstruction quality, various techniques have been proposed. PromptCodec ( Pan et al. 2024 ) introduces additional input prompts to enrich the latent space representation, improving the model’s adaptability to complex audio content. Moreover, DAC ( Kumar et al. 2023 ) combines quantizer dropout with multi-scale STFT discriminators, effectively