EntangleCodec: A Unified Discrete Audio Tokenizer via Semantic-Acoustic Entanglement
Hui Li Yangfan Gao Junlin Shang Changhao Jiang Tao Gui Qi Zhang Xuanjing Huang Fudan University {hui_li25}@m.fudan.edu.cn ∗ {\ast} Equal contribution · 2026
Read EntangleCodec: A Unified Discrete Audio Tokenizer via Semantic-Acoustic Entanglement by Hui Li Yangfan Gao Junlin Shang Changhao Jiang Tao Gui Qi Zhang Xuanjing Huang Fudan University {hui_li25}@m.fudan.edu.cn ∗ {\ast} Equal contribution (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2606-02739.txt.
Introduction
Li and co-authors unify audio tokenization for both understanding and generation. Audio tokenizers bridge continuous sound and audio language models. Reconstruction codecs keep acoustic fidelity but lack semantics. Semantic tokenizers split into separate streams, causing redundancy or misalignment. EntangleCodec learns caption-aligned representations before quantization. Captions carry more than transcripts. They encode content, speaker identity, emotion, prosody, and scene.
A single compact token stream results. A flow-matching diffusion decoder rebuilds high-quality audio from it. Speech, music, and general audio are all covered. Reconstruction rivals specialized codecs. Understanding beats all codec-based baselines, by up to 7.4 percent on MMAR. Text-to-speech and text-to-audio generation both work in the same frame. A 0.6 billion parameter model surpasses continuous-representation models over 13 billion parameters. At 8 billion parameters, new state-of-the-art results appear. Representation quality, the authors argue, rivals model scale.
Catalog record
| CREATOR | Hui Li Yangfan Gao Junlin Shang Changhao Jiang Tao Gui Qi Zhang Xuanjing Huang Fudan University {hui_li25}@m.fudan.edu.cn ∗ {\ast} Equal contribution |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 49 KB |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2606.02739
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
EntangleCodec: A Unified Discrete Audio Tokenizer via Semantic-Acoustic Entanglement Hui Li Yangfan Gao Junlin Shang Changhao Jiang Tao Gui Qi Zhang Xuanjing Huang Fudan University {hui_li25}@m.fudan.edu.cn ∗ {\ast} Equal contribution Abstract Audio tokenizers serve as the discrete interface between continuous audio and Audio Language Models (ALMs), but existing tokenizers often struggle to support both understanding and generation. Reconstruction-oriented codecs preserve acoustic fidelity but lack rich semantics, while semantic-aware tokenizers typically rely on separate semantic and acoustic streams, introducing redundancy or misalignment. We propose EntangleCodec , a unified discrete audio tokenizer that learns caption-aligned semantic-acoustic representations before quantization. By aligning audio with rich captions rather than ASR transcripts, EntangleCodec captures linguistic content, speaker identity, emotion, prosody, and acoustic scenes within a compact token stream. A flow-matching diffusion decoder further enables high-quality reconstruction across speech, music, and general audio. EntangleCodec achieves reconstruction quality competitive with specialized codecs, outperforms all codec-based baselines on audio understanding by up to +7.4% on MMAR, and supports both TTS and TTA generation in a unified framework. Furthermore, EntangleCodec-based audio language models demonstrate strong scaling behavior: even at 0.6B parameters, the model surpasses specialized continuous-representation LLMs with over 13B parameters across three benchmarks using 22 × \times fewer parameters; scaling to 8B further establishes new state-of-the-art results on MMAR, highlighting that representation quality is as critical as model scale in audio language modeling. Code and model weights are available at https://github.com/luckyerr/EntangleCodec . 1 Introduction 20 40 60 80 100 AudioBoxScore (music) AudioBoxScore (sound) UTMOS MMAU-mini MMAR TTS (WER \downarrow ) TTA (CLAP) EntangleCodec Xcodec2 Xcodec WavTokenizer Mimi SpeechTokenizer EnCodec DAC Figure 1: Comprehensive tokenizer comparison across seven dimensions (normalized 0–100; TTS WER inverted so higher = better).Reconstruction: AudioBoxScore (music/sound), UTMOS; Understanding: MMAU-mini, MMAR; Generation: TTS (WER \downarrow ), TTA (CLAP). The recent success of large language models has motivated growing interest in Audio Language Models (ALMs), spanning speech, music, and general audio understanding and generation. A central challenge is how to represent audio as discrete tokens that are both semantically informative and acoustically faithful. Existing audio tokenizers often favor one side of this trade-off: reconstruction-oriented codecs preserve fine acoustic detail but provide limited semantic information, while semantic-aware tokenizers introduce additional semantic streams that require fusion with acoustic representations. Moreover, semantic supervision is often limited to ASR transcripts, ignoring non-textual cues such as speaker identity, emotion, prosody, acoustic scenes, and musical structure. We propose EntangleCodec , a unified discrete audio tokenizer that learns caption-aligned semantic-acoustic representations before quantization. Instead of separating semantic and acoustic information into independent streams, EntangleCodec integrates them within a shared pre-quantization representation, reducing the need for late fusion. To enrich semantic supervision, we align audio with rich captions rather than ASR transcripts through contrastive learning, enabling the resulting tokens to capture linguistic content as well as speaker, affective, prosodic, and environmental cues. A flow-matching diffusion decoder then reconstructs high-quality audio from the discrete tokens, allowing the same tokenizer to support TTS, TTA, and audio QA without task-specific architectural changes. Key empirical findings. EntangleCodec achieves reconstruction quality comparable to specialized codecs despite its unified design (UTMOS: 3.96 vs. 4.02). For audio understanding, it outperforms all codec-based baselines by up to +7.4% on MMAR. Furthermore, a 0.6B EntangleCodec-based language model surpasses specialized continuous-representation models with over 13B parameters across MMAR, MMAU-mini, and MMAU, using 22 \times fewer parameters; scaling to 8B further establishes new state-of-the-art results on MMAR ( 42.6% ). These results highlight that tokenizer representation quality is as critical as model scale in audio language modeling. Our main contributions are: • EntangleCodec , a unified codec-based tokenizer that learns semantic-acoustic entangled representations before quantization, reducing the need for dual-stream separation and late fusion while maintaining competitive reconstruction quality. • Caption-aligned discrete representation learning , which goes beyond ASR-only supervision by contrastively aligning audio representations with rich captions covering linguistic content, speaker identity, emotion, prosody, and acoustic scenes, leading to more expressive discrete audio tokens. • A unified audio language modeling framework for TTS, TTA, and audio QA, requiring no task-specific tokenizer or architectural modification and demonstrating the broad applicability of EntangleCodec across both generation and understanding tasks. Figure 2: Overview of EntangleCodec. Compared with single-encoder acoustic codecs and dual-encoder semantic-acoustic codecs, EntangleCodec learns caption-aligned unified tokens that support both reconstruction and downstream audio-language tasks. The right panel shows the two-stage training strategy: joint codec learning with rich captions, followed by decoder refinement. 2 Related Work 2.1 Audio Codec and Tokenization Neural audio codecs provide the foundation for discrete audio language modeling. Early codecs such as SoundStream Zeghidour et al. 2021 and EnCodec Défossez et al. 2023 introduced residual vector quantization (RVQ) for high-fidelity reconstruction at low bitrates, while DAC Kumar et al. 2023 further improved codebook utilization and reconstruction quality. Recent tokenizers focus more explicitly on language-model compatibility: WavTokenizer Ji et al. 2025 reduces token rates while preserving perceptual quality, Mimi Défossez et al. 2024 supports streaming-friendly tokenization, and SpeechTokenizer Zhang et al. 2024 , XCodec Ye et al. 2025a , and XCodec2 Ye et al. 2025b introduce semantic-aware designs for speech generation and audio modeling. Most existing codecs follow one of two routes. Reconstruction-oriented codecs use a single acoustic encoder but provide limited semantic information for downstream reasoning. Semantic-aware codecs often introduce separate semantic and acoustic streams, which improves task relevance but requires additional fusion between independently learned representations. In contrast, EntangleCodec learns a caption-aligned semantic-acoustic representation before quantization, aiming to preserve both reconstruction fidelity and semantic richness within a unified discrete token stream. 2.2 Audio-Language Models for Understanding Audio-language models extend LLMs to speech, music, and general audio understanding. LTU Gong et al. 2024 and Audio Flamingo Kong et al. 2024 connect audio encoders with language models for audio question answering, while MU-LLaMA Liu et al. 2024 and GAMA Ghosh et al. 2024 target music understanding and general audio reasoning. SALMONN Tang et al. 2024 combines speech and audio encoders to handle diverse audio tasks, and Qwen2-Audio Chu et al. 2024 further improves performance through stronger audio encoders and language model backbones. These systems typically rely on continuous audio representations produced by dedicated encoders. Such representations are effective for understanding, but they are not naturally suited to autoregressive audio generation. EntangleCodec instead provides discrete audio tokens that can be consumed by LLMs, enabling audio understanding and generation to share the same token interface. 2.3 Text-Conditioned Audio Generation Text-conditioned audio generation includes text-to-speech (TTS) and text-to-audio (TTA). Autoregressive models such as AudioLM Borsos et al. 2023 and VALL-E Wang et al. 2023 demonstrate the effectiveness of predicting discrete audio tokens for speech generation, while TTA models often rely on text-audio alignment signals such as CLAP Elizalde et al. 2023 to synthesize audio from natural language descriptions. Strong TTS systems such as NaturalSpeech3 Ju et al. 2024 and CosyVoice Du et al. 2024 achieve high-quality speech synthesis, whereas general TTA systems focus on environmental sound, music, and acoustic scenes. Despite this progress, TTS and TTA are usually treated with separate modeling pipelines, tokenizers, or conditioning mechanisms. EntangleCodec provides a unified discrete representation for both speech and general audio, allowing TTS and TTA to be formulated as text-conditioned token prediction tasks within the same audio language modeling framework. 3 Methods 3.1 Overview EntangleCodec aims to build