OmniCodec: Low Frame Rate Universal Audio Codec with Semantic–Acoustic Disentanglement

Jingbin, Haoyu, Dake, Qirui, Wenhao, Huakang, Guobin, Hanke, Chengyou, Pengyuan, Chuan, Qiang, Lei · 2026

Read OmniCodec: Low Frame Rate Universal Audio Codec with Semantic–Acoustic Disentanglement by Jingbin, Haoyu, Dake, Qirui, Wenhao, Huakang, Guobin, Hanke, Chengyou, Pengyuan, Chuan, Qiang, Lei (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2603-20638.txt.

Introduction

Hu and co-authors present OmniCodec, a neural audio codec that works across speech, music, and sound. Most codecs chase clean rebuilds of speech alone. High rebuild quality does not always give useful meaning for later tasks. OmniCodec uses stacked codebooks at a low frame rate. It splits semantic content from acoustic detail with help from a pretrained encoder. A self guidance trick lifts codebook use. At the same bitrate it beat the Mimi codec on rebuild quality.

Catalog record

CREATORJingbin, Haoyu, Dake, Qirui, Wenhao, Huakang, Guobin, Hanke, Chengyou, Pengyuan, Chuan, Qiang, Lei
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE24975
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2603.20638

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

OmniCodec: Low Frame Rate Universal Audio Codec with Semantic–Acoustic Disentanglement Jingbin, Haoyu, Dake, Qirui, Wenhao, Huakang, Guobin, Hanke, Chengyou, Pengyuan, Chuan, Qiang, Lei Abstract Large Language Models (LLMs) have advanced audio generation through discrete representation learning. However, most existing neural codecs focus on speech and emphasize reconstruction fidelity, overlooking unified low frame rate modeling across diverse audio domains, including speech, music, and general sound. Moreover, high reconstruction quality does not necessarily yield semantically informative representations, limiting effectiveness in downstream generation tasks. We propose OmniCodec, a universal neural audio codec tailored for low frame rate. It adopts a hierarchical multi-codebook design with semantic–acoustic decoupling by leveraging the audio encoder of the pre-trained understanding model, along with a self-guidance strategy to improve codebook utilization and reconstruction. Compared with the Mimi codec, experiments show that OmniCodec achieves outstanding performance at the same bitrate, delivering superior reconstruction quality while also providing more semantically informative representations that benefit downstream generation tasks. Our model and code will be open-sourced 1 1 1 https://github.com/ASLP-lab/OmniCodec . Our demo page is available 2 2 2 https://hujingbin1.github.io/OmniCodec-Demo-Page/ . Hu Zhang Guo Zhan Li Chen Ma Xie Wang Xie Xie Zhang Xie keywords general audio domains codec, low frame rate, semantic, acoustic, decoupling, audio generation † † address: 1 Audio, Speech and Language Processing Group (ASLP@NPU), Northwestern Polytechnical University 2 Shanghai Lingguang Zhaxian Technology † † email: [email protected], [email protected] 1 Introduction In recent years, Large Language Models (LLMs) for audio generation have advanced rapidly [ 2 ] , yet the core challenge remains the design of effective representations. Current approaches are generally based on discrete representations, which naturally align with the token-based space of language models. This alignment leads to more stable generation and better scalability for large-scale downstream generative tasks. Therefore, how to design a better Codec model as an Audio Tokenizer has become increasingly important. In previous studies, the Codec model was mainly designed for audio compression and reconstruction tasks, such as SoundStream [ 36 ] , DAC [ 18 ] , and Encodec [ 5 ] , which have achieved high-quality reconstruction. However, due to the requirement for high fidelity, they usually need extremely high bit rates and frame rates, maintain a multi-codebook structure, and have a significant difference from semantic representations. All these factors make them unsuitable as targets for LLMs to predict. With the advancement of technology, the single-codebook structure was highly valued due to its natural compatibility with language model modeling, such as BigCodec [ 31 ] , SingleCodec [ 19 ] , TS3-Codec [ 29 ] , and FocalCodec [ 7 ] . However, the information contained in the audio is very rich. A single-layer codebook is insufficient to represent the audio signal at low frame rate, and the high frame rate structure is not conducive to the data scaling training of LLM. Moreover, a Codec trained only based on reconstruction loss also has the common problem that semantic information is not structurally modeled. Figure 1: Overview of OmniCodec framework. To address the issue of the absence of semantic information representation, some studies have attempted to use self-supervised learning models for distillation, facilitating the modeling of downstream tasks, such as SpeechTokenizer [ 40 ] , Mimi codec [ 6 ] , and Qwen-TTS-Tokenizer-12Hz [ 14 ] . Some additional work attempts to use the representations extracted by the self-supervised learning model as the input for the semantic branch of the dual-stream Encoder, such as BiCodec [ 28 ] , X-Codec [ 34 ] , X-Codec2 [ 35 ] , DualCodec [ 20 ] , XY-Tokenizer [ 12 ] , FireRedTTS-2-Tokenizer [ 30 ] , and SAC [ 3 ] . Some radical proposals pursue extreme semanticity. They use the Automatic Speech Recognition (ASR) supervised loss for training the Codec model, such as S3Tokenizer [ 9 ] and GLM-4-Voice-Tokenizer [ 38 ] . These advancements have demonstrated the effectiveness of incorporating semantic information in enhancing the modeling capabilities of downstream LLMs. However, due to the limited capacity of a single codebook and the difficulty in achieving high-quality reconstruction at a low frame rate. These studies have shown that low frame rate and multi-codebook structures with semantic supervision are increasingly becoming the current development trend. At the same time, general audio domains generation based on LLMs, which includes speech, music, and general sound, is becoming a strong demand. This demand has led to the development of discrete representations in the general audio domains. Early works, such as WavTokenizer [ 15 ] and SemantiCodec [ 22 ] , attempt to unify the modeling of audio in different domains. In recent work, UniCodec [ 16 ] attempts to enhance its semantic modeling capabilities through self-supervised learning methods. AUV [ 4 ] has further advanced through cross-domain self-supervised feature distillation. Recently, there have also been some works to introduce LLMs at the bottleneck for speech recognition tasks and audio caption tasks, thereby introducing semantic supervision, such as the MiMo-Audio-Tokenizer [ 39 ] and the MOSS-Audio-Tokenizer [ 11 ] . By further scaling the model parameters and dataset size, they achieved extremely high reconstruction quality in general audio domains and low frame rate scenarios while retaining sufficient semantic information. Although recent works have demonstrated the effectiveness of scaling in improving tokenizer performance, there remains a significant gap between academic research and industry resources. This gap makes it challenging to explore effective structural innovations. Additionally, large-scale tokenizers are not ideal for real-time speech interaction in streaming scenarios. With the further development of audio understanding models, the audio encoder used as the audio input for understanding models has been trained on vast datasets covering speech, music, and general sound, leading to robust semantic feature representations through supervised learning. This inspires us to explore using the audio encoder from understanding models as the semantic branch input, aiming to obtain general audio domain representations with semantic properties. By leveraging the data advantages from an open-source understanding model, we aim to achieve better semantic representations using the Qwen3-Omni-AuT-Encoder [ 32 ] . To adapt to the trend of low frame rate, multiple codebooks, general audio domains, and semantic decoupling, and also to integrate the audio encoder capabilities of the understanding model, we propose OmniCodec . We present the first demonstration that an audio encoder trained using supervised learning from understanding models can replace unsupervised models like WavLM [ 1 ] for use as semantic supervision in Codec models. And we have achieved the all-domain decoupling of speech, music, and general sound into semantic and acoustic representations. Moreover, we design a self-guidance mechanism [ 21 ] to improve codebook utilization and stabilize training, leading to improved reconstruction quality. Furthermore, our model achieves two frame rates of 12.5 Hz and 6.25 Hz, and is designed with a purely causal receptive field, supporting fully streaming and fast inference. Experiments have proved that our reconstruction indicators under the same bit rate condition are significantly superior to other competitive models. Moreover, in the downstream task, compared with the Mimi codec, we also achieved better results in the music and general sound domains. To facilitate future research, we open-source all models and code and provide a public demo showcasing generation examples across multiple audio domains. 2 Method Table 1: Reconstruction evaluation

Read the complete text