Speech Token Prediction via Compressed-to-fine Language Modeling

Wenrui Liu, Qian Chen, Wen Wang, Yafeng Chen, Jin Xu, Zhifang Guo, Guanrou Yang, Weiqin Li, Xiaoda Yang, Tao Jin, Minghui Fang, Jialong Zuo, Bai Jionghao, Zemin Liu Zhejiang University · 2025

Read Speech Token Prediction via Compressed-to-fine Language Modeling by Wenrui Liu, Qian Chen, Wen Wang, Yafeng Chen, Jin Xu, Zhifang Guo, Guanrou Yang, Weiqin Li, Xiaoda Yang, Tao Jin, Minghui Fang, Jialong Zuo, Bai Jionghao, Zemin Liu Zhejiang University (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2505-24496.txt.

Introduction

Wenrui Liu and a team of co-authors wrote a 2025 paper on making speech generation models faster by respecting how speech tokens depend on nearby tokens. Neural audio codecs turn speech into long token sequences so language models can generate audio. Those long sequences strain the models that must predict them.

The authors observe a short range dependency: thanks to the steady alignment between text and speech, each token depends mostly on its local context. Distant tokens add little and often repeat information. Their compressed to fine approach keeps the prompt and local tokens in full, while compressing the far context. Prediction then works from coarse to fine detail. The scheme targets the long context bottleneck in codec based text to speech systems.

Catalog record

CREATORWenrui Liu, Qian Chen, Wen Wang, Yafeng Chen, Jin Xu, Zhifang Guo, Guanrou Yang, Weiqin Li, Xiaoda Yang, Tao Jin, Minghui Fang, Jialong Zuo, Bai Jionghao, Zemin Liu Zhejiang University
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE44 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2505-24496

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Speech Token Prediction via Compressed-to-fine Language Modeling Wenrui Liu, Qian Chen, Wen Wang, Yafeng Chen, Jin Xu, Zhifang Guo, Guanrou Yang, Weiqin Li, Xiaoda Yang, Tao Jin, Minghui Fang, Jialong Zuo, Bai Jionghao, Zemin Liu Zhejiang University Abstract . Neural audio codecs, used as speech tokenizers, have demonstrated remarkable potential in the field of speech generation. However, to ensure high-fidelity audio reconstruction, neural audio codecs typically encode audio into long sequences of speech tokens, posing a significant challenge for downstream language models in long-context modeling. We observe that speech token sequences exhibit short-range dependency: due to the monotonic alignment between text and speech in text-to-speech (TTS) tasks, the prediction of the current token primarily relies on its local context, while long-range tokens contribute less to the current token prediction and often contain redundant information. Inspired by this observation, we propose a compressed-to-fine language modeling approach to address the challenge of long sequence speech tokens within neural codec language models: (1) Fine-grained Initial and Short-range Information : Our approach retains the prompt and local tokens during prediction to ensure text alignment and the integrity of paralinguistic information; (2) Compressed Long-range Context : Our approach compresses long-range token spans into compact representations to reduce redundant information while preserving essential semantics. Extensive experiments on various neural audio codecs and downstream language models validate the effectiveness and generalizability of the proposed approach, highlighting the importance of token compression in improving speech generation within neural codec language models. The demo of audio samples will be available at https://anonymous.4open.science/r/SpeechTokenPredictionViaCompressedToFinedLM . CCS: Computing methodologies CCS: Computing methodologies Artificial intelligence CCS: Computing methodologies Natural language processing CCS: Computing methodologies Natural language generation 1. Introduction Recent breakthroughs in neural codec language models have significantly advanced the field of speech generation, enabling the generation of more natural and high-quality speech in zero-shot TTS task ( Chen et al., 2025b ; Borsos et al., 2023 ; Wang et al., 2024 ; Nishimura et al., 2024 ) and voice-interactive scenarios ( Zhang et al., 2023a ; Défossez et al., 2024 ; Fu et al., 2025 ; Chen et al., 2025a ; Li et al., 2025 ) . Neural codec language models translate input text into a sequence of speech tokens, which are then decoded into the target speech waveform by neural audio codecs ( Défossez et al., 2022 ; Zeghidour et al., 2021 ; Hsu et al., 2021 ; Xin et al., 2024 ; Zhang et al., 2023b ; Ji et al., 2024 ) . Existing neural codec language models are built upon diverse architectures, including autoregressive decoders ( Casanova et al., 2024 ; Ji et al., 2024 ) , non-autoregressive decoders ( Wang et al., 2024 ) , and hybrid transformers ( Chen et al., 2025b ; Song et al., 2024 ; Han et al., 2024 ; Chen et al., 2024a ; Nishimura et al., 2024 ) . To ensure high-fidelity audio reconstruction, neural audio codecs often discretize speech waveform into long sequences of speech tokens. Although current research trends favor using neural audio codecs with lower frame rates ( Ren et al., 2024 ; Xin et al., 2024 ; Ji et al., 2024 ) and fewer codebooks ( Yang et al., 2023 ; Nishimura et al., 2024 ) , encoding a speech segment of 7-8 seconds still requires hundreds or even thousands of speech tokens. The excessive length of the speech token sequences presents challenges to long-context modeling, leading to the following critical issues: Sparsity undermines local alignment. To convey the same semantic information, speech token sequences are typically much longer than text token sequences ( Wang et al., 2025 ) , resulting in both semantic information sparsity and attention sparsity . ( Kavaki and Mandel, 2025 ) suggests that audio and speech exhibit information sparsity and that long-term dependency, which can be captured by transformer models, may not be critical for audio applications: attention between distant time frames may be deemed unnecessary ( Kim et al., 2020a ) . Furthermore, attention sparsity is widely observed in transformer-based language models, where more tokens in long sequences dilute the attention distribution ( Shi et al., 2023 ; Tikochinski et al., 2025 ; Lu et al., 2025 ) . These two forms of sparsity converge in neural codec language models: (1) semantic information sparsity arises from the imbalance between text and speech token lengths; (2) attention sparsity stems from lengthy speech token sequences. The sparsity distracts attention weights between speech tokens and corresponding text tokens, and hinders the language model to capture local fine-grained alignment between text and speech. Redundancy introduces semantic noise . Linguistic research found that acoustic redundancy naturally exists in speech and is closely related to duration and prosody ( Aylett and Turk, 2006 ; Jurafsky et al., 2008 ; Malisz et al., 2018 ) . ( Dieleman et al., 2021 ; Ren et al., 2024 ; Sicherman and Adi, 2023 ) demonstrate that this redundancy persists in discrete representations derived from speech signals, manifesting as repetitive timbres ( Ren et al., 2024 ) , silences ( Dieleman et al., 2021 ) , inconsistent units representing the same phoneme ( Sicherman and Adi, 2023 ) and so on. Such redundant information is irrelevant to the current text, contributes less to the prediction of the current speech token and may even pose semantic noise, hence reducing the robustness of the neural codec language model ( Sicherman and Adi, 2023 ) . The mapping from text to speech inherently relies on local alignment between text and speech tokens, as exemplified by the monotonic alignment between phonemes and acoustic features ( Kim et al., 2020b ; Shih et al., 2021 ) . However, as explained above, sparsity and redundancy in long sequences of speech tokens disrupt the alignment process, leading to reduced prediction accuracy of speech tokens and finally degraded TTS performance. To address these challenges in neural codec language models, two primary approaches have been explored: (1) Ultra high-compression-rate speech tokenizers ( Cho et al., 2024 ; Baade et al., 2024 ) encode speech into syllable-like units, dynamically reducing token rates from 50Hz ( Hsu et al., 2021 ) to 4Hz-5Hz to preserve the main semantic information. However, high compression at the level of neural audio codecs risks losing acoustic details such as prosody and timbre characteristics, which are essential for high-quality speech reconstruction and generation. (2) Language modeling for robust speech token generation , including group codec modeling ( Chen et al., 2024a ) and forced alignment between text-speech tokens ( Han et al., 2024 ; Song et al., 2024 ) . UniAudio ( Yang et al., 2024 ) and Mars6 ( Baas et al., 2025 ) employ the local-global-local modeling approach, which compresses and recovers speech tokens at the level of language models. However, these methods depend on particular architectures to mitigate sparsity, but redundancy remains inadequately addressed. To address both sparsity and redundancy challenges and overcome limitations of prior approaches, in this paper, we propose an innovative and general solution for long-context modeling in neural codec language models. Our main contributions are as follows: • We analyze the phenomenon of short-range dependency in neural codec language models, where prompt and local tokens provide necessary information for language models

Read the complete text