Entropy-based Coarse and Compressed Semantic Speech Representation Learning
Jialong Zuo, Guangyan Zhang, Minghui Fang, Shengpeng Ji, Xiaoqi Jiao, Jingyu Li, Yiwen Guo, Zhou Zhao · 2025
Read Entropy-based Coarse and Compressed Semantic Speech Representation Learning by Jialong Zuo, Guangyan Zhang, Minghui Fang, Shengpeng Ji, Xiaoqi Jiao, Jingyu Li, Yiwen Guo, Zhou Zhao (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2509-00503.txt.
Introduction
Jialong Zuo leads eight authors in total, with Guangyan Zhang, Minghui Fang, Shengpeng Ji, Xiaoqi Jiao, Jingyu Li, Yiwen Guo, and Zhou Zhao, on compressing speech codes. Speech is often turned into discrete tokens, small units, at twenty-five or fifty per second, though speech carries only a few words per second.
Zuo and his co-authors use predictive entropy, a measure of how unsure a model is about the next token, to decide where to merge tokens into segments. A cross-attention step, which weighs links between parts, fuses each segment, and changing a threshold controls how much compression occurs. Tests on recognition, translation, and voice conversion match or beat dense token sequences.
Catalog record
| CREATOR | Jialong Zuo, Guangyan Zhang, Minghui Fang, Shengpeng Ji, Xiaoqi Jiao, Jingyu Li, Yiwen Guo, Zhou Zhao |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 46 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2509-00503
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Entropy-based Coarse and Compressed Semantic Speech Representation Learning Jialong Zuo, Guangyan Zhang, Minghui Fang, Shengpeng Ji, Xiaoqi Jiao, Jingyu Li, Yiwen Guo, Zhou Zhao Abstract Discrete speech representation learning has recently attracted increasing interest in both acoustic and semantic modeling. Existing approaches typically encode 16 kHz waveforms into discrete tokens at a rate of 25–50 tokens per second. However, given that speech generally conveys only 2–5 words per second, such fine-grained tokenization introduces redundancy and hinders efficiency in downstream training and inference. Moreover, semantic speech representations at this frequency primarily capture phonetic-level information, while semantic understanding may not require such detailed token-level resolution. To address these limitations, we propose an entropy-based dynamic aggregation framework for learning compressed semantic speech representations. A speech language model is first pre-trained via next-token prediction on large-scale unlabeled data to capture frequent token patterns. Predictive entropy is then used to adaptively determine aggregation boundaries, followed by a cross-attention module that fuses information within each segment. By adjusting the entropy threshold, the granularity and compression ratio of the representations can be flexibly controlled. Experiments on ASR, speech-to-text translation, and voice conversion tasks demonstrate that the compressed representations perform on par with or better than dense token sequences, demonstrating the effectiveness of the proposed approach. 1 Introduction In recent years, discrete representation learning of speech Défossez et al. 2022 ; Kumar et al. 2023 ; Zhang et al. 2023 ; Ji et al. 2024 has achieved remarkable success and has gradually emerged as a crucial bridge between the speech modality and large language models Xie and Wu 2024a ; Xie and Wu 2024b ; Xu et al. 2025 . A notable direction within this paradigm is semantic-level discretization, aimed at retaining only the high-level semantic content of audio signals, enabling much lower bitrates. For tasks that primarily rely on speech content such as automatic speech recognition (ASR) or speech-to-text translation, semantic tokens provide a more efficient and task-relevant representation. Self-supervised learning (SSL) of speech Schneider et al. 2019 ; Baevski et al. 2020 ; Hsu et al. 2021 ; Chung et al. 2021 ; Chen et al. 2022 has demonstrated remarkable success in capturing rich and transferable speech representations across a wide range of downstream tasks. Building on this foundation, a growing number of studies have sought to extract semantic units from SSL-derived speech features. HuBERT Hsu et al. 2021 represents a pioneering effort in this direction, employing k-means clustering to tokenize speech, where the resulting discrete units are used as training targets for the speech encoder and are found to exhibit strong phoneme-level correlations Choi et al. 2024 . Subsequent works have adopted similar approaches to derive semantic representations: Spirit-LM Nguyen et al. 2025 extracts HuBERT features and applies k-means clustering with 500 centroids to define its basic semantic units, while Vevo Zhang et al. 2025 follows a comparable methodology. SpeechTokenizer Zhang et al. 2023 leverages semantic representations extracted from a HuBERT-based teacher model to guide the distillation of the first layer of residual vector quantization (RVQ), thereby generating discrete units intended to capture semantic content. Similarly, AudioLM Borsos et al. 2023 utilizes semantic tokens derived from w2v-BERT Chung et al. 2021 to encode high-level semantic information from audio inputs. In addition to extracting semantic units from self-supervised speech representations, several methods employ supervised approaches such as ASR or phoneme recognition to obtain semantic tokens. For instance, CosyVoice Du et al. 2024 proposes to use a supervised ASR module to construct a supervised semantic speech (S3) tokenizer. FACodec Ju et al. 2024 directly leverages phoneme labels (specifically, frame-level phoneme annotations obtained from internal alignment tools) to train discrete content representations in a supervised manner. While the effectiveness of these semantic tokenizers in capturing high-level semantic units has been validated across various downstream tasks such as ASR Yang et al. 2023 , text-to-speech (TTS) Du et al. 2024 ; Wang et al. 2024 , and voice conversion (VC) Kim et al. 2023 ; Liu 2024 ; Zuo et al. 2025 , several critical limitations remain to be addressed. First, existing semantic discretization approaches typically compress speech into 25 or more tokens per second, meaning that long speech utterances often result in an excessive number of discrete units. This introduces a substantial computational burden for training and inference in downstream models. Second, the semantic tokens often encode excessive phonetic detail Sicherman and Adi 2023 ; Cho et al. 2023 , which may be unnecessarily fine-grained for tasks focused on semantic understanding. Therefore, both further compression and coarser-grained semantic representations are necessary and desirable. In this study, we propose an entropy-based semantic token compression strategy for dynamically extracting semantic speech representations at varying levels of granularity. Specifically, building upon HuBERT units (discretized via a K-Means clustering method), we achieve further sequence compression by training a lightweight auto-regressive language model (LM) on large-scale pretrained speech data and computing the entropy of the next-token distribution. By leveraging the LM’s ability to model frequently occurring patterns (i.e., sequences of adjacent tokens), we define an entropy threshold to segment the token stream and merge adjacent tokens accordingly. This approach effectively reduces the sequence length. Furthermore, we demonstrate that modifying the entropy threshold and segmentation criteria enables flexible control over compression granularity, which subsequently influence the performance of downstream semantic understanding and generation tasks. The contributions of this work are summarized as follows: • We propose an entropy-based token aggregation framework that adaptively compresses semantic speech representations by merging adjacent tokens based on predictive uncertainty derived from a lightweight autoregressive language model. • Our method enables controllable granularity in semantic token sequences, significantly reducing sequence length while retaining essential semantic content, thus enhancing efficiency for downstream speech understanding and generation tasks. • Empirical results across multiple benchmarks demonstrate that our approach achieves a favorable balance between compression rate and task performance, validating its effectiveness and scalability. 2 Related Work 2.1 Self-supervised Speech Representation Learning Self-supervised learning (SSL) has emerged as a powerful paradigm for extracting rich and generalizable representations from large-scale unlabeled speech corpora. Recent advances such as wav2vec Baevski et al. 2020 , HuBERT Hsu et al. 2021 , and WavLM Chen et al. 2022 have demonstrated the effectiveness of SSL in capturing both phonetic and semantic information from raw audio signals. These models typically rely on pretext tasks such as masked prediction or contrastive learning to model the temporal structure of speech without requiring manual annotations. For example, wav2vec employs a contrastive loss to distinguish true audio segments from distractors, thereby learning high-level representations. HuBERT adopts a masked prediction objective, where targets are derived from offline k-means clustering, encouraging the model to learn context-aware representations. W2v-BERT Chung et al. 2021 further combines contrastive and masked learning in an end-to-end framework to leverage the benefits of both objectives. Meanwhile, WavLM further introduces a denoising objective to enhance robustness against noise and speaker variation. Figure 1: Illustration of the proposed entropy-based semantic token compression framework. Figure (a) shows the next-token entropy curve predicted by an autoregressive language model trained on HuBERT units, where a designed entropy threshold is used to segment token groups. Figure (b) illustrates the entropy language model, which is trained to estimate the conditional entropy of the next token given preceding