Speech Codec Probing from Semantic and Phonetic Perspectives

Xuan, Chang, Tiantian, Shih-Heng, Jianbo, Shrikanth · 2026

Read Speech Codec Probing from Semantic and Phonetic Perspectives by Xuan, Chang, Tiantian, Shih-Heng, Jianbo, Shrikanth (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2603-10371.txt.

Introduction

Xuan Shi and co-authors probe what widely used speech tokenizers actually store, testing them from semantic and phonetic angles. Such tokenizers link speech to large language models and are expected to keep both meaning and sound for later tasks. Through three tasks, the team measured lexical-semantic and phonetic content across several tokenizers. The finding is a mismatch: current tokenizers mostly capture phonetic structure, not the lexical meaning that the word semantic suggests in text work. The authors draw practical lessons for designing the next generation of speech tokenization.

Catalog record

CREATORXuan, Chang, Tiantian, Shih-Heng, Jianbo, Shrikanth
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE22068
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2603.10371

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Speech Codec Probing from Semantic and Phonetic Perspectives Xuan, Chang, Tiantian, Shih-Heng, Jianbo, Shrikanth Abstract Speech tokenizers are essential for connecting speech to large language models (LLMs) in multimodal systems. These tokenizers are expected to preserve both semantic and acoustic information for downstream understanding and generation. However, emerging evidence suggests that what is termed ”semantic” in speech representations does not align with text-derived semantics—a mismatch that can degrade multimodal LLM performance. In this paper, we systematically analyze the information encoded by several widely used speech tokenizers, disentangling their semantic and phonetic content through word-level probing tasks, layerwise representation analysis, and cross-modal alignment metrics such as CKA. Our results show that current tokenizers primarily capture phonetic rather than lexical-semantic structure, and we derive practical implications for the design of next-generation speech tokenization methods. Shi Zeng Feng Wang Ma Narayanan keywords speech codec, semantic, phonetic, attributes probing † † address: 1 University of Southern California, USA 2 Dolby Laboratories, USA † † email: [email protected] 1 Introduction The rapid advancement of large language models (LLMs) has driven significant interest in extending their sequence modeling capabilities to the speech modality. A growing body of research, including GPT-4o [ 12 ] , Qwen2.5-Omni [ 30 ] , and Moshi [ 6 ] , demonstrates the potential of unified multimodal large language models (MLLMs) that can simultaneously understand and generate both text and speech. Central to these systems is the speech tokenizer, which converts continuous speech waveforms into discrete token sequences amenable to the autoregressive modeling paradigm of LLMs, enabling a shared vocabulary and modeling framework for diverse tasks spanning automatic speech recognition (ASR), spoken language understanding, text-to-speech synthesis, and voice conversion [ 29 , 26 , 7 ] . The design of speech tokenizers has gradually evolved from pure signal compression to semantic-aware encoding. Early neural audio codecs such as SoundStream [ 32 ] , EnCodec [ 5 ] , and DAC [ 15 ] employ convolutional encoder-decoder architectures with Residual Vector Quantization (RVQ) [ 32 ] , optimizing reconstruction fidelity through perceptual and adversarial losses. While effective at preserving acoustic details, their representations are not explicitly designed to capture linguistic content [ 32 , 5 , 15 ] . To address this, AudioLM [ 1 ] introduced a two-stage paradigm that separates “semantic tokens” derived from SSL models (e.g., HuBERT [ 10 ] ) from “acoustic tokens” produced by neural codecs. This semantic-acoustic decomposition has since become widely adopted [ 8 ] , with MIMI [ 6 ] internalizing it by distilling WavLM [ 2 ] features into its first codebook layer. More recently, end-to-end systems such as Qwen2.5-Omni [ 30 ] and MiMo-Audio [ 29 ] directly jointly optimize speech tokenizer and LLM. However, a critical question remains: does the information labeled as “semantic” in speech tokens genuinely reflect linguistic meaning? Choi et al. [ 3 ] demonstrated that self-supervised learning (SSL) speech representations are more phonetic than semantic, exhibiting greater similarity between near-homophones (e.g., “accept” and “except”) than between synonyms (e.g., “big” and “large”). This suggests that the “semantic tokens”may more accurately be characterized as phonetic representations. If this is the case, the misalignment between phonetic speech tokens and semantic text tokens may contribute to the performance degradation observed in MLLMs on speech understanding tasks [ 28 , 4 ] . Therefore, probing the information encoded at each codebook layer is essential for understanding the algorithmic limitations and guiding future design. In this paper, we systematically probe the knowledge encoded in speech codec representations from both semantic and phonetic perspectives. We first clarify the conceptual distinction between “semantic” and “phonetic” as used throughout our analysis in Sec. 2.1 . We then conduct three complementary probing experiments across four representative speech codecs, namely EnCodec [ 5 ] , DAC [ 15 ] , MIMI [ 6 ] , and MIMO [ 29 ] , that span diverse architectural designs and training paradigms: (1) we first extend the SSL-based probing analysis [ 3 ] to neural codecs to examine the semantic-phonetic knowledge encoded in their representation. Specifically, we conduct this probing analysis using synonym and near-homophone word pairs in Sec. 2.3 ; (2) We then introduce an articulatory phonetic probing analysis using Vocal Tract Distance (VTD) features [ 13 , 22 ] extracted from real-time Magnetic Resonance Imaging (rt-MRI). This experiment provides a physiologically grounded assessment of phonetic encoding in Sec. 2.4 ; and (3) we finally evaluate the cross-modal semantic alignment between speech and text token spaces using Centered Kernel Alignment (CKA) in Sec. 2.5 . Through these analyses, we aim to provide actionable insights regarding the nature of information captured by current speech tokenizers, and to inform the design of future codecs that better serve language model integration. 2 Probing Experiment Settings In this section, we introduce the experiment settings for the probing experiments regarding semantic and phonetic distribution in speech codecs. 2.1 Probing Task: Semantic and Phonetic Within the speech processing community, the term “semantic” is often used to describe representations derived from self-supervised speech models such as HuBERT [ 10 ] and WavLM [ 2 ] . However, this usage conflates two fundamentally distinct linguistic concepts. In this paper, we adopt the following definitions. Semantic refers to lexical meaning — the level at which synonyms such as “big” and “large” are considered close. Phonetic refers to speech production — the level at which near-homophones such as “accept” and “except” are considered similar. 2.2 Model Selection We select four representative speech codec models for our probing experiments: EnCodec [ 5 ] , DAC [ 15 ] , MIMI [ 6 ] , and MIMO [ 29 ] . These models span a diverse range of architectural designs, training strategies, and intended applications, enabling a comprehensive analysis of how different codec design choices affect the amount of semantic and phonetic information preserved in the learned discrete units. Table 1 summarizes the key design choices of each model. EnCodec [ 5 ] employs a convolutional encoder-decoder architecture (SEANet [ 24 ] ) with Residual Vector Quantization (RVQ) [ 9 , 25 ] . It is trained end-to-end with a combination of reconstruction, perceptual, and adversarial losses. As one of the earliest and most widely adopted neural audio codecs, EnCodec is a pure compression model without explicit semantic or phonetic modeling. DAC [ 15 ] builds upon the RVQ-based codec framework with several architectural improvements, including factorized and L2-normalized codebook vectors [ 31 ] for better codebook utilization, snake activation functions [ 33 ] for improved periodic signal modeling, and quantizer dropout during training. MIMI [ 6 ] is the codec component of the Moshi real-time dialogue system. Its key distinguishing feature is a “semantic”-acoustic disentangled codebook design: the first quantizer layer is distilled from WavLM [ 2 ] , a SSL model, to capture higher-level “semantic” representations, while the remaining layers encode residual acoustic information through RVQ. MIMO-Audio-Tokenizer [ 29 ] (hereafter referred to as “MIMO”) is a Transformer-based codec developed within the MiMo-Audio framework. Unlike the codecs above, MIMO is trained from scratch through a two-stage process: joint audio reconstruction and ASR alignment with a LLM, followed by adversarial fine-tuning. Table 1 : Comparison of different speech tokenizers across various dimensions. SR

Read the complete text