An Empirical Analysis of Discrete Unit Representations in Speech Language Modeling Pre-training
Yanis Labrak, Richard Dufour, Mickaël Rouvier · 2025
Read An Empirical Analysis of Discrete Unit Representations in Speech Language Modeling Pre-training by Yanis Labrak, Richard Dufour, Mickaël Rouvier (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2509-05359.txt.
Introduction
Turning speech into discrete units, like words for sound, is examined by Yanis Labrak, Richard Dufour, and Mickaël Rouvier in a 2025 paper on speech language models. These models adapt text-trained systems to spoken language, and the pre-training stage is where the sound units matter most.
The authors vary model design, unit representation, and training robustness to see what drives results. Encoder choice and cluster size, which sets how many distinct sound units exist, matter differently at different model scales. They also study how the units line up with speech sounds and carry tone and speaker traits. Matching the clustering data to the target domain proved important for robustness.
Catalog record
| CREATOR | Yanis Labrak, Richard Dufour, Mickaël Rouvier |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 21 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2509-05359
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
An Empirical Analysis of Discrete Unit Representations in Speech Language Modeling Pre-training Yanis Labrak, Richard Dufour, Mickaël Rouvier Abstract This paper investigates discrete unit representations in Speech Language Models (SLMs), focusing on optimizing speech modeling during continual pre-training. In this paper, we systematically examine how model architecture, data representation, and training robustness influence the pre-training stage in which we adapt existing pre-trained language models to the speech modality. Our experiments highlight the role of speech encoders and clustering granularity across different model scales, showing how optimal discretization strategies vary with model capacity. By examining cluster distribution and phonemic alignments, we investigate the effective use of discrete vocabulary, uncovering both linguistic and paralinguistic patterns. Additionally, we explore the impact of clustering data selection on model robustness, highlighting the importance of domain matching between discretization training and target applications. 1 Introduction The rapid advancement of pre-trained Large Language Models (LLMs) [ 21 , 25 ] has transformed Natural Language Processing (NLP), enabling systems with remarkable capabilities in text understanding and generation. However, these models remain largely text-based, overlooking the richness of spoken language, which conveys prosody, emotion, and speaker characteristics essential for human communication. Recent advances in self-supervised learning have made significant strides toward integrating speech into language modeling. Models such as WavLM [ 6 ] , HuBERT [ 10 ] and Wav2Vec 2 [ 3 ] have proven particularly effective at learning meaningful speech representations without explicit supervision [ 24 ] . The first attempts to bridge this gap between speech and language modeling emerged with Generative Spoken Language Models (GSLM) [ 13 ] , demonstrating the possibility of learning directly from raw audio without relying on text supervision. This breakthrough was followed by various approaches to integrate speech into language models [ 19 , 18 ] , primarily by incorporating discrete speech representations into their vocabularies [ 9 , 28 , 27 ] . While speech-extended LLMs have demonstrated promising results in downstream tasks [ 7 , 17 ] , their performance remains limited [ 9 ] and the fundamental challenge of optimizing speech modeling during continual pre-training remains largely unexplored. This stage is critical, as it determines how models initially learn to process speech input and serves as the foundation for all subsequent speech-related capabilities [ 18 , 15 ] . In this work, we systematically investigate discrete speech unit representations in language modeling. Through extensive experiments across model scales (135M to 1.7B parameters), encoder architectures, and discretization strategies, we address four fundamental questions: the optimal discretization granularity for different encoders, the impact of model scale on semantic information capture, the robustness of speech units to acoustic perturbations, and the nature of linguistic information embedded in these discrete representations. Our primary contributions are threefold: • First, we trained 51 speech language models of varying capacities (135M to 1.7B parameters) on the spoken language modeling objective introduced by SpeechGPT [ 28 ] through LoRA fine-tuning of pre-trained textual language models of the SmolLM family. • Second, we discovered a direct correlation between models’ speech modeling capabilities and discrete unit granularity, noting that smaller SLMs struggle to capture semantic information from higher discretization granularity units. • Finally, we observed that discrete units’ information strongly aligns with phonemes while simultaneously capturing other forms of acoustic information. Figure 1 : Overview of a Speech Language Model. 2 Spoken Language Modeling This section details our methodology for training and comparing our speech-extended language models, with a strong focus on speech representations. The studied SLM architecture follows the approach introduced by SpeechGPT [ 28 ] and relies on discrete units and vocabulary expansion. 2.1 Model Architecture We experiment with different variants of SmolLM [ 4 ] , using three model sizes: 135M, 360M, and 1.7B parameters. The core architecture remains unchanged from the original text models, with the only modification being the expansion of the tokenizer vocabulary to incorporate the newer tokens corresponding to the discrete units (see Section 2.2 ). The training objective follows a standard autoregressive language modeling approach with negative log-likelihood loss. For a sequence of tokens x=(x_{1},...,x_{T}) , the loss is computed as: \mathcal{L}=-\sum_{t=1}^{T}\log p(x_{t}|x_{<t}) (1) where p(x_{t}|x_{<t}) represents the probability of token x_{t} given all previous tokens in the sequence. Our approach does not aim for full acoustic reconstruction but instead prioritizes semantic modeling of speech as we are focusing on the first stage of speech adaptation of pre-trained textual language models. This stage consists exclusively of learning to process speech units alongside its existing text capabilities, as shown in Figure 1 . Training is conducted on 16 Nvidia H100 80GB GPUs with a batch size of 16 and gradient accumulation of 1. Using a context window of 2,048 tokens, we process 524,288 tokens per step. The training runs for 300 steps, processing approximately 157 million tokens in total. To optimize training efficiency and resource utilization, we incorporate several technical improvements such as LoRA adapters [ 11 ] (rank 64, alpha 16) for parameter-efficient fine-tuning. We chose BFloat16 precision and Flash Attention 2 to reduce memory overhead. It uses AdamW [ 14 ] optimization with a learning rate of 3\times 10^{-4} and applies a weight decay coefficient of 0.1 . To ensure reproducible results, the random seed is set to 42 . 2.2 Speech Encoding and Discretization To convert the raw speech signal from a continuous form into a discrete one that can be incorporated into the text input of the LLM, we need to have two components: an encoder and a discretizer. Here, we will evaluate four widely used self-supervised speech encoders: WavLM [ 6 ] , HuBERT [ 10 ] , XLS-R [ 2 ] , and Wav2Vec 2 [ 3 ] . For all encoders, we extract features from the final hidden layer, as prior work suggests that this layer provides a strong balance between acoustic and linguistic information [ 26 , 23 ] . No additional fine-tuning of the encoders is performed to maintain a fair comparison of their base capabilities. Each encoder extracts frame-level representations at 50 Hz (20 ms frames), which are then discretized into k clusters that will represent speech units using k-means, following standard practices in spoken language modeling [ 28 ] . To examine the impact of vocabulary size on modeling performance, we experiment with cluster counts of k\in\{125,250,500,1000,2500,5000\} . The k-means clustering used for speech encoders is trained on 2,000 hours of unlabeled speech for each of the following corpora: LibriHeavy [ 12 ] , GigaSpeech [ 5 ] , People’s Speech [ 8 ] , or CommonVoice 19 [ 1 ] . None of the data selected to build the k-means overlaps with the speech modeling dataset. 2.3 Speech modeling dataset We train the language models to process speech modality using LibriSpeech [ 22 ] , a widely used speech corpus containing 960 hours of read English speech. This dataset comprises three subsets (100h, 360h, and 500h), providing a diverse range of speakers and recording conditions. Speech segments are processed through our encoding pipeline (Figure 1 ) and using the newly built discrete speech units that serve as input to the language model. 2.4 Evaluation Methodology The effectiveness of each speech unit configuration is measured using Negative Log-Likelihood (NLL)