Exploring the Effect of Segmentation and Vocabulary Size on Speech Tokenization for Speech Language Models
Shunsuke, Yusuke, Shinnosuke · 2025
Read Exploring the Effect of Segmentation and Vocabulary Size on Speech Tokenization for Speech Language Models by Shunsuke, Yusuke, Shinnosuke (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2505-17446.txt.
Introduction
Turning speech into short code units is the first step for models that work with spoken words, yet the best way to cut and group those units is unclear. Shunsuke Kando and co-authors studied two choices in work from twenty twenty-five: how wide each cut should be, and how many groups the units should form.
The team tried fixed and varied cut widths, then grouped units with a standard grouping method at many sizes. Tests on spoken tasks showed that somewhat wide cuts and larger group counts helped most. The best trade-off also cut training data needs in half and training time by far more. Their point is plain: joining units into bigger chunks can help a model grasp fine points of speech.
Catalog record
| CREATOR | Shunsuke, Yusuke, Shinnosuke |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 20 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2505-17446
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Exploring the Effect of Segmentation and Vocabulary Size on Speech Tokenization for Speech Language Models Shunsuke, Yusuke, Shinnosuke Abstract The purpose of speech tokenization is to transform a speech signal into a sequence of discrete representations, serving as the foundation for speech language models (SLMs). While speech tokenization has many options, their effect on the performance of SLMs remains unclear. This paper investigates two key aspects of speech tokenization: the segmentation width and the cluster size of discrete units. First, we segment speech signals into fixed/variable widths and pooled representations. We then train K-means models in multiple cluster sizes. Through the evaluation on zero-shot spoken language understanding benchmarks, we find the positive effect of moderately coarse segmentation and bigger cluster size. Notably, among the best-performing models, the most efficient one achieves a 50% reduction in training data and a 70% decrease in training runtime. Our analysis highlights the importance of combining multiple tokens to enhance fine-grained spoken language understanding. Kando Miyao Takamichi The University of TokyoJapan Keio UniversityJapan keywords speech language models, spoken language understanding † † email: {skando,yusuke}@is.s.u-tokyo.ac.jp, [email protected] 1 Introduction With the recent breakthroughs in large language models for textual natural language processing, speech language models (SLMs) have emerged as a new paradigm for spoken language processing [ 1 , 2 , 3 , 4 ] . SLMs are built by training language models on top of discrete speech representations (called “discrete units”). The process of converting a speech signal into a discrete unit sequence is called “speech tokenization”. Speech tokenization is typically performed by quantizing representations obtained from self-supervised learning (SSL) models [ 5 , 6 , 7 ] . Leveraging the rich representations of SSL models, SLMs trained on these discrete units have demonstrated strong performance in zero-shot spoken language understanding (SLU) [ 8 ] , spoken dialogue [ 9 ] , speech-to-speech translation [ 10 ] , and other related tasks. Various speech tokenization techniques have been proposed to enhance SLM performance. Generative Spoken Language Modeling (GSLM) and its variants simply apply K-means clustering to the SSL model representations as is [ 1 , 11 , 2 ] . However, since SSL model representations typically correspond to approximately \SI 20 \milli \second speech, the resulting discrete unit sequence tends to be long. This severely affects the training of the Transformer-based language model [ 12 ] as computation cost increases quadratically with respect to the sequence length. Besides, previous study suggests that speech SSL representations primarily encode phonetic rather than semantic feature [ 13 ] , which might impair the capability of SLMs on a deeper understanding of spoken language. To address these issues, previous research has invented speech tokenization techniques that segment input speech into fixed or variable-width units before discretization [ 14 , 15 , 16 ] . While segmentation reduces sequence length, it might cause a loss of information preserved in the original representation, and there is no clear agreement on the optimal tokenization scheme for this tradeoff and its reason. Figure 1: Overview of our research. First, we extract continuous speech representation from the SSL model. We add segments to the representation sequence by \SI [parse-numbers=false]N \milli \second and pooled them. We apply K-means clustering to pooled representations with the cluster size of K . By training SLMs in multiple settings of N and K , we explore the optimal choice for spoken language understanding. This paper examines two key aspects of speech tokenization: the segmentation width and the cluster size of discrete units. As depicted in Figure 1 , we first segment the SSL representation sequence in a fixed width and pooled features within each segment to obtain coarser representations. Using these pooled representations, we then train K-means models to generate discrete unit sequences. By applying multiple segmentation widths and varying the cluster sizes of the K-means model, we explore the optimal configurations for zero-shot SLU tasks. Through comparative experiments, we find the positive effect of segmenting by moderately coarse width and making cluster size bigger at the same time. We qualitatively suggest that larger segmentation width requires a larger vocabulary to accurately represent input speech. Notably, a large segmentation setting reduces sequence length, enabling more lightweight training without sacrificing performance. We also observe that specific benchmarks have different optimal settings, highlighting the importance of combining multiple tokens for SLU. Besides fixed-width segmentation, we also investigate variable segmentation based on linguistic units (i.e., phonemes, syllables, and words) and compare their performances. Our results demonstrate that variable segmentation does not show a clear advantage over fixed-width segmentation, suggesting that simpler segmentation methods may be preferable. The experimental code is made publicly available 1 1 1 https://github.com/mynlp/speechlm . 2 Tokenization Methods to be Explored As depicted in Figure 1 , we first extract continuous speech representations of input speech using the SSL model. Throughout this study, we used HuBERT [ 7 ] as an SSL model and extracted representations from the ninth layer. On top of this representation sequence, we performed speech tokenization in three steps. 1. Segment a sequence by \SI [parse-numbers=false]N \milli \second and apply mean pooling. 2. Apply K-means clustering with the cluster size K . 3. Deduplicate units. (e.g. 54 54 54 88 88 3 \rightarrow 54 88 3) Since each HuBERT representation corresponds to \SI 20 \milli \second speech, N is chosen as a multiple of 20. We experiment with eight values: {20, 40, 80, 120, 160, 200, 240, 280}, where N=20 corresponds to the original sequence. As N becomes larger, the resulting sequence length becomes shorter by the factor of N/20 . As for K , the choice of value is not consistent across SLM studies, ranging from {50, 100, 200} [ 1 ] to {5k, 10k, 20k} [ 16 ] . To comprehensively cover this range, we experiment with eight values of powers of two: from 2^{7}=128 to 2^{14}=16384 . In total, we employed 8\times 8=64 tokenization methods. Previous studies typically set a smaller cluster size for methods with small segment widths [ 1 , 3 ] , while larger segment widths are paired with larger cluster sizes [ 15 , 16 ] . This research supplements the cases of large/small cluster size and small/large segment width, aiming to gain deeper insight into these two aspects. 3 Experimental Setup 3.1 Dataset Table 1: Minimum and maximum number of tokens for every segmentation width. Minimum and maximum values correspond to K=2^{7} and K=2^{14} , respectively. N 20 40 80 120 160 200 240 280 min 87M 63M 39M 27M 21M 17M 14M 12M max 127M 77M 42M 28M 22M 17M 14M 12M As a training set for SLM, we used LibriSpeech [ 17 ] , a 960-hour English audiobook corpus. Although this dataset is relatively small for SLM studies, our preliminary experiments showed that using a larger dataset (LibriLight [ 18 ] ; 60k hours audiobook corpus) did not lead to performance improvements. A recent study on SLM based on syllable-level units [ 16 ] also supports the use of LibriSpeech, as it reports better performance compared to baselines trained on the larger dataset. Table 1 presents statistics on the training data. As described in Section 2 , the sequence length is smaller when N is larger,