On-device Streaming Discrete Speech Units
Kwanghee, Masao, Emma, Shinji · 2025
Read On-device Streaming Discrete Speech Units by Kwanghee, Masao, Emma, Shinji (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2506-01845.txt.
Introduction
Kwanghee Choi and co-authors shrink speech coding units so they can stream on small devices, in a 2025 paper. Speech can be split into discrete units that carry sound detail, travel cheaply, and pair well with language models. Standard methods need the whole recording at once and use heavy models, which blocks live use.
The team cuts both the listening window and the model size while keeping the units useful. Computing cost fell by half, while letter errors rose only slightly on a benchmark set. Tests used a public multilingual speech challenge set with one hour of data. The authors say the trade-off opens real-time speech tools for phones and other tight hardware, where power and memory are scarce.
Catalog record
| CREATOR | Kwanghee, Masao, Emma, Shinji |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 30 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2506-01845
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
On-device Streaming Discrete Speech Units Kwanghee, Masao, Emma, Shinji Abstract Discrete speech units (DSUs) are derived from clustering the features of self-supervised speech models (S3Ms). DSUs offer significant advantages for on-device streaming speech applications due to their rich phonetic information, high transmission efficiency, and seamless integration with large language models. However, conventional DSU-based approaches are impractical as they require full-length speech input and computationally expensive S3Ms. In this work, we reduce both the attention window and the model size while preserving the effectiveness of DSUs. Our results demonstrate that we can reduce floating-point operations (FLOPs) by 50% with only a relative increase of 6.5% in character error rate (CER) on the ML-SUPERB 1h dataset. These findings highlight the potential of DSUs for real-time speech processing in resource-constrained environments. Choi Someki Strubell Watanabe Carnegie Mellon UniversityUSA keywords discrete speech units, streaming, on-device, speech recognition † † email: {kwanghec,msomeki,estrubel,swatanab}@andrew.cmu.edu 1 Introduction Self-supervised speech models (S3Ms) have demonstrated remarkable efficacy in various speech-related tasks, including automatic speech recognition (ASR), text-to-speech synthesis, speech translation, speaker identification, and emotion recognition [ 1 , 2 , 3 ] . With the advent of S3Ms, many have started to leverage their features for tokenizing speech, i.e. , producing discrete speech units (DSUs) [ 4 , 5 , 6 , 7 , 8 , 9 , 10 ] . DSUs are typically generated by applying clustering methods such as k-means to S3M features, with the resulting clusters treated as DSUs. The characteristics of DSUs are well adapted to on-device streaming scenarios, offering cheap transmission and easy adaptation to existing large language models (LLMs). Unlike high-dimensional speech features, such as spectral speech features or S3M features, DSUs offer improved data storage and transmission efficiency [ 4 , 10 ] while maintaining comparable ASR performance [ 5 ] . These advantages make DSUs a promising medium for speech data transmission. For instance, a 1-second 16kHz audio signal requires 512kbps in raw form. S3M features ( e.g. , wav2vec 2.0-large [ 2 ] ) increase this to 1600kbps. However, DSUs with 2k clusters reduce this to just 0.6kbps, significantly compressing the data by 3-4 orders of magnitude. DSUs also offer a promising way to extend LLMs to speech modalities [ 8 , 11 , 9 ] . A practical application involves generating DSUs on client devices and transmitting them to server-side LLMs for processing. This may also provide benefits for anonymizing personally identifiable information for privacy concerns [ 12 ] as S3Ms lose speaker information at later layers [ 13 ] . (a) (b) (c) Figure 1: Pareto tradeoff between ASR performance (WER on Librispeech, CER on ML-SUPERB 1h) and computational efficiency (TFLOPs per 1 minute input) for various modifications on WavLM-large. Results show that we can extract DSUs in a lightweight way with limited impact on ASR performance. However, DSUs have yet to be leveraged for streaming on edge devices due to two major challenges. First, conventional S3Ms rely on bidirectional Transformer encoder architectures, with full attention window size to cover the entire input speech, which is not immediately compatible with the streaming setting. Second, state-of-the-art S3Ms typically require hundreds of millions of parameters, which can be computationally infeasible to execute on edge devices with limited computational capacity. It poses a problem for on-device streaming, where GPU memory restricts the total sequence length and prevents real-time generation of DSUs. In summary, a model for extracting DSUs must work well even with a limited window of future frames while being computationally lightweight. Evidence from previous work suggests this should be possible. For example, prior work has indicated that S3Ms predominantly encode phonetic information, with limited syntactic and semantic content [ 14 , 13 , 15 , 16 ] . Furthermore, studies have shown a strong correlation between extracted DSUs and phonemes [ 2 , 17 ] . These findings indicate that DSUs can be extracted with a smaller window size. Regarding the model size, we hypothesize that we can reduce the number of required parameters by solely targeting the specific input distribution, similar to the ideas of previous work [ 18 , 19 ] . Typical S3Ms are self-supervised on diverse data to perform well on a wide variety of audio, i.e. , “universal audio feature.” However, in practice, DSUs would still be useful even if tailored for specific target data and tasks. In this work, we investigate the feasibility of DSUs for streaming on edge devices by exploring the following two key questions: 1. How much temporal window size is needed? ( Section 3 ); 2. How much model capacity (parameters) is needed? ( Section 4 ). We investigate tradeoffs between computational efficiency and downstream performance in this setting by varying the attention window size and the number of layers of WavLM-large [ 3 ] , a representative S3M for extracting DSUs [ 6 ] . For each setting, a smaller model is trained by regarding the full model’s DSUs as ground truth. We evaluate feasibility by measuring the computational cost, comparing with ASR performance on Librispeech [ 20 ] and ML-SUPERB [ 21 ] . Our contributions in this paper are summarized as follows: • To the best of our knowledge, this study is the first to explore DSUs in the on-device streaming setting, exploring the feasibility of real-time lightweight computation of DSUs. • We explore the trade-off between downstream performance and computational overhead by varying the attention window size and the number of layers of the original S3M, obtaining Pareto optimal trade-off curve. • By optimizing the model’s attention window size and number of layers, we reduce FLOPs by 50% while having only a 6% relative increase in CER on the ML-SUPERB 1h dataset. 2 Experimental Settings Our work investigates whether DSUs can be accurately predicted using a smaller model. To evaluate the predicted DSUs, we focus on the discrete ASR system. We use the same experimental settings across all experiments. 2.1 Discrete ASR System Discrete ASR system contains two main modules: speech-to-unit (S2U) module and unit-to-text (U2T) module. S2U module generates DSUs, i.e. , speech discretization from continuous S3M features. Given the input raw speech \mathbf{x}\in\mathbb{R}^{t} with length t , f_{\text{S3M}} transforms speech into S3M features: \displaystyle\mathbf{S}=f_{\text{S3M}}(\mathbf{x})\in\mathbb{R}^{T\times F}, (1) where F is the feature dimension and T is the temporal dimension roughly proportional to t . Then, the pretrained k-means algorithm is applied to yield the stream of DSUs \mathbf{D}=f_{\text{kmeans}}(\mathbf{S}) . The cascade of S3M and k-means becomes the S2U module: \displaystyle f_{\text{S2U}}(\mathbf{x})=f_{\text{kmeans}}(f_{\text{S3M}}(\mathbf{x}))=\mathbf{D}=[D_{1},D_{2},\cdots,D_{T}], (2) where D_{i}\in\{1,2,\cdots,V\} is the DSU at timestep i and V is the vocabulary size, i.e. , k in k-means. After generating DSUs from speech through the S2U module ( eq. 2 ), U2T module translates DSUs to text transcriptions. To reduce the sequence length, additional post-processing methods are often applied, such as deduplication or byte-pair encoding tokenization [ 5 , 6 ] : \displaystyle\mathbf{D}^{\prime}=f_{\text{subword}}(\mathbf{D})=[D^{\prime}_{1},D^{\prime}_{2},\cdots,D^{\prime}_{T^{\prime}}], (3) where D^{\prime}_{j}\in\{1,2,\cdots,V^{\prime}\} is the post-processed DSU at timestep j . Often, the sequence length T^{\prime} becomes shorter and vocabulary size V^{\prime} becomes larger, i.e. , T^{\prime}\leq T , and V^{\prime}\geq V . Finally, the discrete ASR model f_{\text{ASR}} is trained to predict the transcription \mathbf{y} . The cascade of post-processing methods ( eq. 3 ) and discrete ASR