The Limits of Data Scaling: Sub-token Utilization and Acoustic Saturation in Multilingual ASR
2025
Read The Limits of Data Scaling: Sub-token Utilization and Acoustic Saturation in Multilingual ASR (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2510-22492.txt.
Introduction
How much speech is needed before a multilingual recognition model stops finding new sound pieces? Siyu Liang, Nicolas Ballier, Gina-Anne Levow, and Richard Wright ask that question in a 2025 paper, tracking the Whisper model as it decodes speech in forty nine languages.
Liang and co-authors log the small token pieces Whisper considers and watch their discovery slow over time. Total discovery barely depends on how much training speech a language had, and each language reaches an acoustic saturation point. Token patterns follow a known skewed rank law, while average token length rises with resource level and differs by writing script.
Catalog record
| YEAR | 2025 |
|---|---|
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 31 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2510-22492
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
The Limits of Data Scaling: Sub-token Utilization and Acoustic Saturation in Multilingual ASR Abstract How much audio is needed to fully observe a multilingual ASR model’s learned sub-token inventory across languages, and does data disparity in multilingual pre-training affect how these tokens are utilized during inference? We address this question by analyzing Whisper’s decoding behavior during inference across 49 languages. By logging decoding candidate sub-tokens and tracking their cumulative discovery over time, we study the utilization pattern of the model’s sub-token space. Results show that the total number of discovered tokens remains largely independent of a language’s pre-training hours, indicating that data disparity does not strongly influence lexical diversity in the model’s hypothesis space. Sub-token discovery rates follow a consistent exponential saturation pattern across languages, suggesting a stable time window after which additional audio yields minimal new sub-token activation. We refer to this convergence threshold as acoustic saturation time (AST). Further analyses of rank–frequency distributions reveal Zipf-like patterns better modeled by a Zipf–Mandelbrot law, and mean sub-token length shows a positive correlation with resource level. Additionally, those metrics show more favorable patterns for languages in the Latin script than those in scripts such as Cyrillic, CJK and Semitic. Together, our study suggests that sub-token utilization during multilingual ASR inference is constrained more by the statistical, typological, and orthographical structure of the speech than by training data scale, providing an empirical basis for more equitable corpus construction and cross-lingual evaluation. Keywords: multilingual ASR, corpus evaluation, fairness, large language model, tokenization, language representation The Limits of Data Scaling: Sub-token Utilization and Acoustic Saturation in Multilingual ASR Siyu Liang 1 , Nicolas Ballier 2 , Gina-Anne Levow 1 , Richard Wright 1 1 Department of Linguistics, University of Washington, Seattle, WA, USA 2 ALTAE, Université Paris Cité, F-75013 Paris, France {liangsy, levow, rawright}@uw.edu, [email protected] Abstract content 1. Introduction How much audio is required to observe a multilingual ASR model’s learned sub-token inventory, and does data disparity in multilingual pre-training shape how those tokens are utilized at inference? This question matters for language resource planning, especially in low-resource settings where collection costs are high and test-set design must be principled ( Javed et al. 2022 ; Sehar et al. 2025 ) . We address this problem empirically by probing Whisper ( Radford et al. 2023 ) during decoding with speech from the Common Voice dataset ( Ardila et al. 2024 ) . Rather than focusing on output error rates, we analyze decoder dynamics: at every decoding step we log sub-token candidates and track their cumulative discovery as input audio increases. This procedure directly measures how a model activates its multilingual sub-token space over time. Our analysis spans 49 languages covering a broad typological, orthographic, and resource spectrum. We investigate how the activation and utilization patterns of sub-tokens relate to model pre-training scale, how discovery trajectories stabilize as additional audio is introduced, and how these processes reflect underlying regularities in sub-token frequency and segmentation. Together, we aim to characterize when a model’s learned vocabulary becomes effectively "saturated" for a given language and input duration. The results reveal consistent and interpretable patterns. The total number of discovered sub-tokens at a fixed duration shows non-significant correlation with pre-training hours, suggesting that data scale alone does not determine sub-token utilization in the model’s hypothesis space. The rates at which new sub-tokens are activated follow a consistent exponential saturation trend across languages, revealing a stable acoustic saturation time (AST)—typically around two hours—after which additional audio yields minimal sub-token discovery. Rank–frequency distributions exhibit patterns that can be characterized by a Zipf–Mandelbrot model with script-specific parameters, while the length of active sub-token vocabularies shows modest but significant correlation with training scale among languages sharing the Latin script. Our findings complement output-centric evaluations by revealing cross-lingual regularities in the sub-token utilization space. By demonstrating that sub-token utilization is shaped more by statistical, typological, and orthographical structures than by data scale, this study offers an empirical basis for more equitable multilingual corpus construction and connects inference-time dynamics to broader discussions of tokenization fairness in large multilingual models ( Sennrich et al. 2016 ; Petrov et al. 2023 ; Ahia et al. 2024 ; Tjandra et al. 2023 ) . 2. Related Work 2.1. Multilingual ASR and Resource Disparities Large-scale multilingual ASR systems exhibit pronounced performance variation across languages, much of which correlates with training data availability ( Radford et al. 2023 ; Pratap et al. 2024 ) . While data imbalance remains a key determinant of word error rate (WER) and character error rate (CER), most studies assess disparities through output-level metrics rather than inference-time dynamics ( Zee et al. 2024 ; Swain et al. 2024 ) . This limits understanding of how model behavior differs internally across languages. Recent work has begun to probe representation and decoding mechanisms in large speech language models. Langedijk et al. 2024 proposed layerwise interpretability methods for encoder–decoder transformers, and Ballier et al. 2024 examined sub-token probabilities and decoder confidence for pronunciation and learner speech assessment. Related analyses highlight structural bias in multilingual decoders ( Koenecke et al. 2024 ; Muller et al. 2021 ) and the importance of uncertainty calibration ( Guo et al. 2017 ) . Our study extends this line by focusing on sub-token discovery-the emergence and saturation of candidate tokens during decoding—as a window into how multilingual ASR models allocate representational capacity across languages of differing resource levels. 2.2. Tokenization and Multi-lingual Modeling Tokenization is a central component of multilingual modeling. Byte-pair encoding (BPE) ( Sennrich et al. 2016 ) remains the standard for both text and speech models ( Radford et al. 2023 ; Pratap et al. 2024 ) , but multilingual vocabularies inherently encode linguistic and resource biases. Prior work has shown that sub-token segmentation varies widely across languages, with low-resource or morphologically complex languages often fragmented into shorter subwords ( Petrov et al. 2023 ; Ahia et al. 2024 ) . Typological studies further link BPE compression behavior to morphological structure ( Gutierrez-Vasques et al. 2023 ) , while scaling analyses demonstrate that tokenization strategy substantially impacts performance across 70+ languages ( Tjandra et al. 2023 ) . Alternative approaches explore discrete acoustic or self-supervised tokens ( Guo et al. 2025 ; Cui et al. 2024 ) , and adapter-based multilingual representations ( Song et al. 2024 ; Aggarwal et al. 2025 ) . While prior research has examined tokenization primarily as a training-time design variable—shaping vocabulary efficiency, cross-lingual transfer, and morphological alignment—its implications for inference remain underexplored. In particular, few studies have investigated how multilingual ASR models deploy their learned sub-token inventories during decoding: how many distinct units are activated in response to natural speech, how activation evolves with input duration, and whether these dynamics correlate with training scale or linguistic properties. This study advances that perspective by systematically quantifying sub-token utilization across 67 languages, linking tokenization behavior to multilingual data disparity and model representational capacity. 2.3. Corpus Design and Resource Evaluation A recurring challenge in ASR corpus development is determining how much speech data is needed for robust evaluation and meaningful cross-lingual comparison. While large-scale efforts such as Common Voice provide the scale for such analyses ( Ardila et al. 2020 ) , most prior work on multilingual