Which Data Matter? Embedding-Based Data Selection for Speech Recognition
Zakaria Aldeneh 1, Skyler Seto 1, Maureen de Seyssel 1, Jie Chi 1, Zijin Gu 1, Takuya Higuchi 1, Jee-weon Jung 1, Shinji Watanabe 2, David Grangier 1, Barry-John Theobald 1, Tatiana Likhomanenko 1 · 2026
Read Which Data Matter? Embedding-Based Data Selection for Speech Recognition by Zakaria Aldeneh 1, Skyler Seto 1, Maureen de Seyssel 1, Jie Chi 1, Zijin Gu 1, Takuya Higuchi 1, Jee-weon Jung 1, Shinji Watanabe 2, David Grangier 1, Barry-John Theobald 1, Tatiana Likhomanenko 1 (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2603-05819.txt.
Introduction
Aldeneh and team study picked choice for expert speech spotting models. All-round models benefit from broad pseudo-labeled data, but experts lack room to absorb it all and suffer training-test mismatch. The authors show speech samples with vectors holding speaker traits, sound content, and meaning, then analyze how fit and range along those axes affect downstream scores. With CTC-based Conformer models, training on a chosen with care 5 percent subset beats full training, cutting word error rate by up to 36.8 percent compared with full training on target domains.
Catalog record
| CREATOR | Zakaria Aldeneh 1, Skyler Seto 1, Maureen de Seyssel 1, Jie Chi 1, Zijin Gu 1, Takuya Higuchi 1, Jee-weon Jung 1, Shinji Watanabe 2, David Grangier 1, Barry-John Theobald 1, Tatiana Likhomanenko 1 |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 43425 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2603.05819
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Which Data Matter? Embedding-Based Data Selection for Speech Recognition Zakaria Aldeneh 1, Skyler Seto 1, Maureen de Seyssel 1, Jie Chi 1, Zijin Gu 1, Takuya Higuchi 1, Jee-weon Jung 1, Shinji Watanabe 2, David Grangier 1, Barry-John Theobald 1, Tatiana Likhomanenko 1 Abstract Modern ASR systems are typically trained on large-scale pseudo-labeled, in-the-wild data spanning multiple domains. While such heterogeneous data benefit generalist models designed for broad deployment, they pose challenges for specialist models targeting specific domains: specialist models lack the capacity to learn from all available data, and one must pay closer attention to addressing the mismatch between training and test conditions. In this work, we study targeted data selection as a strategy to address these challenges, selecting relevant subsets from 100 100 k hours of in-the-wild training data to optimize performance on target domains. We represent speech samples using embeddings that capture complementary characteristics—speaker attributes, phonetic content, and semantic meaning—and analyze how relevance and diversity along these axes when performing data selection affect downstream ASR performance. Our experiments with CTC-based Conformer models show that training on a strategically selected 5 % 5\% subset can exceed the performance of models trained on the full dataset by up to 36.8 % 36.8\% relative WER reduction on target domains. \metadata [Correspondence] Zakaria Aldeneh: [email protected] ; Tatiana Likhomanenko: [email protected] 1 Introduction The curation of large-scale training datasets has had a notable impact on speech processing tasks, including automatic speech recognition (ASR) ( Ardila et al., 2020 ; Chan et al., 2021 ; Wang et al., 2021 ; Radford et al., 2023 ) . Such datasets expose speech models to a wide range of acoustic conditions, speakers, and recording scenarios, enabling them to learn robust representations and achieve strong performance. However, such success is typically achieved with generalist models that have the capacity to leverage the scale and heterogeneity of the training data. In practice, we often deploy specialist models tailored to specific domains or tasks—models that lack the capacity to learn from all available training data effectively. For these specialist models, domain mismatch between training and test conditions becomes essential: a specialist model cannot capture all the details of a large, heterogeneous dataset and must instead focus on data closely aligned with the target conditions. This scenario motivates our central question: Can we strategically select subsets of large-scale, in-the-wild data that enable specialist models to outperform models trained on the full dataset? Mismatch between the train and test domains along any dimension of variation in speech can degrade performance. Prior work has shown, for example, that models trained primarily on read speech perform poorly when evaluated on spontaneous speech ( Likhomanenko et al., 2021 ) . Similarly, the performance of models trained on native accents degrades when applied to accented or non-native speech ( Fukuda et al., 2018 ; Ghorbani and Hansen, 2018 ; Zhang et al., 2022 ) . These observations highlight that not all training data are equally useful for a given target domain, and an effective data selection strategy cannot rely on a single notion of similarity. Effective data selection, therefore, requires capturing relevance across multiple, complementary axes of speech variability, rather than relying on a single notion of similarity when addressing the mismatch between train and test conditions. We study targeted data selection for ASR in the regime of large-scale in-the-wild training data (over 100 k hours) and specialist models with approximately 10 - 100 M parameters (production-size models). We represent speech samples using embeddings that capture diverse properties—speaker, phonetic, and semantic—and use them to select subsets of data that are both relevant to the target domain and diverse along these dimensions. We analyze which characteristics matter most for downstream performance, and whether combining multiple representations yields additional gains. Our experiments span multiple target datasets and operate at a scale representative of modern ASR training pipelines that rely on large-scale in-the-wild pseudo-labeled data, offering insights into effective data selection in that regime. Specifically, we show that one can achieve up to 36.8\% improvement in performance on target domains with data selection in this training regime, even with 5\% of the total available training data. 2 Related Work Data selection has been studied in the context of ASR ( Wu et al., 2007 ; Wei et al., 2014b ; Wei et al., 2014a ; Park et al., 2022 ; Zhang and Rudnicky, 2006 ; Lin and Bilmes, 2009 ; Zheng et al., 2023 ; Doulaty et al., 2015 ; Afshan et al., 2021 ; Lu et al., 2022 ; Kothawade et al., 2023 ) . Prior work approached the problem from several perspectives: improving training efficiency by reducing dataset size ( Wu et al., 2007 ; Wei et al., 2014b ; Wei et al., 2014a ; Park et al., 2022 ) , selecting samples for active learning ( Zhang and Rudnicky, 2006 ; Zheng et al., 2023 ) , selecting samples to annotate by humans under limited annotation budgets ( Lin and Bilmes, 2009 ; Nallasamy et al., 2012 ) , and improving domain-specific performance by prioritizing relevant data ( Doulaty et al., 2015 ; Afshan et al., 2021 ; Lu et al., 2022 ; Kothawade et al., 2023 ) . Our work focuses on the last setting: selecting relevant samples from large-scale in-the-wild training corpora to accommodate train/test mismatches to improve performance on a specific target domain. Confidence-based strategies have been widely used in semi-supervised and active learning settings, where samples are chosen based on model prediction confidence. However, selecting data solely based on confidence has been shown to be suboptimal for downstream performance ( Zhang and Rudnicky, 2006 ) . These findings motivated distribution-matching approaches that account for domain mismatch. For example, Wu et al., 2007 selected subsets with uniform coverage over linguistic units such as words or phonemes. Other methods used learned representations for data selection. Acoustic-based approaches have used i-vectors ( Siohan and Bacchiani, 2013 ) and other low-level signal statistics to measure similarity between candidate and target samples. Lu et al., 2022 proposed unsupervised selection using discrete speech units derived from self-supervised models. Afshan et al., 2021 used sequence-level confidence scores from pre-trained ASR models to identify informative samples for domain adaptation. Rangappa et al., 2025 combined domain classifiers with pseudo-label filtering to select unlabeled data for fine-tuning. In the context of self-supervised learning, Gody and Harwath, 2023 explored data selection for fine-tuning pre-trained speech models on downstream tasks. We differ from prior work in three key aspects: (1) Scale : we operate on 100 k+ hours of in-the-wild pseudo-labeled data from Granary ( Li et al., 2023 ; Rao Koluguri et al., 2025 ) to train production-scale specialist models (10-100M parameters), whereas most prior work used small-scale datasets (e.g., Switchboard, LibriSpeech); (2) Multi-embedding selection : we capture complementary speech characteristics—speaker, phonetic, and semantic—and analyze both their individual impact and the benefits of combining them, rather than focusing on a single aspect; (3) Multi-target selection : we balance relevance and diversity while selecting for multiple target domains simultaneously, rather than optimizing for one domain at a time. Overall, we study a setting increasingly common in practice: using large-scale, diverse in-the-wild pseudo-labeled data to train specialist ASR models for specific target domains. 3 Approach