Improving Large-Scale Weakly Supervised ASR by Filtering and Selection

Kohei, Masato · 2026

Read Improving Large-Scale Weakly Supervised ASR by Filtering and Selection by Kohei, Masato (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2606-28728.txt.

Introduction

Kohei Matsuura and Masato Mimura squeeze more accuracy out of imperfect speech data. End-to-end speech recognition needs enormous training sets. The practical supply is weakly supervised audio whose transcripts are noisy and off-domain, which caps how well the resulting models can perform.

Their recipe trains in three passes over the same material. First, pretrain on everything, noise included. Second, continue pretraining on a filtered subset chosen by character error rate, discarding the worst transcripts. Third, fine-tune on a small set of samples acoustically similar to the target domain. On a ninety-thousand-hour Japanese corpus, the filtering and selection steps each cut the error rate further, by up to about six and four percent respectively. That synergy is earned entirely from reusing data already seen.

Catalog record

CREATORKohei, Masato
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE23 KB
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2606.28728

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Improving Large-Scale Weakly Supervised ASR by Filtering and Selection Kohei, Masato Abstract Leveraging large-scale weakly supervised datasets is crucial to train robust end-to-end automatic speech recognition (ASR) models. However, such datasets often contain noisy labels and lack domain specificity, limiting their effectiveness. To address these issues and make better use of weakly supervised datasets, we propose a novel training approach incorporating data filtering and selection. Our approach consists of three steps: pretraining on the entire dataset, continued pretraining on a filtered subset based on character error rate (CER), and fine-tuning on a small number of acoustically similar samples to the target domain, selected from the filtered subset. In experiments with a 90,000-hour weakly supervised Japanese dataset, the proposed filtering and selection methods synergistically reduced CER by up to 6.4% and 4.0%, respectively, even though these steps reused training samples already used in the first pretraining step. Matsuura Mimura NTT CorporationJapan Kyoto UniversityJapan keywords end-to-end automatic speech recognition, large-scale weakly supervised pretraining, data filtering and selection † † email: [email protected] 1 Introduction End-to-end automatic speech recognition (ASR), which converts spoken utterances into transcriptions using a single deep learning model, has seen remarkable progress over the past five years [ 1 ] . In particular, large-scale self-supervised [ 2 , 3 ] and weakly supervised [ 4 , 5 ] pretraining has played a key role. Self-supervised learning (SSL) leverages unlabeled speech to learn general speech representations and is applicable beyond ASR. Meanwhile, weakly supervised learning, which we focus on in this study, relaxes the need for manually verified and precise labels when collecting task-specific training samples typically from internal speech archives and the Internet. Since it does not require expensive high-quality transcriptions, we can obtain very large-scale ASR datasets that encompass diverse speakers, speaking styles, and background noises. Although weakly supervised learning is effective for ASR, it has two potential issues. One is the existence of low-quality labels, such as misspellings and unspoken supplementary text. These noisy labels may negatively affect ASR accuracy. In semi-supervised learning, where an ASR model generates labels for unlabeled speech to retrain itself, it is common practice to filter out noisy labels considering confidence scores [ 6 ] . However, the effectiveness of such filtering strategies remains largely under-explored on weakly supervised datasets. The other issue stems from the diversity of these datasets. Trained with vast and diverse samples, ASR models may remain suboptimal for specific target domains. To alleviate this problem, fine-tuning the pretrained ASR models using samples similar to those in the target domains (i.e., data selection) is promising. Nevertheless, most studies have investigated data selection in the context of core-set selection [ 7 , 8 ] , where the primary motivation is not domain adaptation but reducing the training data. Few studies tackle on data selection to further improve robust pretrained models using weakly supervised datasets. Figure 1: Overview of our proposed approach, which appropriately narrows a single weakly supervised dataset to gradually improve an ASR model. PT, CPT, and FT denote pretrained, continually pretrained, and fine-tuned models, respectively. To address these issues and fully exploit weakly supervised datasets, we propose a novel training approach incorporating data filtering and data selection. Our approach, as depicted in Figure 1 , involves the following three steps: (1) First, we pretrain an encoder-only ASR model based on connectionist temporal classification (CTC) [ 9 ] with the entire large-scale weakly supervised dataset. Then, we use this pretrained model to transcribe all speech in the dataset and compute a character error rate (CER) between the predicted and the original noisy labels to each sample. (2) We filter out samples with CER higher than a predefined threshold r and continue pretraining the model using the remaining data. (3) If a target-domain training set is not available, we fine-tune the continually pretrained model obtained in Step 2 by selecting samples acoustically similar to the target-domain speech. Since complex similarity measures are not practical for large-scale datasets, we propose a simple method using cosine similarity on speech SSL embeddings. To evaluate our approach, we conducted experiments on a 90,000-hour Japanese weakly supervised ASR dataset, collected following [ 10 ] , and three public evaluation datasets. While most previous studies utilize such datasets only for Step 1 (i.e., simple pretraining), our experiments demonstrated further significant improvements through both Step 2 and 3. We also found that the filtering threshold r should be neither too low nor too high, revealing a trade-off between label quality and data diversity. Moreover, this improvement and trade-off trend persisted after fine-tuning when target-domain training sets were available, suggesting that our approach is widely applicable. 2 Related Work 2.1 Large-scale ASR pretraining and data filtering In recent years, ASR models trained on very large datasets, such as Whisper [ 5 ] , OWSM [ 11 ] , and Canary [ 12 ] , have emerged. Notably, Whisper achieves high robustness across diverse speech inputs by utilizing large-scale weakly supervised datasets collected from the Internet. However, weakly supervised datasets lack careful annotations and contain noisy labels with misspellings, word omissions or insertions due to misaligned timestamps, and unspoken supplementary text. Despite noisy labels potentially degrading recognition accuracy, the Whisper models are trained on most of the collected data with only minor heuristic filtering. Several studies have explored extensive data filtering. [ 13 ] applied CER-based filtering, removing samples with a CER above 50% using a transducer ASR model; however, they did not experimentally validate its effectiveness. [ 10 ] evaluated CTC score-based filtering, but their analysis was limited to a 160-hour dataset. Hence, the impact of filtering on large-scale weakly supervised datasets remains largely unexplored. 2.2 Data selection from large-scale datasets In ASR research, data selection from large-scale datasets has been widely studied in the context of core-set selection . Core-set selection aims to reduce training time by selecting a small subset, while maintaining recognition accuracy comparable to that obtained using the full dataset. Siohan et al. proposed a greedy method that minimizes the Kullback-Leibler divergence between the i-vector distributions of the selected and target dataset [ 14 ] . Subsequently, [ 15 ] applied submodular optimization [ 16 ] for efficient selection and replaced i-vectors with Gaussian Mixture Models (GMMs) of acoustic features. More recently, [ 8 ] leveraged n-gram language models trained on discrete representations from w2v-BERT [ 17 ] or VQ-VAE [ 18 ] . A few studies have explored data selection to improve recognition accuracy in the target domain. [ 19 ] selected speech from speakers similar to the target using Euclidean distance and cosine similarity of GMM super-vectors for speaker adaptation. [ 20 ] efficiently selected speech similar to the target accent using submodular mutual information functions. The most relevant work is by Lagos et al., who fine-tuned a pretrained ASR model using samples selected via SSL embeddings for domain adaptation [ 21 ] . However, their methods differ from ours in that they used separate datasets for pretraining and data selection, which may limit its applicability when an additional dataset is unavailable. Moreover, their dataset for data selection is only one-tenth the size of ours. 3 Methodology 3.1 Pretraining (Step 1) This section describes the proposed training approach on large-scale weakly supervised dataset.

Read the complete text