From Weak Labels to Strong Results: Utilizing 5,000 Hours of Noisy Classroom Transcripts with Minimal Accurate Data

Ahmed Adel, Dorottya, Jing, Carol · 2025

Read From Weak Labels to Strong Results: Utilizing 5,000 Hours of Noisy Classroom Transcripts with Minimal Accurate Data by Ahmed Adel, Dorottya, Jing, Carol (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2505-17088.txt.

Introduction

Ahmed Adel Attia and three co-authors train speech recognition on noisy labels in twenty twenty-five. Classroom recordings are plentiful, but their transcripts are weak and error-filled. Gold-standard transcripts exist only in small amounts, and re-transcribing is costly. The authors ask how best to learn in exactly that mix.

Their answer is Weakly Supervised Pretraining, a two-step process. Models first pretrain on the weak transcripts in a supervised way. They then fine-tune on the small accurate set. The team tested the idea on five thousand hours of classroom audio. Both synthetic and real weak transcripts were tried. The method beat the alternatives in their tests. Readers in low-resource speech work get strong results from mostly weak labels.

Catalog record

CREATORAhmed Adel, Dorottya, Jing, Carol
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE25 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2505-17088

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

From Weak Labels to Strong Results: Utilizing 5,000 Hours of Noisy Classroom Transcripts with Minimal Accurate Data Ahmed Adel, Dorottya, Jing, Carol Abstract Recent progress in speech recognition has relied on models trained on vast amounts of labeled data. However, classroom Automatic Speech Recognition (ASR) faces the real-world challenge of abundant weak transcripts paired with only a small amount of accurate, gold-standard data. In such low-resource settings, high transcription costs make re-transcription impractical. To address this, we ask: what is the best approach when abundant inexpensive weak transcripts coexist with limited gold-standard data, as is the case for classroom speech data? We propose Weakly Supervised Pretraining (WSP), a two-step process where models are first pretrained on weak transcripts in a supervised manner, and then fine-tuned on accurate data. Our results, based on both synthetic and real weak transcripts, show that WSP outperforms alternative methods, establishing it as an effective training methodology for low-resource ASR in real-world scenarios. Attia Demszky Liu Espy-Wilson University of Maryland Stanford University University of Maryland keywords ASR, weakly supervised learning † † email: [email protected], [email protected], [email protected], [email protected] 1 Introduction Automatic speech recognition (ASR) systems have seen remarkable improvements in recent years, largely driven by the availability of large-scale labeled speech datasets and advances in deep learning. State-of-the-art models like Whisper [ 1 ] are trained on more than half a million hours of webscale transcripts. However, in many domains, acquiring high-quality transcriptions remains a costly and time-consuming process, limiting the amount of labeled data available for training. In fact, high-quality transcriptions can cost upwards of $150 per hour of speech [ 2 , 3 ] . This high cost results in data scarcity in many domains which hinders the development of robust ASR models for low-resource and domain-specific tasks. A potential solution is to leverage weak transcriptions, which may be imperfect or noisy with many errors but still retain useful information and can be acquired cheaply. The central question we explore in this work is how to effectively integrate weak transcriptions with limited gold-standard transcriptions to improve ASR performance in low-resource domains. This approach falls under the domain of Weakly Supervised Learning (WSL) [ 4 ] . Unlike fully supervised learning, where the model is trained on a precisely labeled dataset, and self-supervised or unsupervised learning where the model is trained on unlabeled data, WSL is a paradigm of machine learning where models are trained on partially labeled or imprecisely labeled data. In the task of ASR, WSL, particularly inaccurate supervision [ 5 , 6 ] , has enabled the use of imprecise or inaccurate transcription. WSL has proved to be crucial in many ASR models, due to the large costs of accurate transcription. Perhaps the most well-known example of a weakly-supervised ASR model is Whisper, where the large training corpus was not precisely vetted, and portions of it were known to be inaccurate. However, the model’s contextual capacity and the large size of the dataset proved to be adequate, so that the model can learn well from the noisy labels. Whisper’s success with inaccurate supervision motivated us to explore the applicability of weak transcripts in low-resource ASR, primarily in classroom speech. Classrooms present a unique challenge in ASR since they are multi-speaker environments with multiple target speakers (teachers and students) whose speech sometimes overlap [ 7 , 8 ] . Additionally, classrooms are noisy environments characterized by children’s babble noise, which is the noise of multiple background speakers and is considered one of the most difficult noises in ASR [ 9 ] . However, there is a huge potential for sufficiently accurate ASR to help in the objective analysis and understanding of classroom dynamics [ 10 , 11 , 12 ] . All of these challenges are complicated by the fact that classroom speech is a low-resource task [ 13 , 14 ] due to the protections around children [ 15 , 16 ] . This means great care must be taken to best utilize existing data. One of the largest resources of classroom audio data is the National Center for Teacher Effectiveness (NCTE) classroom speech corpus [ 17 ] . The NCTE dataset contains 5000 hours of transcribed classroom speech, but these transcripts are very inaccurate as they were not intended for ASR training. They are not verbatim and all names of the students and teachers were de-identified to protect their privacy. These transcripts have not been used, and a small number of recordings were accurately re-transcribed using human gold-standard transcriptions [ 18 ] . While Whisper has shown that inaccurate transcripts are useful if paired with sufficient accurate transcripts, it is not clear if imprecise transcripts in the NCTE dataset would be helpful, given that they far outnumber precise transcripts. In this study, we explore how to best utilize these weak transcriptions to improve the performance of classroom ASR systems. We preface our experiments using the NCTE corpus with controlled experiments on synthetically corrupted transcripts from the TEDLIUM-3 [ 19 ] corpus. We propose Weakly Supervised Pretraining (WSP), a weakly supervised learning paradigm where we initially use imprecise transcription as an intermediate supervised step, followed by fine-tuning on a limited amount of gold-standard data. Our results indicate that this WSP paradigm is very effective in increasing the utility of limited gold-standard data, outperforming previous methods with limited classroom data. 2 Datasets 2.1 TEDLIUM We use the TEDLIUM 3 [ 19 ] dataset for our synthetic experiments. TEDLIUM is a collection of transcribed recordings of TED talks amounting to 540 hours from 2028 speakers. We chose TEDLIUM as it is a high-quality and public dataset that provides a good benchmark for our experiments. Unlike Librispeech [ 20 ] , TED talks are a better representation of everyday spontaneous speech, making them more suitable for experiments with transcription errors. 2.2 NCTE The NCTE dataset consists of video and audio recordings of 2128 4th and 5th-grade elementary math classrooms [ 17 ] . Each classroom is recorded with 2 to 3 microphones that range from lanyards worn by teachers capturing near-field speech to far-field stationary microphones placed in the corner of the classroom. In total, the dataset contains 5235 hours of recordings. 2.2.1 NCTE-Weak The majority of these recordings were transcribed, but they were not intended for ASR tasks. These transcriptions contain a large number of substitutions and deletions, and all the names of students and teachers were omitted to protect their privacy. Most importantly, the transcripts were not properly time-stamped. Loose inaccurate minute marks are provided that are often off by tens of seconds. In addition, in many instances, utterances have been labeled as “unintelligible” or “side conversation” that are still perfectly intelligible to both ASR models and humans. Thus far, these transcriptions have not been used for ASR training, and previous works only utilized them for analysis [ 17 ] or for training n-gram language models (LMs) for beam-search decoding [ 18 ] . However, in this research, we attempt to utilize these transcriptions as weak transcriptions in intermediate training. We call this dataset NCTE-Weak . We pass the dataset through a forced aligner [ 21 , 22 ] to obtain more accurate timestamps for preprocessing the dataset before ASR training. While the forced

Read the complete text