Evaluating ASR robustness to spontaneous speech errors: A study of WhisperX using a Speech Error Database

John, Macarious Kin Fung, Aanchan · 2025

Read Evaluating ASR robustness to spontaneous speech errors: A study of WhisperX using a Speech Error Database by John, Macarious Kin Fung, Aanchan (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2508-13060.txt.

Introduction

John Alderete and two co-authors turn a speech error trove into a test for voice software in a 2025 paper. The trove holds thousands of slips from real English talk, each tagged with what the speaker meant and what came out. The tags sort slips by sound level, word damage, self-fixes, and place in the word.

The team ran a top speech-to-text system over 5,300 of the slips. Its misses lined up with the tag classes in telling ways. A bank built for language science thus doubles as a sharp tool for grading speech software where real talk, not clean scripts, is the test.

Catalog record

CREATORJohn, Macarious Kin Fung, Aanchan
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE21 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2508-13060

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Evaluating ASR robustness to spontaneous speech errors: A study of WhisperX using a Speech Error Database John, Macarious Kin Fung, Aanchan Abstract The Simon Fraser University Speech Error Database (SFUSED) is a public data collection developed for linguistic and psycholinguistic research. Here we demonstrate how its design and annotations can be used to test and evaluate speech recognition models. The database comprises systematically annotated speech errors from spontaneous English speech, with each error tagged for intended and actual error productions. The annotation schema incorporates multiple classificatory dimensions that are of some value to model assessment, including linguistic hierarchical level, contextual sensitivity, degraded words, word corrections, and both word-level and syllable-level error positioning. To assess the value of these classificatory variables, we evaluated the transcription accuracy of WhisperX across 5,300 documented word and phonological errors. This analysis demonstrates the database’s effectiveness as a diagnostic tool for ASR system performance. Alderete Hui Mohan Simon Fraser UniversityCanada Northeastern University, Vancouver, BCCanada keywords automatic speech recognition, model evaluation, benchmarks, model training, speech errors, psycholinguistics, contextual sensitivity † † email: [email protected], [email protected], [email protected] 1 Introduction Automatic speech recognition (ASR) technology relies heavily on high-quality datasets for both training and evaluation. Current datasets effectively capture many dimensions of speech complexity, including variations in speech style (read versus spontaneous), accent, speaker characteristics, noise conditions, and linguistic genres [ 1 , 2 ] . However, these datasets operate under an implicit assumption that transcriptions are free of human-produced errors—an assumption that overlooks a fundamental characteristic of natural speech. Research has consistently shown that spontaneous speech contains speech errors, or slips of the tongue, which represent deviations from intended speech plans [ 3 , 4 ] . Consequently, ASR systems are sometimes trained and evaluated using these human-produced errors as ground truth, potentially impacting model development and assessment. Speech errors occur approximately once or twice a minute in natural conversation [ 5 ] , making them a significant factor in spoken language processing. At typical speaking rates of 150 words per minute [ 6 ] , speech errors affect approximately 1% of utterances. Recent research has shown that training data containing even small amounts of human transcription errors leads to a two-fold increase in Word Error Rate (WER) [ 7 , 8 ] , indicating the importance of accurate speech error detection in ASR training data. Given the current focus on WER reduction in end-to-end models [ 9 ] , addressing the problem of speech errors may represent low-hanging fruit for model improvement. Beyond model performance, speech errors provide a valuable test case for ASR robustness. While ASR research has traditionally focused on accommodating linguistic variation and degraded speech signals [ 10 ] , speech errors present unique challenges. These include mid-word interruptions, phoneme additions, deletions, and substitutions, as well as word substitutions that lack explicit phonetic cues to the speaker’s intended utterance. Evaluating models against these natural distortions provides insight into their capacity to handle extreme cases of signal variation. Speech errors also exhibit systematic patterns that provide unique testing opportunities for ASR models. For example, most sound errors are contextual in the sense that they involve substitutions of sounds that occur in the neighboring linguistic context [ 11 ] . This context dependency provides an opportunity to investigate the local context encoding and forward prediction capabilities in ASR models, particularly in streaming applications where future context is unavailable. Understanding ASR model performance on speech errors also has broader implications for processing impaired speech, such as aphasic speech, where error rates are substantially higher [ 12 ] . Advances in error transcription could therefore directly benefit assistive speech technologies. Figure 1 : Recognition lattice with speech errors. Finally, labeled speech error datasets enable the construction of recognition lattices that incorporate both the produced error and the intended target word. Figure 1 illustrates this lattice structure with the error word Dad and the intended word Mom . The resulting lattice supports three potential transcription paths: faithful error reproduction ( Dad ), error correction to the intended word ( Mom ), or alternative high-probability candidates ( Tom ). While traditional sequence discrimination models utilizing the lower two lattice paths have achieved significant WER reductions [ 13 , 14 ] , the inclusion of the error path creates new optimization possibilities. Models can be tuned to prioritize either error correction by weighting intended word paths more heavily, or verbatim transcription by emphasizing actually produced error words. To demonstrate this approach, we analyzed WhisperX [ 15 ] transcriptions of 5,300 speech errors from SFUSED English [ 16 ] , the largest corpus of spontaneous speech errors derived from audio recordings. Our analysis examines not only overall transcription accuracy but also explores how error contexts and conditions affect model performance, leveraging SFUSED English’s rich cross-classification of linguistic variables. 2 Methods 2.1 SFUSED English The Simon Fraser University Speech Error Database (SFUSED) English contains 10,000 labeled speech errors recorded across 360 hours of spontaneous speech from third-party podcast series. The source material includes podcasts like The Astronomy Podcast and Rooster Teeth that were selected on the basis of the quantity of unscripted speech, gender balance, and high production quality. Each podcast episode is between 30-60 minutes long. The SFUSED English documentation describes speaker characteristics, data quality, error mark-up, workflows, and access to the audio [ 16 ] . 1 1 1 https://osf.io/8c9rg/ The audio data comes with marked up speech errors, but it does not come with human ground-truth transcriptions. As a result, we are unable to provide a baseline Word Error Rate. We extracted 5,300 speech errors from SFUSED English, cross-classified by several experimental conditions relevant to ASR testing. 2 2 2 The complete dataset and classification script described below are available at: https://osf.io/6x4se/. As shown in Table 1 , they differ in error type: word substitution errors (2,4) involve a complete replacement of the intended word for a different word, whereas sound errors (1,3) involve slight mispronunciations of an intended word, including sound substitutions, additions, and deletions. Word errors present a greater challenge for ASR models than sound errors because the intended and error words are entirely different and often there is no phonetic evidence for the intended word. In addition to labeled error words, SFUSED English’s analysis also includes a variable for the intended word, which is inferred either from corrected errors or the logic of the conversation (as in password in (4)). The SFUSED English documentation states that intended words can be ascertained in 98% of cases, and the remaining examples are labeled for low-confidence of the intended word. Both error types are categorized by the following experimental conditions: • Contextual influence : Error units may appear in the surrounding linguistic context, as in the bolded words in (1-3), or not (4) • Correction status : Errors may be corrected by the talker, as in (1-3), or not (4) • Completion status : Words may be aborted mid-production (i.e., incomplete), as (2) and (3), or not: (1) and (4) Contextual errors have the effect of doubling the error term in the local context. This may not have an impact on sound errors, but doubling in word errors may disrupt transcription in infelicitous positions (e.g., username in (4)). While corrected errors

Read the complete text