Lost in Transcription: Subtitle Errors in Automatic Speech Recognition Reduce Speaker and Content Evaluations CCS: Human-centered computing Empirical studies in HCI CCS: Human-centered computing HCI design and evaluation methods

Kowe Kadoma, Priyal Shrivastava, Mor Naaman · 2026

Read Lost in Transcription: Subtitle Errors in Automatic Speech Recognition Reduce Speaker and Content Evaluations CCS: Human-centered computing Empirical studies in HCI CCS: Human-centered computing HCI design and evaluation methods by Kowe Kadoma, Priyal Shrivastava, Mor Naaman (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2603-15807.txt.

Introduction

Kadoma, Shrivastava, and Naaman test how bad subtitles change viewer judgment. Speech tools err more for some accents than for others. The team ran a preregistered online study with 207 U.S. crowdworkers. Viewers watched accented talks with clean or error-prone subtitles. Faulty subtitles lowered scores for speakers and for content alike. The drop hit all accent groups evenly in the study. Speakers whose accents trip ASR tools may still face harsher viewer scores. Error-prone captions act as a second penalty layer.

Catalog record

CREATORKowe Kadoma, Priyal Shrivastava, Mor Naaman
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE39769
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2603.15807

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Lost in Transcription: Subtitle Errors in Automatic Speech Recognition Reduce Speaker and Content Evaluations CCS: Human-centered computing Empirical studies in HCI CCS: Human-centered computing HCI design and evaluation methods Kowe Kadoma, Priyal Shrivastava, Mor Naaman Abstract . Researchers have demonstrated that Automatic Speech Recognition (ASR) systems perform differently across demographic groups. In this work, we examined how subtitle errors affect evaluations of speakers and their content using a preregistered online experiment (N=207, U.S.-based crowdworkers). Participants watched speakers with various accents deliver a talk in which the subtitles were accurate or error-prone. Our results indicate that error-prone subtitles consistently reduce both speaker and content evaluations for all speakers. We did not see disparate impact between the accent groups, controlling for subtitle quality. Taken together, though, the findings of this short paper imply that speakers with accents for which ASR systems perform poorly are likely to be further penalized by viewers with lower evaluations. 1. Introduction Automatic speech recognition (ASR) systems convert spoken language into text using signal processing and machine learning ( 38 ; 55 ) . These systems now power increasingly ubiquitous technologies, from voice assistants like Apple Siri and Amazon Alexa to transcription services and automatic subtitles on platforms like YouTube, Zoom, and Microsoft Teams. Automatic subtitles facilitate accessibility, enable cross-linguistic communication, and support users in noisy environments. Beyond convenience, they play a critical role in providing equitable access to information for individuals who are deaf or hard of hearing ( 30 ; 3 ; 14 ) as well as non-native speakers ( 50 ) . Despite their promise, ASR systems are not error-free, and their accuracy is not distributed evenly across different users. ASR systems often exhibit biases, or systematic differences in performance, with respect to race ( 28 ; 15 ; 22 ) , gender ( 25 ; 57 ; 26 ) , and dialect ( 23 ; 57 ) . For example, researchers found that the embeddings (or word associations) in the pre-trained speech models of ASR systems often associated positive words with abled individuals over those with disabilities, White Americans over Black Americans, men over women, and U.S. accents over non-U.S. accents ( 51 ) . These performance differences have real consequences for the individuals whose speech is being analyzed ( 22 ; 62 ; 33 ; 27 ) . Wenzel et. al ( 62 ) found that when ASR errors occur, Black speakers experience significantly higher levels of self-consciousness and lower self-esteem compared to White speakers. In the context of automatic subtitles for Q&A sessions in conferences, Li et. al ( 33 ) demonstrated that non-native English speakers experience frustration and confusion when subtitle errors occur. Furthermore, non-native English speakers were concerned about the potential social stigma that comes with using ASR tools. Participants in the study expressed concern that using an ASR tool could lead the audience to perceive them as less competent, hinting at the importance of how the audience perceives the speaker ( 33 ) . If an ASR tool can affect how the audience perceives the speaker, then individuals whose accents produce more ASR errors may face a compounded disadvantage. Not only are their talks more difficult to understand and less enjoyable due to subtitle errors ( 7 ; 56 ) , but their clarity and competence are also undermined ( 33 ) , which could lead to further negative consequences in professional and social settings. In this short paper, we examine the relationship between subtitle quality and evaluations of the speaker and the content through a carefully designed online experiment. Our work builds on Kim et al. ( 27 ) , who compared subtitle errors across native and non-native speakers. Their design, however, did not control for confounders like accent with speaker identity and appearance. In this work, we overcome this limitation by using generative AI to manipulate the accent while holding all other speaker characteristics constant. In our experiment, participants watched videos of broad-audience talks and evaluated the speaker and the content of each talk. Participants watched two videos each, which were randomly selected from a set of four videos. Our experimental manipulation could present each of these videos alongside accurate or error-prone subtitles (the first manipulated factor) with the speaker using a Standard American English (SAE) accent or a non-Standard American English (non-SAE) accent (the second factor). A non-SAE accent could refer to regional English varieties within the United States or non-standard or globally regional accents; however, in this work, we examine an English variety spoken in India. Our results show that subtitle errors negatively impact the evaluation of the speaker and the content. Given the widespread deployment of ASR systems across professional, educational, and social platforms, such errors may disadvantage a large number of speakers. The negative evaluations could be particularly concerning for individuals whose accents trigger higher error rates; however, we did not see direct evidence that the speaker’s accent further impacts their evaluations when subtitle errors occur. In addition to our empirical findings, our work offers a key methodological contribution in the form of a novel experimental design that controls for the speakers’ appearance. In total, we provide more robust evidence of the downstream effects of subtitle errors, highlighting the need to consider the potential adverse impacts of these technologies. 2. Background and Hypotheses Subtitles are widely used in asynchronous settings, such as movies, TV, and online platforms like YouTube, as well as in live, synchronous settings, such as video conferencing software like Zoom. While subtitles were originally introduced in film and television to support deaf and hard-of-hearing audiences ( 19 ) , research has shown that hearing viewers also benefit from subtitles through improved comprehension and engagement ( 7 ; 58 ; 29 ) . These benefits are often explained by the dual coding theory, which posits that people remember information more effectively when it is presented through both audio and visual channels ( 9 ) . The effectiveness of subtitles, however, hinges on their quality. Many factors impact subtitle quality, including accuracy, display size, and speed ( 7 ; 31 ; 48 ) . Researchers, user associations, and regulators have extensively studied subtitle quality, specifically subtitle accuracy, finding that inaccurate subtitles reduce the audience’s comprehension and engagement ( 2 ; 1 ; 60 ) . Inaccurate subtitles do more than limit audience understanding—they can potentially negatively impact the speaker. ASR-generated subtitles are often less accurate than human-generated subtitles ( 47 ; 46 ; 7 ) , meaning that speakers in video lectures, interviews, or live calls may be unfairly penalized by errors outside their control. ASR systems are often less accurate due to biases that affect the system’s performance ( 4 ; 17 ) . Biases occur when the model produces different outcomes for individuals belonging to different subgroups (e.g., gender or race) despite having the same ground truth (or human-verified scores). Many biases are seen as byproducts of the models’ construction and, in particular, the training data used ( 49 ; 61 ) . It is important to note that while training data is often seen as a source of bias, it is not the only source of bias in the machine learning life cycle. For example, bias could emerge when

Read the complete text