A Sociophonetic Analysis of Racial Bias in Commercial ASR Systems Using the Pacific Northwest English Corpus

2025

Read A Sociophonetic Analysis of Racial Bias in Commercial ASR Systems Using the Pacific Northwest English Corpus (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2510-22495.txt.

Introduction

Commercial speech recognition systems may not hear all speakers equally well. Michael Scott, Siyu Liang, Alicia Wassink, and Gina-Anne Levow test that concern in a 2025 paper, using recordings of Pacific Northwest English from speakers of four ethnic backgrounds.

Scott and co-authors measure errors in four major commercial systems and link them to eleven speech features. A new phonetic error rate ties mistakes to specific sound patterns rather than word choice. Vowel quality differences show the strongest link to unequal errors, with the largest effects for African American speakers. The findings point to training data that better covers dialect sound variation.

Catalog record

YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE36 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2510-22495

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

A Sociophonetic Analysis of Racial Bias in Commercial ASR Systems Using the Pacific Northwest English Corpus Abstract This paper presents a systematic evaluation of racial bias in four major commercial automatic speech recognition (ASR) systems using the Pacific Northwest English (PNWE) corpus. We analyze transcription accuracy across speakers from four ethnic backgrounds (African American, Caucasian American, ChicanX, and Yakama) and examine how sociophonetic variation contributes to differential system performance. We introduce a heuristically-determined Phonetic Error Rate (PER) metric that links recognition errors to specific linguistically motivated variables derived from sociophonetic annotation. Our analysis of eleven sociophonetic features reveals that vowel quality variation, particularly resistance to the low-back merger and pre-nasal merger patterns, is systematically associated with differential error rates across ethnic groups, with the most pronounced effects for African American speakers across all evaluated systems. These findings demonstrate that acoustic modeling of dialectal phonetic variation, rather than lexical or syntactic factors, remains a primary source of bias in commercial ASR systems. The study establishes the PNWE corpus as a valuable resource for bias evaluation in speech technologies and provides actionable guidance for improving ASR performance through targeted representation of sociophonetic diversity in training data. Keywords: automatic speech recognition, evaluation, speech corpus, bias, sociophonetics, dialectal variation A Sociophonetic Analysis of Racial Bias in Commercial ASR Systems Using the Pacific Northwest English Corpus Michael Scott, Siyu Liang, Alicia Wassink, Gina-Anne Levow Department of Linguistics, University of Washington [email protected], {liangsy, wassink, levow}@uw.edu Abstract content 1. Introduction Automatic Speech Recognition (ASR) systems demonstrate significantly higher error rates for speakers from marginalized communities. Koenecke et al. 2020 found that commercial ASR systems exhibit average word error rates of 35% for African American speakers compared to 19% for Caucasian American speakers across five major commercial systems. This disparity represents a critical fairness issue as speech technologies become increasingly ubiquitous in everyday applications, from virtual assistants to automated closed captioning, medical dictation, and educational technology ( Cunningham et al. 2024 ) . While these disparities are well-documented across multiple studies ( Tatman 2017 ; Tatman and Kasten 2017 ; Martin and Wright 2023 ) , the underlying linguistic mechanisms remain underexplored. Wassink et al. 2022 provided crucial evidence that phonetic rather than lexical or syntactic factors drive these differences, showing that ASR systems exhibit higher error rates for African American English even when lexical content is controlled. This finding suggests that acoustic modeling, rather than language modeling, is the primary source of bias — a hypothesis further supported by subsequent investigations ( Chang et al. 2024 ; Johnson et al. 2024 ) . Recent work has begun to address these disparities through various approaches. Dorn 2019 explored dialect-specific models and targeted training strategies, while others like Zhang et al. 2023 ; Wang et al. 2024 ; Li et al. 2024 have investigated how large-scale multilingual models and self-supervised learning systems handle dialectal variation. Studies using diverse corpora such as AfriSpeech-200 have demonstrated the potential to improve ASR performance across African accents through better data representation ( Olatunji et al. 2023 ) . However, these studies often focus on model architectures or training strategies rather than systematically identifying which specific phonetic features of the underrepresented dialect contribute to transcription errors. Understanding these linguistic mechanisms is essential for developing effective mitigation strategies. This paper contributes a systematic evaluation methodology for measuring ASR bias using the Pacific Northwest English (PNWE) corpus ( Wassink 2007 ) , focusing on eleven sociophonetic variables that exhibit variation across the ethnic communities represented in the corpus. We evaluate four major commercial ASR systems and demonstrate that specific phonetic features, particularly resistance to the low-back merger and variation in pre-nasal /æ/ patterns, are disproportionately associated with transcription errors for African American speakers. Our approach extends existing ASR evaluation frameworks by incorporating detailed sociolinguistic analysis, providing both traditional word error rates and a novel heuristically-determined phonetic error rate that links errors to specific phonological features documented in the sociolinguistic literature ( Thomas 2007 ; Green 2002 ) . By identifying the specific phonetic characteristics to evaluate current ASR systems, this work provides actionable guidance for improving system performance and emphasizes the PNWE corpus as a valuable resource for bias evaluation in speech technologies. 2. Related Work 2.1. Racial Disparities in Commercial ASR Systems The work of Koenecke et al. 2020 demonstrated substantial racial disparities in commercial ASR systems, finding almost double average word error rates for African American speakers compared to Caucasian American speakers across five major systems. Tatman 2017 ; Tatman and Kasten 2017 further confirmed these disparities across diverse datasets and speech contexts. These errors have real-world impacts, with users often feeling “othered” by technology that fails to recognize their speech ( Cunningham et al. 2024 ) . However, most prior work had focused on documenting disparities rather than identifying specific linguistic mechanisms driving differential performance. 2.2. Linguistic Factors in ASR Performance Disparities Wassink et al. 2022 provided crucial evidence that phonetic rather than lexical or syntactic factors drive performance differences for African American English speakers, showing that ASR systems exhibit higher error rates for African American speech even when lexical content is controlled. This finding clarifies that the disparity arises primarily from acoustic modeling rather than from language modeling, as text-based models are not the focus of this comparison. Taken together with the findings of Koenecke et al. 2020 , this implicates acoustic modeling as the primary source of bias. Sociolinguistic research has extensively documented systematic phonetic variation in AAL ( Thomas 2007 ; Green 2002 ) , including variable low-back merger, pre-nasal /æ/ raising, /ay/ monophthongization, and consonant reduction – features that create mismatches with models trained on Mainstream American English. Recent investigations of newer architectures reveal persistent challenges. Chang et al. 2024 showed that self-supervised speech representations continue to struggle with AAL, while Li et al. 2024 highlighted confounding factors like audio quality in performance assessments. Despite advances in large-scale multilingual systems ( Zhang et al. 2023 ) and dialect-specific models ( Dorn 2019 ) , few studies systematically connect specific sociophonetic variables to ASR errors. Our work addresses this gap through the quantitative analysis of eleven phonetic features across multiple commercial systems. 2.3. Resources for ASR Bias Evaluation Evaluating ASR fairness requires corpora with demographically diverse speakers and controlled recording conditions. The Corpus of Regional African American Language ( Kendall and Farrington 2018 ) provides extensive AAL recordings, although the audio quality varies. Mozilla Common Voice ( Ardila et al. 2020 ) includes demographic metadata and has recently begun adding spontaneous speech, though the release used here primarily contains read prompts that may miss naturalistic variation. The Speech Accent Archive ( Weinberger 2013 ) focuses on L2 English, while the Artie Bias Corpus ( Meyer et al. 2020 ) has limited sample sizes. AfriSpeech-200 ( Olatunji et al. 2023 ) offers 200 hours across 120 African accents but targets clinical domains rather than sociolinguistic variation. The Pacific Northwest English corpus ( Wassink 2007 ) offers distinct advantages: balanced sampling across five ethnic groups within one region, sociolinguistic interviews eliciting multiple speech styles, consistent high-quality recordings, detailed sociophonetic annotation, and sufficient sample sizes for statistical analysis. Our work validates PNWE for bias evaluation while demonstrating how to connect performance disparities to

Read the complete text