Towards Fair ASR for Second Language Speakers using Fairness Prompted Finetuning

Monorama Swain 1 , Bubai Maji 2 , Jagabandhu Mishra 3 , Markus Schedl 1 , Anders Søgaard 4 , Jesper Rindom Jensen 5 · 2025

Read Towards Fair ASR for Second Language Speakers using Fairness Prompted Finetuning by Monorama Swain 1 , Bubai Maji 2 , Jagabandhu Mishra 3 , Markus Schedl 1 , Anders Søgaard 4 , Jesper Rindom Jensen 5 (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2510-18374.txt.

Introduction

Monorama Swain and her five co-authors study fairness in automatic speech recognition for people who speak English as a second language. Their checks of two widely used systems, Whisper and Seamless-M4T, found large swings in word error rate across twenty-six accent groups. Such gaps mean some speakers are understood far worse than others by the same tool, even when they say similar things.

To narrow the gap, the team fine-tuned the systems with small adapter modules and extra training goals that reward even performance across groups. Three known fairness methods were blended with the standard training loss. The combined approach cut average word error rates sharply compared with the original large models, and it also beat plain fine-tuning, while keeping overall accuracy in place.

Catalog record

CREATORMonorama Swain 1 , Bubai Maji 2 , Jagabandhu Mishra 3 , Markus Schedl 1 , Anders Søgaard 4 , Jesper Rindom Jensen 5
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE17 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2510-18374

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Towards Fair ASR for Second Language Speakers using Fairness Prompted Finetuning Monorama Swain 1 , Bubai Maji 2 , Jagabandhu Mishra 3 , Markus Schedl 1 , Anders Søgaard 4 , Jesper Rindom Jensen 5 Abstract In this work, we address the challenge of building fair English ASR systems for second-language speakers. Our analysis of widely used ASR models, Whisper and Seamless-M4T, reveals large fluctuations in word error rate (WER) across 26 26 accent groups, indicating significant fairness gaps. To mitigate this, we propose fairness-prompted finetuning with lightweight adapters, incorporating Spectral Decoupling (SD), Group Distributionally Robust Optimization (Group-DRO), and Invariant Risk Minimization (IRM). Our proposed fusion of traditional empirical risk minimization (ERM) with cross-entropy and fairness-driven objectives (SD, Group DRO, and IRM) enhances fairness across accent groups while maintaining overall recognition accuracy. In terms of macro-averaged word error rate, our approach achieves a relative improvement of 58.7 % 58.7\% and 58.5 % 58.5\% over the large pretrained Whisper and Seamless-M4T, and 9.7 % 9.7\% and 7.8 % 7.8\% over them, finetuning with standard empirical risk minimization with cross-entropy loss. † † address: 1 Johannes Kepler University Linz, Austria, 2 IIT Kharagpur, India, 3 University of Eastern Finland, 4 University of Copenhagen, Denmark, 5 Aalborg University, Denmark 1 Introduction Speech technologies can be a driver of equality, making information accessible to social groups with limited access: dyslexics, non-literates, congenitally blind, children, the elderly, and language learners, if they work well across all social groups, including non-native speakers [ 1 ] . Performance drops for non-native speakers can often be traced back to the influence of the speaker’s mother tongue and its effect on pronunciation [ 2 ] , [ 3 ] . Accents may exhibit a lot of variation: In Indian English, for example, the /z/ sound is often replaced by /s/, especially in the speech of those whose native languages do not distinguish between the two. Moreover, very might be pronounced as [”Ve:Ri], using the retroflex /ó/ instead of the alveolar /r/ [ 4 ] . Automatic speech recognition (ASR) systems have achieved remarkable performance in high-resource languages such as English. Yet, several studies have shown that these systems exhibit unequal performance across demographic and accent groups [ 5 , 6 , 7 , 8 , 9 ] , a disparity further illustrated in Figure 1 . Such fairness gaps highlight the pressing need for systematic methodologies to evaluate and mitigate bias, aligning with the call in [ 9 ] that “Every voice matters.” A key reason for these disparities is the over-representation of mainstream accents in training and evaluation data [ 10 ] , leading to substantial performance gaps for underrepresented accents. Figure 1: Best WER (%) per accent group for Whisper models before fine-tuning. For each group, the WER shown corresponds to the lowest WER obtained among Whisper variants (i.e., tiny, base, small, medium, large) In this paper, we focus on improving the fairness and robustness of ASR across various English accents, with particular attention to second-language (L2) speakers. There has been some attempts already, however, to study fairness in ASR. For example, in [ 11 ] , a mixed-effects poisson regression has been introduced as a statistical approach to measure and interpret word error rate (WER) differences among subgroups. This method has been shown to address challenges such as handling unobserved heterogeneity across speakers and identifying the sources of WER gaps between subgroups. Similarly, [ 12 ] introduced counterfactual training methods for Connectionist Temporal Classification-based ASR, where demographic information such as gender, age, education are counterfactually modified while the linguistic content remains fixed. Their method helps to reduce the variance in character error rate across demographic groups. Recent work, such as [ 13 ] , has benchmarked ASR performance across languages and accents, highlighting performance disparities of foundational ASR models across accent groups. Similarly, [ 5 ] leveraged one-hot accent embeddings to improve adaptation to diverse speakers. However, prior studies have primarily focused on modeling accent variation, and to the best of our knowledge, none have explored fairness prompting as a means to fine-tune foundational models. Our work addresses this gap by explicitly targeting L2 English varieties and incorporating fairness disparities into the learning objective, thereby enabling fair finetuning . Our main contributions are: • We conducted systematic fairness analyzes targeting L2 English speakers, highlighting disparities that remain hidden when treating all accents uniformly. • We evaluated two families of off-the-shelf speech models (Whisper and SeamlessM4T) under standard vanilla fine-tuning (i.e., empirical risk minimization (ERM)) as the baseline with the fairness-promoting algorithms: Group-DRO; spectral decoupling (SD); and invariant risk minimization (IRM). • Finally, we propose a novel fusion of ERM, SD, Group-DRO, and IRM (see Figure 2 ) to evaluate its effectiveness in reducing performance gaps across 26 L2 English accents. In addition, we also analyze how model size and linguistic/data properties influence the effectiveness of fairness-promoting fine-tuning. 2 Methodology 2.1 Problem Statement Our goal is to minimize the disparities in ASR performance across 26 groups. Each group contains English utterances shaped by a distinct L2 corresponding to one native language. Thus, each sample consists of the information ( x , y , g ), where x\in\mathbb{R}^{T} is the speech sample, where T is the length of the input speech signal, y=(y_{1},\dots,y_{M}) is the English transcription sequence, and g\in\{1,\dots,26\} denotes the accent group. 2.2 ASR Model We adopt large-scale pretrained sequence-to-sequence ASR models, i.e., Whisper family and SeamlessM4T: Whisper [ 14 ] : A family set of five ASR models (tiny, base, small, medium, and large) 1 1 1 All Whisper model variants are available at https://huggingface.co/models?search=openai/whisper . , which are trained in the same way, but of different sizes, developed by OpenAI. The Whisper models are trained on 680,000 hours of data sourced from the web in 97 languages. During training, Whisper uses data augmentation techniques that transform audio spectrograms by applying methods like time warping, frequency masking, and time masking. These techniques enhance the model’s ability to generalize across various acoustic environments. Whisper is built as a traditional encoder-decoder system and is trained in a weakly supervised manner on unreleased multilingual audio data. To handle long audio, the system divides the audio into 30-second chunks for processing. SeamlessM4T [ 15 ] : is a model developed by MetaAI that supports speech-to-speech translation, speech-to-text translation, text-to-speech translation, text-to-text translation, and ASR for up to 100 languages. The model has two versions 2 2 2 Both Seamless models are available at https://huggingface.co/models?search=seamless . : Medium with 1.2 billion parameters and a size of 6.4 GB, and Large with 2.3 billion parameters and a size of 10.7 GB. Figure 2: The fairness-prompting speech recognition system. 2.3 Finetuning with ERM ERM [ 16 ] : The ERM baseline minimizes the empirical mean cross-entropy over the pooled training set, which helps to maximize the likelihood of the reference transcription. Accordingly, the ERM objective is formulated as: \mathcal{L}_{\text{ERM}}=\frac{1}{N}\sum_{n=1}^{N}\mathcal{L}^{(n)}_{\text{ASR}} (1) where N is the total number of training utterances and \mathcal{L}^{(n)}_{\text{ASR}} is the ASR loss (cross-entropy) for the n -th utterance. This standard training objective, which optimizes average loss, tends to favor majority groups and can result in higher WER for underrepresented accents. 2.4 Fairness-Promoting Finetuning To adapt the considered models fairly

Read the complete text