CTC-DRO: Robust Optimization for Reducing Language Disparities in Speech Recognition

Martijn Bartelds, Ananjan Nandi, Moussa Koulako Bala Doumbouya, Dan Jurafsky Tatsunori Hashimoto Karen Livescu · 2025

Read CTC-DRO: Robust Optimization for Reducing Language Disparities in Speech Recognition by Martijn Bartelds, Ananjan Nandi, Moussa Koulako Bala Doumbouya, Dan Jurafsky Tatsunori Hashimoto Karen Livescu (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2502-01777.txt.

Introduction

Speech recognition scores well on average yet fails some languages badly, and the usual fix for weak groups misfires here. Martijn Bartelds and co-authors present a corrected fix in this 2025 paper. The standard method trains toward the worst group's loss. In speech, that loss grows with input length and shifts with sound patterns, so groups look worse for the wrong reasons.

Their method, CTC-DRO, smooths how group weights update, so one stubborn group cannot dominate. Batches are also matched by input length to cancel the scaling quirk. Tests covered speech recognition across five languages. Fairer training could narrow the gap between major and minority tongues. The work targets a known blind spot in robust learning.

Catalog record

CREATORMartijn Bartelds, Ananjan Nandi, Moussa Koulako Bala Doumbouya, Dan Jurafsky Tatsunori Hashimoto Karen Livescu
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE83 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2502-01777

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

CTC-DRO: Robust Optimization for Reducing Language Disparities in Speech Recognition Martijn Bartelds, Ananjan Nandi, Moussa Koulako Bala Doumbouya, Dan Jurafsky Tatsunori Hashimoto Karen Livescu Abstract Modern deep learning models often achieve high overall performance, but consistently fail on specific subgroups. Group distributionally robust optimization ( group DRO ) addresses this problem by minimizing the worst-group loss, but it fails when group losses misrepresent performance differences between groups. This is common in domains like speech, where the widely used connectionist temporal classification (CTC) loss not only scales with input length but also varies with linguistic and acoustic properties, leading to spurious differences between group losses. We present CTC-DRO , which addresses the shortcomings of the group DRO objective by smoothing the group weight update to prevent overemphasis on consistently high-loss groups, while using input length-matched batching to mitigate CTC’s scaling issues. We evaluate CTC-DRO on the task of multilingual automatic speech recognition (ASR) across five language sets from the diverse ML-SUPERB 2.0 benchmark. CTC-DRO consistently outperforms group DRO and CTC-based baseline models, reducing the worst-language error by up to 47.1 % 47.1\% and the average error by up to 32.9 % 32.9\% . CTC-DRO can be applied to ASR with minimal computational costs, and, while motivated by multilingual ASR, offers the potential for reducing group disparities in other domains with similar challenges. 1 Introduction State-of-the-art deep learning models are often highly accurate on training data populations, while consistently underperforming on specific subpopulations or groups ( Hashimoto et al., 2018 ; Duchi et al., 2023 ) . One practical setting where this issue has very large effects is multilingual automatic speech recognition (ASR), where performance varies substantially between languages ( Radford et al., 2023 ; Pratap et al., 2024 ; Shi et al., 2024 ) . Such models, which jointly perform language identification (LID) and ASR in many languages, could help improve accessibility and increase digital participation for speakers worldwide ( Besacier et al., 2014 ) . Distributionally robust optimization ( DRO ), particularly group DRO ( Sagawa et al., 2020 ) , has the potential to mitigate language disparities in multilingual ASR. Group DRO improves group robustness by up-weighting high-loss groups during training, and has been shown to outperform other approaches where the goal is to achieve high performance, even on the worst-performing group ( Koh et al., 2021 ) . However, it requires comparable training losses between groups to perform well ( Oren et al., 2019 ; Sagawa et al., 2020 ) , and this condition is often not met in ASR model training, because of differences in input length and speaker and acoustic characteristics across language-specific datasets. In this paper, we focus on a training approach that has been successful on multilingual ASR benchmarks: pre-trained self-supervised models fine-tuned with the connectionist temporal classification (CTC; Graves et al., 2006 ) objective ( Rouditchenko et al., 2023 ; Chen et al., 2024 ; Pratap et al., 2024 ) . CTC-based models built on encoders such as XLS-R ( Babu et al., 2022 ) and MMS ( Pratap et al., 2024 ) are widely adopted and offer advantages over autoregressive models like Whisper ( Radford et al., 2023 ) , including faster inference and reduced hallucinations ( Koenecke et al., 2024 ; Peng et al., 2024 ) , which are crucial for many downstream applications. However, differences in CTC-based training losses due to length, speaker, and acoustics may lead to varying magnitudes and irreducible components of losses across different groups. As a result, the group DRO weights do not have the desired effect. To address these issues, we present CTC-DRO , which optimizes a generalization of the group DRO objective, specifically by smoothing the up-weighting of high-loss groups. This new objective prevents overemphasis on groups with consistently and disproportionately high training losses. Also, we use length-matched group losses to mitigate the scaling properties of CTC. We evaluate CTC-DRO using language sets randomly selected from the ML-SUPERB 2.0 ( Shi et al., 2024 ) benchmark collection, which includes multilingual speech data from 15 diverse corpora across multiple domains, speaking styles and recording conditions. In this setting, CTC-DRO models outperform both group DRO and CTC-based baseline models across five language sets, regardless of whether balanced or unbalanced amounts of training data per language are used during training. Specifically, CTC-DRO models reduce the error rate of the worst-performing language in all of the five sets, with improvements of up to 47.1\% , while also improving the average performance across all languages by up to 32.9\% . While motivated by multilingual ASR, CTC-DRO offers the potential for reducing group disparities in other domains with incomparable training losses between groups, such as medical applications ( Ganz et al., 2021 ; Petersen et al., 2023 ) . Our code and newly trained models are publicly available at https://github.com/Bartelds/ctc-dro . 2 Background 2.1 Group DRO Given a family of models \Theta , loss function \ell and training data (x,y) drawn from empirical distribution \hat{P} , the standard training procedure for label prediction involves minimizing the expected loss over the training data: \min_{\theta\in\Theta}\mathbb{E}_{(x,y)\sim\hat{P}}\left[\ell(\theta;(x,y))\right]. (1) In contrast, group DRO aims to minimize the worst-case expected loss over a set of pre-defined groups or sub-distributions \{\hat{P}_{g}:g\in G\} in the training data: \min_{\theta\in\Theta}\Big\{\max_{g\in G}\mathbb{E}_{(x,y)\sim\hat{P}_{g}}\left[\ell(\theta;(x,y))\right]\Big\}. (2) Following Sagawa et al. (2020) , this objective can be rewritten as: \min_{\theta\in\Theta}\Big\{\sup_{q\in\Delta_{|G|}}\sum_{g\in G}q_{g}\mathbb{E}_{(x,y)\sim\hat{P}_{g}}\left[\ell(\theta;(x,y))\right]\Big\}, (3) where \Delta_{|G|} is the |G| -dimensional probability simplex, and q_{g} is a weight for group g\in G . Sagawa et al. (2020) propose an online algorithm to optimize this objective, treating the problem as a minimax game and interleaving gradient ascent updates on q=\{q_{g}:g\in G\} with gradient descent updates on \theta for training data mini-batches (see Algorithm 2 in Appendix C ). 2.2 CTC The CTC objective ( Graves et al., 2006 ) defines a method to learn a mapping between an input sequence X=(x_{1},x_{2},\ldots,x_{D}) and an output sequence Y=(y_{1},y_{2},\ldots,y_{U}) without requiring a known alignment between them, but assuming U\leq D and a monotonic alignment. CTC uses a blank output token \epsilon to handle x_{d}\in X that do not map to any output symbol. Consider \mathcal{Z} , which is the set of all sequences of length D that are composed of tokens from Y , and \epsilon . Each sequence Z\in\mathcal{Z} is a potential alignment between X and Y . CTC defines a collapsing function that merges consecutive, identical symbols and removes \epsilon in an alignment Z . The set of alignments Z\in\mathcal{Z} that collapse to Y using this function forms the set of valid alignments \mathcal{A}(X,Y) . For example, a possible alignment Z\in\mathcal{A}(X,Y) for D=2U+2 could be: [\epsilon,y_{1},\epsilon,y_{2},y_{2},\epsilon,\ldots,\epsilon,y_{U},\epsilon] . The conditional probability P_{CTC}(Z|X) for any alignment Z is computed as: P_{CTC}(Z|X)=\prod_{d=1}^{D}p(z_{d}|X), (4) where Z=(z_{1},z_{2},\ldots,z_{D}) and p(z_{d}|X) is the model’s predicted probability for symbol z_{d}\in Z at time d . The predicted probability of the output sequence Y , P_{CTC}(Y|X) , is then computed by marginalizing over valid alignments Z\in\mathcal{A}(X,Y) : P_{CTC}(Y|X)=\sum_{Z\in\mathcal{A}(X,Y)}P_{CTC}(Z|X). (5) The CTC loss function for (X,Y) is then defined as: \mathcal{L}_{CTC}=-\log P_{CTC}(Y\mid X). (6) 2.3 Limitations of Group DRO Applied to CTC The CTC loss, as defined in Equation 6 , scales with the length of the input sequence D and the length of the output sequence U . This

Read the complete text