Identifying and typifying demographic unfairness in phoneme-level embeddings of self-supervised speech recognition models

Felix Herron, Solange Rossato, Alexandre Allauzen, François Portet · 2026

Read Identifying and typifying demographic unfairness in phoneme-level embeddings of self-supervised speech recognition models by Felix Herron, Solange Rossato, Alexandre Allauzen, François Portet (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2604-22631.txt.

Introduction

Herron and three colleagues typify two kinds of phoneme error in speech recognition systems. Modern speech recognition systems work better for certain speaker groups than others. One barrier to fairer systems is understanding errors, particularly embedding differences between high- and low-performance groups. The framework distinguishes random error, high phoneme variance, from systematic error, embedding bias. Training phoneme probes only on a single disadvantaged group sometimes improves performance for that group, evidence of group-level bias in phoneme embeddings. Speakers and groups with higher phoneme variance match lower-performance groups.

Catalog record

CREATORFelix Herron, Solange Rossato, Alexandre Allauzen, François Portet
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE45019
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2604.22631

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Identifying and typifying demographic unfairness in phoneme-level embeddings of self-supervised speech recognition models Felix Herron, Solange Rossato, Alexandre Allauzen, François Portet Abstract Modern automatic speech recognition (ASR) systems have been observed to function better for certain speaker groups (SGs) than others, despite recent gains in overall performance. One potential impediment to progress towards fairer ASR is a more nuanced understanding of the types of modeling errors that speech encoder models make, and in particular the difference between the structure of embeddings for high-performance and low-performance SGs. This paper proposes a framework typifying two types of error that can occur in modeling phonemes in ASR systems: random error/high variance in phoneme embedding, vs systematic error/embedding bias. We find that training phoneme classification probes only on a single, typically disadvantaged SG, sometimes improves performance for that SG, which is evidence for the existence of SG-level bias in phoneme embeddings. On the other hand, we find that speakers and SGs with higher levels of phoneme variance are the same as those with worse phoneme prediction accuracy. We conclude that both types of error are present in phoneme embeddings and both are candidate causes for SG-level unfairness in ASR, though random error is likely a greater hindrance to fairness than systematic error. Furthermore, we find that finetuning encoder models using a fairness-enhancing algorithm (domain enhancing and adversarial training) changes neither the benefits of in-domain phoneme classification probe training, nor measured levels of random embedding error. 1 Introduction Automatic speech recognition (ASR) systems have been repeatedly observed to perform better for certain speaker groups (SGs) than for others 56 ; 2 ; 18 ; 49 ; 55 ; 4 . Inasmuch there has been ample research in trying to improve ASR fairness 14 ; 32 ; 1 ; 44 ; however, resulting fairer models leave ample room for improvement. One potential drawback to existing fairness enhancing studies is that they tend to treat ASR systems like black boxes - it is hard to design an intervention to reduce demographic unfairness without better understanding the mechanisms involved in engendering unfairness to begin with. This paper provides a framework for understanding two sources of modeling error with respect to unfair SG-level treatment (see Figure 1 ): embedding bias/systematic error , in which phonemes are modeled around different modes depending on SG; vs unequal variance/random error , in which phonemes are modeled around the same modes but with more noise for one SG than another. We perform probing experiments on latent embeddings of state of the art attention-based speech encoder models, in particular on ASR-finetuned self-supervised speech processing models (S3Ms), to evaluate the bias and variance of phoneme embeddings across SGs. First, we evaluate whether phoneme recognition (PR) probes trained on a single SG perform better on unseen speakers from that same SG and worse for unseen speakers from unseen SGs. We find statistically significant improvement for in-domain training for some but not all SGs, at some but not all layers, for some but not all phonemes, thus indicating the existence of embedding bias in phoneme embedding at those layers for those models. However, we find that regardless of training data, the same SGs almost always perform best or worst respectively on PR, thus indicating the prevalence of higher levels of embedding variance for some SGs. Second, we directly calculate the variance in phoneme embeddings using a k-nearest neighbors (KNN) distance heuristic. We find that the same SGs which experience worse PR performance also have higher variance in PR, and there is statistically significant correlation between the two metrics for every SG and every speech encoder model. Finally, we investigate the difference between how phonemes are embedded for different SGs when we finetune for ASR using the fairness enhancing algorithm Domain Enhancing/Adversarial Training (DET/DAT). We find that it changes neither the relative KNN distance between phonemes, nor the benefit achieved from in-domain PR training. Thus, we conclude that to the extent that KNN distance is a useful indicator of precise phoneme modeling, that DET/DAT, as constituted in our study, is not useful at rendering the phoneme embedding space fairer. 2 Related work in interpretability of speech processing systems 39 ; 40 show that certain types of information inherent to or contained within an utterance, such as speaker identity or word meaning, are maximized at different model layers of S3Ms. 41 use explainable AI methods to show the impact of linguistic and paralinguistic features on transcription in Wav2vec 2.0 models. 34 find that artificially generated non-native speech is represented in a similar manner to real non-native speech in ASR systems. This suggests that ASR systems model non-native speech in a systematically different and reproducible manner, and anecdotally supports the embedding bias hypothesis (see Figure 1 ). 18 ; 19 find many of the same phonemes are hardest for ASR systems to understand regardless of SG or ASR architecture. 31 ; 37 show that phoneme information and speaker information are modeled in orthogonal subspaces of S3M embeddings. This would suggest that lack of demographic parity in ASR is unlikely due to speaker-specific phoneme modeling differences, providing anecdotal support of the higher variance hypothesis . 22 shows that layers of pretrained self-supervised speech processing models (S3Ms) that are best capable of modeling ASR are also the least fair, and that this effect is not attenuated after ASR finetuning. Furthermore, they show that even S3Ms pretrained on non-English corpora display the same biases as S3Ms pretrained only on English. These findings motivate the notion that certain speakers are inherently harder to model, regardless of training data and model architecture. Finally, 24 shows that pretrained S3Ms model certain SGs in a differentiable manner on an utterance scale, though to a lesser extent after ASR finetuning; 42 shows similar results for accent encoding. Other studies have delved into ASR errors due to hallucination in decoding 30 ; 20 ; 8 ; 27 . Our study doesn’t touch on this - we consider only speech encoders , and perform PR using a single “decoder” linear layer to map from embedding space to phoneme space. Furthermore, we simplify the task by avoiding the challenges of sequence-to-sequence mapping, instead focusing on single isolated phoneme embeddings. However, it is important to note that the two types of encoding error which we study are not the only potential sources of error in ASR. 3 Motivation Figure 1: Toy visualization of high variance vs embedding bias in by-SG modeling of phonemes, along with corresponding linear separators trained on the SG of the corresponding cover (orange trained on both SGs). If, compared to an equal-error world (upper left), the S3M embeds phonemes from SG ’B’ with higher variance than SG ’A’ (upper right), the learned ASR head won’t necessarily learn a different transformation (thought it might due to the training data being less noisy) but it will return higher error rate for SG ’B’. However, if phoneme 1 is modeled about different modes depending on SG, representing embedding bias (lower left), this will bias the PR head towards the more heavily represented SG in the training set. The objective of this paper is to examine the differences between how S3Ms model phonemes of various SGs. In an ideal world for ASR,

Read the complete text