Fairness of Automatic Speech Recognition in Cleft Lip and Palate Speech

Susmita Bhattacharjee, Jagabandhu Mishra, H.S. Shekhawat, S. R. Mahadeva Prasanna · 2025

Read Fairness of Automatic Speech Recognition in Cleft Lip and Palate Speech by Susmita Bhattacharjee, Jagabandhu Mishra, H.S. Shekhawat, S. R. Mahadeva Prasanna (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2505-03697.txt.

Introduction

Susmita Bhattacharjee and co-authors study fairness in speech recognition for people with cleft lip and palate, in a 2025 paper. Their speech is often nasal in quality, which shifts sound patterns and lowers recognition accuracy. The head of the work frames the fix as mixing training speech by severity level.

The team defines a fairness score that balances overall error against the gap between groups. They test three recognition systems on English and Kannada datasets. Mixing typical speech with cleft speech, matched by severity, improved fairness on both datasets. Word error rates fell by more than ten points in the best cases. The authors argue severity aware mixing is a practical step toward fairer speech tools.

Catalog record

CREATORSusmita Bhattacharjee, Jagabandhu Mishra, H.S. Shekhawat, S. R. Mahadeva Prasanna
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE58 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2505-03697

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Fairness of Automatic Speech Recognition in Cleft Lip and Palate Speech Susmita Bhattacharjee, Jagabandhu Mishra, H.S. Shekhawat, S. R. Mahadeva Prasanna Abstract Speech produced by individuals with cleft lip and palate (CLP) is often highly nasalized and breathy due to structural anomalies, causing shifts in formant structure that affect automatic speech recognition (ASR) performance and fairness. This study hypothesizes that publicly available ASR systems exhibit reduced fairness for CLP speech and confirms this through experiments. Despite formant disruptions, mild and moderate CLP speech retains some spectro-temporal alignment with normal speech, motivating augmentation strategies to enhance fairness. The study systematically explores augmenting CLP speech with normal speech across severity levels and evaluates its impact on ASR fairness. Three ASR models—GMM-HMM, Whisper, and XLS-R—were tested on AIISH and NMCPC datasets. Results indicate that training with normal speech and testing on mixed data improves word error rate (WER). Notably, WER decreased from 22.64 % 22.64\% to 18.76 % 18.76\% (GMM-HMM, AIISH) and 28.45 % 28.45\% to 18.89 % 18.89\% (Whisper, NMCPC). The superior performance of GMM-HMM on AIISH may be due to its suitability for Kannada children’s speech, a challenge for foundation models like XLS-R and Whisper. To assess fairness, a fairness score was introduced, revealing improvements of 17.89 % 17.89\% (AIISH) and 47.50 % 47.50\% (NMCPC) with augmentation. 1 Introduction CLP is a congenital abnormality of the craniofacial region [ 1 , 2 , 3 ] . Globally, the incident of CLP was recorded to be 1,92,708 in 2019 [ 4 ] . The abnormality in the craniofacial region causes defects in the produced speech [ 1 , 2 , 3 , 5 ] . Mostly, due to the opening between the lip and nasal cavity, the produced speech is breathy and highly nasalized [ 6 ] , [ 7 ] . Further, the opening between, the oral and nasal cavity shifts the speech resonances [ 1 ] , [ 8 ] . Hence, the produced speech is not like normal speech. However, to avoid the digital divide, paying attention and making the speech technologies fair for CLP speech is necessary. There exist several attempts in the literature using CLP speech, mostly to perform classification and intelligibility enhancement. The work in [ 9 ] , utilizes wav2vec2 embeddings and uses them with machine learning classifiers and in [ 10 ] , transformer classifier to perform normal and CLP classification. [ 11 ] reports the intelligibility of the speech can be improved by compensating the hypernasality. The work in [ 12 ] , proposed a technique using glottal activity region to compute the hypernasality score, whereas in [ 13 ] deep learning approaches are used to estimate hypernasality. In [ 14 ] , the work proposed a procedure to detect the stop consonant and trill sounds and then replace them to enhance the speech perception. The paper [ 15 ] used a MaskCycleGAN approach to enhance the CLP speech and show the improvement in the phone decoding performance, using the GMM-HMM. Further, several works also attempt to enhance the intelligibility of CLP speech using the generative end-to-end deep learning frameworks [ 16 , 17 ] . However, as per the authors’ knowledge, no work exists in the literature, that studies the fairness of ASR in CLP speech. This study first examines the performance gap in ASR between normal and CLP speech by evaluating them using the state-of-the-art Google ASR API. To further analyze the similarities and differences between normal and CLP speech, we conduct cross-training and testing between the two speech types. Given the available data, we evaluate ASR performance using both the traditional GMM-HMM approach and foundation models, namely cross-lingual self-supervised speech representation (XLS-R) and Web-scale Supervised Pretraining for Speech Recognition (Whisper). XLS-R [ 18 ] leverages self-supervised learning, making it highly effective for low-resource, cross-lingual, and multilingual scenarios. In contrast, Whisper [ 19 ] is trained using supervised learning, enabling it to generalize across multiple speech tasks with minimal fine-tuning while maintaining strong robustness to noisy data. Figure 1 illustrates the speech signal, spectrogram, and formant contours for speech produced by normal, mild, moderate, and severely affected CLP subjects. The figure indicates a progressive shift in formant locations from normal to severe, likely due to the increasing degree of oral-nasal tract opening. However, while the shape of the formant contours remains largely intact in normal and mild speech, it gradually deteriorates in moderate and severe cases. This suggests that ASR performance may decline due to formant frequency shifts. Figure 1 : Speech signal produced by Normal, mild, moderate and severe subjects, and their corresponding spectrogram and formant contour, while producing the sound "kage" . However, the preservation of formant contour shapes, particularly in mild cases, motivates further investigation into strategies for developing a fair ASR system. Inspired by this, we train the ASR system by augmenting normal speech with various combinations of CLP speech across severity levels. The resulting performance is then analyzed with a focus on system fairness. The main contributions of this work are summarized as follows: 1. This study evaluates the fairness of publicly available ASR systems for CLP speech and confirms its adverse impact on recognition performance. 2. An augmentation strategy combining CLP and normal speech is proposed to enhance ASR fairness across varying severity levels. 3. A fairness score is introduced to quantify and assess improvements in ASR performance for CLP speech. 2 Related studies on CLP speech Research on cleft lip and palate (CLP) speech has primarily focused on hypernasality detection, intelligibility assessment, and speech enhancement. Hypernasality, a key characteristic of CLP speech, has been extensively studied using various signal processing techniques. Early works such as [ 20 ] and [ 21 ] employed Zero Time Windowing (ZTW) and Homomorphic processing-based cepstral features for hypernasality detection, with support vector machines (SVMs) used for classification. Other studies, including [ 22 ] , analyzed formant frequency shifts to measure vowel space area reduction in CLP children. Additionally, [ 23 ] proposed a posterior probability-based approach for estimating hypernasality scores, overcoming limitations of traditional perceptual evaluation methods. Deep learning-based methods, such as the attention-based BLSTM models [ 24 ] , have further improved automatic hypernasality detection, reducing dependence on specialized clinical expertise. Studies such as [ 25 ] and [ 26 ] introduced pitch-adaptive features and constant-Q cepstral coefficients (CQCC) to enhance classification accuracy, while [ 27 ] explored sinusoidal model-based features for robust hypernasality detection. More recent works, such as [ 28 ] , investigated advanced spectral and articulatory-based features for improved classification. Intelligibility assessment studies have employed a variety of methodologies to estimate speech intelligibility, including Gaussian posteriorgrams [ 13 ] , dynamic time warping (DTW) [ 29 ] , and regression models [ 30 ] . These approaches have demonstrated strong correlations with human perceptual ratings, underscoring their effectiveness for objective intelligibility assessment. For instance, studies like [ 31 ] have analyzed spectral moments of fricatives in CLP speech, while [ 32 ] leveraged epoch-synchronous features to detect nasalized voiced stops. In the realm of speech enhancement, significant efforts have been made to improve intelligibility using both signal processing and deep learning techniques. Early works, such as [ 33 ] , focused on modifying phoneme

Read the complete text