Modeling Multi-Level Hearing Loss for Speech Intelligibility Prediction
2025
Read Modeling Multi-Level Hearing Loss for Speech Intelligibility Prediction (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2507-22599.txt.
Introduction
Xiajie Zhou and co-authors build a system that predicts how well a listener with hearing loss will understand speech. Standard hearing tests measure which quiet tones a person can hear, but not how blurred their hearing is in pitch and timing. The method here simulates that blur directly on the speech signal.
It widens cochlear filters and smooths the timing envelope, matched to the level of loss. Features from the altered signal train a Vision Transformer model to score intelligibility. Such a predictor could help tune hearing aids without long listening tests for every setting. A fast, test-free estimate would also help fit hearing support in clinics that lack specialist staff.
Catalog record
| YEAR | 2025 |
|---|---|
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 24 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2507-22599
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Modeling Multi-Level Hearing Loss for Speech Intelligibility Prediction Abstract The diverse perceptual consequences of hearing loss severely impede speech communication, but standard clinical audiometry, which is focused on threshold-based frequency sensitivity, does not adequately capture deficits in frequency and temporal resolution. To address this limitation, we propose a speech intelligibility prediction method that explicitly simulates auditory degradations according to hearing loss severity by broadening cochlear filters and applying low-pass modulation filtering to temporal envelopes. Speech signals are subsequently analyzed using the spectro-temporal modulation (STM) representations, which reflect how auditory resolution loss alters the underlying modulation structure. In addition, normalized cross-correlation (NCC) matrices quantify the similarity between the STM representations of clean speech and speech in noise. These auditory-informed features are utilized to train a Vision Transformer–based regression model that integrates the STM maps and NCC embeddings to estimate speech intelligibility scores. Evaluations on the Clarity Prediction Challenge corpus show that the proposed method outperforms the Hearing-Aid Speech Perception Index v2 (HASPI v2) in both mild and moderate-to-severe hearing loss groups, with a relative root mean squared error reduction of 16.5% for the mild group and a 6.1% reduction for the moderate-to-severe group. These results highlight the importance of explicitly modeling listener-specific frequency and temporal resolution degradations to improve speech intelligibility prediction and provide interpretability in auditory distortions. \NoHyper 1 Introduction Hearing loss is one of the most prevalent chronic health conditions globally, with significant impacts on communication, education, and quality of life [ 1 ] . Recent epidemiological analyses have highlighted a growing burden of hearing impairment in all age groups and regions, with hearing loss projected to affect more than 630 million individuals by 2030 and nearly 900 million by 2050 if no effective intervention is implemented [ 2 , 3 ] . Hearing loss is not a uniform condition: it varies widely in severity, ranging from mild to profound. These deficits result in reduced frequency selectivity and impaired temporal resolution, both of which degrade the listener’s ability to comprehend speech. Speech intelligibility—the degree to which a listener correctly recognizes speech—directly reflects the listener’s ability to comprehend speech, especially when hearing is degraded. Speech intelligibility is thus a key outcome for evaluating real-world listening performance in hearing aid usage [ 4 ] . While audiograms are widely used to measure hearing thresholds at different frequencies, they often fall short in explaining why some people understand speech better than others [ 5 , 6 ] . Even listeners with similar audiograms can perform quite differently, especially in noisy situations [ 7 ] . This suggests that evaluating hearing loss should go beyond thresholds and also consider how well the ear processes sounds, particularly its ability to distinguish frequencies and follow rapid changes over time. Although audiometric thresholds provide a basic estimate of hearing sensitivity, they fail to capture the auditory abilities essential for speech perception. One such ability is frequency resolution—the capacity to distinguish adjacent spectral components of sound—which is especially important in noisy environments. Impaired spectral resolution, which is commonly associated with sensorineural hearing loss, has been shown to significantly degrade intelligibility even when audibility is restored through amplification [ 8 ] . Simulation studies have further demonstrated that broader auditory filters based on individual audiograms lead to substantial reductions in speech understanding in complex acoustic scenes [ 9 ] . These findings indicate that speech intelligibility prediction must consider hearing thresholds and the extent of auditory filter degradation. In addition to frequency resolution, temporal resolution—the ability to track rapid amplitude fluctuations—is essential for speech intelligibility [ 10 ] . These modulations convey critical cues related to voicing, syllabic timing, and phoneme boundaries. Temporal resolution is often measured using the temporal modulation transfer function (TMTF), which reflects a listener’s sensitivity to amplitude modulation at various rates. Research indicates that individuals with sensorineural hearing loss commonly show reduced modulation sensitivity and lower cutoff frequencies, especially at higher modulation rates [ 11 , 12 ] . These deficits vary considerably across listeners. Some maintain relatively intact temporal processing, while others—particularly those with more severe or sloping high-frequency losses—exhibit pronounced reductions in TMTF sensitivity [ 13 ] . This variability underscores the importance of accounting for individual differences in temporal resolution when predicting speech intelligibility. Fig. 1: Overview of proposed method. (a) STM representations are extracted from both clean speech and speech in noise (SPIN) after peripheral processing including inverse equal-loudness contour (ELC) correction, producing 4D tensors \mathbf{I}_{\mathrm{clean}} and \mathbf{I}_{\mathrm{SPIN}} with dimensions (N_{S},N_{T},C,F) , where N_{S} and N_{T} are the number of spectral and temporal modulation channels, C is the number of cochlear frequency bands, and F is the number of time frames. (b) The two STM tensors are preprocessed into stacked image inputs via normalization (Norm), bilinear resizing (BLI), and dimensional stacking (DimStack). Data augmentation (e.g., affine transform, random erasing, and noise injection) is optionally applied. In parallel, normalized cross-correlation (NCC) is computed between clean speech and SPIN STM representations to generate a modulation similarity matrix \mathbf{M}_{\mathrm{NCC}}\in\mathbb{R}^{N_{S}\times N_{T}} . Finally, a Vision Transformer (ViT) backbone based on the ViT-Base architecture receives the STM representations and the flattened NCC matrix. The transformer’s classification token ( \mathrm{[CLS]}\ \mathrm{Token} ) is concatenated with the NCC embedding and passed through a regression head to predict the final speech intelligibility score. Since spectral and temporal resolution are degraded under hearing loss, the ability to extract dynamic acoustic cues is compromised. However, these two domains are not processed in isolation. Natural speech exhibits spectro-temporal modulations—joint fluctuations in frequency and time—that are believed to be encoded by the auditory cortex in the auditory system [ 14 ] . We adopt a spectro-temporal modulation (STM) analysis based on two-dimensional Gabor filters to capture this integrated structure [ 15 ] . This biologically inspired framework mirrors cortical tuning patterns and provides a unified representation of modulation content, allowing us to assess how hearing loss alters the encoding of speech-relevant modulations. This study aims to predict speech intelligibility under hearing loss by explicitly accounting for listener-specific degradations in frequency and temporal resolution. (1) We simulate the effects of hearing loss using auditory-inspired transformations that reflect varying degrees and types of spectral and temporal deficits. (2) We characterize speech signals through STM analysis, which captures how these degradations distort the modulation patterns critical for speech intelligibility. Unlike prior methods relying solely on audiometric thresholds or broadband metrics, our approach generates a time–frequency–modulation representation tailored to the listener’s perceptual limitations. These STM representations are then utilized to train a regression model for prediction. 2 Related work The Hearing-Aid Speech Perception Index v2 (HASPI v2) is adopted as the baseline intrusive model in the Clarity Prediction Challenge [ 16 , 17 , 18 ] . It simulates impaired auditory processing using the Gammatone filterbank, dynamic range compression, and envelope extraction. Hearing loss effects are incorporated via audiogram-based attenuation and reduced frequency selectivity, with speech intelligibility estimated by correlating clean speech and speech in noise (SPIN) envelopes across time–frequency bands. To account for binaural listening, HASPI computes left and right ear scores and selects the higher one ( HASPI v2 ). A logistic function is applied to map HASPI scores to the final prediction: \hat{y}=\frac{1}{1+e^{-a(\text{HASPI}-b)}}, (1) where a and b are optimized scaling parameters, and