Non-Intrusive Binaural Speech Intelligibility Prediction Using Mamba for Hearing-Impaired Listeners
Katsuhiko, Koichi · 2025
Read Non-Intrusive Binaural Speech Intelligibility Prediction Using Mamba for Hearing-Impaired Listeners by Katsuhiko, Koichi (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2507-05729.txt.
Introduction
Katsuhiko Yamamoto and Koichi Miyazaki propose in 2025 a leaner way to predict how well a hearing-impaired listener will understand speech, without playing the sound to anyone. Such predictors serve as objective scores for hearing technology. Leading models use transformer networks on two-ear recordings and score well, but their attention step is heavy in computing and memory, a poor fit for small, low-power devices.
The pair swap the transformer blocks for Mamba blocks, a newer design built for sequences. Their model matched the strong baseline's accuracy while keeping the parameter count low. Analysis showed the two-way Mamba version reads both context and the spatial cues that separate the ears. The authors frame the work as a step toward intelligibility prediction that can run inside hearing devices themselves, rather than only in the lab.
Catalog record
| CREATOR | Katsuhiko, Koichi |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 20 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2507-05729
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Non-Intrusive Binaural Speech Intelligibility Prediction Using Mamba for Hearing-Impaired Listeners Katsuhiko, Koichi Abstract Speech intelligibility prediction (SIP) models have been used as objective metrics to assess intelligibility for hearing-impaired (HI) listeners. In the Clarity Prediction Challenge 2 (CPC2), non-intrusive binaural SIP models based on transformers showed high prediction accuracy. However, the self-attention mechanism theoretically incurs high computational and memory costs, making it a bottleneck for low-latency, power-efficient devices. This may also degrade the temporal processing of binaural SIPs. Therefore, we propose Mamba-based SIP models instead of transformers for the temporal processing blocks. Experimental results show that our proposed SIP model achieves competitive performance compared to the baseline while maintaining a relatively small number of parameters. Our analysis suggests that the SIP model based on bidirectional Mamba effectively captures contextual and spatial speech information from binaural signals. Yamamoto Miyazaki CyberAgentJapan keywords speech intelligibility, non-intrusive metrics, hearing loss, clarity challenge, state-space model † † email: {yamamoto_katsuhiko, miyazaki_koichi_xa}@cyberagent.co.jp 1 Introduction Speech intelligibility (SI) is an index used to evaluate how spoken words and sentences are heard under noisy and distorted conditions. This helps assess the listening abilities of hearing-impaired (HI) individuals and the effectiveness of speech enhancement (SE) processes in hearing aids (HAs) [ 1 , 2 , 3 ] . Many SI prediction (SIP) models have been suggested for modeling speech perception and developing objective evaluation tools [ 4 , 5 , 6 ] . However, establishing a unified framework for comparing SIP models has been challenging due to the difficulty in collecting extensive subjective evaluation data. To address this issue, the Clarity Project 1 1 1 https://claritychallenge.org/ includes the Clarity Enhancement Challenge (CEC) and the Clarity Prediction Challenge (CPC) [ 7 , 8 , 9 , 10 ] . As a representative method in CPC2, a non-intrusive binaural SIP model ( E011 ) [ 11 ] using the transformer [ 12 ] architecture has been proposed, with “speech foundation model (SFM)” features extracted from Whisper [ 13 ] and WavLM [ 14 ] . It outperformed the binaural model of the hearing-aid speech perception index (HASPI) version 2 ( beHASPI ) [ 15 ] and all other metrics [ 10 ] . Figure 1: (a) Overview of the monaural SIP model based on E011 [ 11 ] . The numbers in parentheses indicate the size of the matrix, where L is the number of encoder layers in Whisper, t is the number of frames obtained as the output of the encoder layer, p is the temporal pooling size, and d is the feature dimension. (b) Architectures of the temporal transform blocks. Input X is the feature of a channel, and output Y is the feature obtained from the output of the blocks. As exemplified by the E011 model in CPC2, transformer-based methods have been used de facto in non-intrusive binaural SIP models [ 16 , 17 , 18 , 19 , 20 ] . However, the self-attention mechanism theoretically requires \mathcal{O}(L^{2}) computations and significant memory. This requirement creates a bottleneck in embedded environments such as HAs, where long-duration input, ultra-low latency, and power efficiency are essential. By contrast, state-space sequence models (SSMs), such as Mamba, can perform inference in linear time with constant memory [ 21 ] . This capability allows them to maintain long-range dependencies while operating in real time, making them suitable for low-computational and low-power devices. Therefore, in this paper, we propose a Manba-based SIP model to address the limitations of transformer-based models. Instead of using a temporal transformer, our model leverages Mamba, which is effective for time-series data, including speech applications, with lower inference costs [ 21 , 22 ] . Using monaural and binaural models, we compared the prediction performance of SI between transformer- and Mamba-based models. Additionally, we further investigated the impact of reducing the temporal pooling size of Whisper features on time resolution and how it affects SI in the Mamba-based model. The contributions of this study are as follows: • Our proposed model, featuring a Mamba-based temporal transform block, performed competitively in SI prediction compared to the transformer-based non-intrusive binaural SIP model [ 11 ] . • The experimental results show that our proposed model captures temporal information in Whisper features to perform contextual and binaural processing using Mamba 2 Mamba The Mamba architecture [ 21 ] was proposed as a sequence-to-sequence model for mapping an input sequence \mathbf{x}\in\mathbb{R}^{D} to \mathbf{y}\in\mathbb{R}^{D} with an element-wise hidden state \mathbf{h}_{d}\in\mathbb{R}^{N} , which can be formulated as follows: [ 22 ] : \displaystyle h_{t,d} \displaystyle=\bar{\mathbf{A}}h_{t-1,d}+\bar{\mathbf{B}}x_{t,d}, (1) \displaystyle y_{t,d} \displaystyle=\mathbf{C}h_{t,d}, (2) \displaystyle\bar{\mathbf{A}},\bar{\mathbf{B}} \displaystyle=\exp(\Delta\mathbf{A}),\Delta\mathbf{B}, (3) where \mathbf{A}\in\mathbb{R}^{N\times N} , \mathbf{B}\in\mathbb{R}^{N\times 1} , \mathbf{C}\in\mathbb{R}^{1\times N} and \Delta\in\mathbb{R}_{+} represent continuous SSM parameters [ 23 ] . The Mamba approach proposes a selective scan mechanism to derive input-dependent parameters with \mathbf{B}_{t} , \mathbf{C}_{t} and \Delta_{t} as follows: \displaystyle\mathbf{B}_{t},\mathbf{C}_{t} \displaystyle=\mathbf{W}_{B}\mathbf{x_{t}},(\mathbf{W}_{C}\mathbf{x_{t}})^{\top}, (4) \displaystyle\Delta_{t} \displaystyle=\text{softplus}(\mathbf{W}_{\Delta}\mathbf{x_{t}}), (5) where \mathbf{W}_{B}\in\mathbb{R}^{N\times D} , \mathbf{W}_{C}\in\mathbb{R}^{N\times D} , \mathbf{W}_{\Delta}\in\mathbb{R}^{1\times D} , and (\cdot)^{\top} denotes the transpose. Mamba also uses a parallel scan [ 24 , 25 ] and hardware-efficient algorithms for input-dependent SSM to mitigate the limitations of global convolution [ 23 ] . The Mamba block combines the prior SSM architecture (H3) [ 26 ] with a multi-layer perceptron (MLP) block, as detailed in [ 21 ] . 3 Proposed Model 3.1 Monaural SIP Model 3.1.1 Overall Architecture Figure 1 (a) shows the overall architecture of the monaural version of the E011 model [ 11 ] . The model is based on the network structure of a method called Whisper-AT [ 27 ] , which was proposed to perform speech recognition and environmental sound classification jointly. The model inputs are a single-channel speech signal and audiogram data. The audiogram represents the listener’s hearing threshold in an ear, measured for subjective evaluation experiments at various frequencies [ 9 , 10 ] . Speech Feature Extraction Blocks: The encoder layers of Whisper are used as the backbone for feature extraction. We also utilized the encoders of the Whisper Large-v2 model, which consist of 1280-dimensional features and 32 transformer layers. The parameters of the base model are fixed at their pre-trained values. Note that the input speech signal is temporarily downsampled to 16 kHz. Feature sequences from Whisper encoders are reduced in the time direction by applying an average pooling size p=20 to decrease computational cost [ 11 , 18 ] . Temporal Transform Block: The Whisper feature sequence is linearly projected into a 384 -dimensional space. Then, a 384 -dimensional temporal transform block described in Sec. 3.1.2 is applied to each layer to create a sequence with contextual features. Global average pooling is applied along the time axis to each layer’s sequence, producing a 384 -dimensional embedding for each layer. These embeddings are concatenated into an L\times 384 -dimensional matrix. Audiogram Features: The listener’s audiogram information is linearly projected onto a 384 -dimensional space and then concatenated with a layer-directional embedding representation, incorporating the listener’s characteristics [ 11 ] . Layer-wise SIP Block: Each feature is concatenated and used as sequence data for layer-wise representation, resulting in an (L+1)\times 384 -dimensional matrix. This matrix is input into the single-head layer-directional transformer and then compressed again by global average pooling, yielding a 384 -dimensional representation. The representation is linearly projected into a