Leveraging Spatial Cues from Cochlear Implant Microphones to Efficiently Enhance Speech Separation in Real-World Listening Scenes
Feyisayo Olalere 1 , Kiki van der Heijden 1,2 , Christiaan H Stronks 3,4 , Jeroen Briaire 3 , Senior Member, IEEE , Johan HM Frijns 3,4,5 , Senior Member, IEEE , Marcel van Gerven 1 · 2025
Read Leveraging Spatial Cues from Cochlear Implant Microphones to Efficiently Enhance Speech Separation in Real-World Listening Scenes by Feyisayo Olalere 1 , Kiki van der Heijden 1,2 , Christiaan H Stronks 3,4 , Jeroen Briaire 3 , Senior Member, IEEE , Johan HM Frijns 3,4,5 , Senior Member, IEEE , Marcel van Gerven 1 (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2501-14610.txt.
Introduction
Feyisayo Olalere and co-authors ask how much echoes and room layout help or hinder software that separates overlapping voices for cochlear implant users. Separation tools shine on clean studio mixtures but struggle in real rooms. The team measured that drop, then tested two kinds of spatial help: cues the model can pick up by itself from the sound, and extra location features fed in by hand.
Both kinds helped, especially when talkers sat close together or their voices sounded alike, the cases where tone alone cannot separate them. Hand-fed cues paid off most where the built-in ones were weak, as with a single implant microphone rather than two. The authors stress that training on real-world recordings is what makes such systems carry over to everyday listening.
Catalog record
| CREATOR | Feyisayo Olalere 1 , Kiki van der Heijden 1,2 , Christiaan H Stronks 3,4 , Jeroen Briaire 3 , Senior Member, IEEE , Johan HM Frijns 3,4,5 , Senior Member, IEEE , Marcel van Gerven 1 |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 41 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2501-14610
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Leveraging Spatial Cues from Cochlear Implant Microphones to Efficiently Enhance Speech Separation in Real-World Listening Scenes Feyisayo Olalere 1 , Kiki van der Heijden 1,2 , Christiaan H Stronks 3,4 , Jeroen Briaire 3 , Senior Member, IEEE , Johan HM Frijns 3,4,5 , Senior Member, IEEE , Marcel van Gerven 1 Abstract Speech separation approaches for single-channel, dry speech mixtures have improved greatly. However, speech separation in real-world, spatial and reverberant acoustic environments remains challenging. This limits the efficiency of existing approaches for real-world speech separation applications in assistive hearing devices such as cochlear implants (CIs). To address this issue, we quantify the impact of real-world acoustic scenes on speech separation and investigate to what extent spatial cues from such real-world scenes can improve separation quality in an efficient manner. Crucially, we characterize speech separation performance as a function of implicit spatial cues (i.e., cues inherently present in the acoustic input that can be learned by the model), as well as of explicit spatial cues (i.e., manually calculated spatial features added as auxiliary input to the model). Our findings show that spatial cues (both implicit and explicit) improve separation performance for mixtures with spatially separated talkers, but also for mixtures with nearby talkers. Further, we demonstrate that spatial cues enhance speech separation in particular when spectral cues for separation are ambiguous, that is, when voices are similar. Finally, we show that the addition of explicit, auxiliary spatial cues is particularly beneficial when implicit spatial cues are weak. For example, microphone recordings from a single CI contain weaker implicit spatial cues than when microphone recordings from two, bilateral CIs are combined. These findings emphasize the importance of training models on real-world data to improve generalisability to everyday listening situations and contribute to the development of more efficient speech separation approaches for CIs or other assistive hearing devices in such real-world listening situations. I Introduction Hearing impairments affect more than 5% of the global population and result in significant communication difficulties, even with the use of assistive hearing devices such as cochlear implants (CIs) [ 1 , 2 ] . CIs are implanted neuroprosthetic devices which directly stimulate the auditory nerve to restore hearing for individuals with severe-to-profound sensorineural hearing loss [ 3 ] . While CI users experience tremendous improvements in speech perception in quiet environments, communication difficulties persist in noisy, everyday listening scenes such as the classroom, the office, or at social gatherings [ 4 , 5 , 6 , 7 ] . These difficulties are a consequence of peripheral processing deficits, which hamper a CI user’s ability so selectively attend to a talker in noisy listening scenes [ 7 ] . Implementing a speech denoising algorithm as a front-end processing step in a CI or other assistive hearing device significantly improves speech perception for hearing impaired listeners, because this provides the user with clean speech signals. However, everyday listening environments often consist of multiple, simultaneous talkers in addition to various background noise sources [ 8 ] . As voices have similar acoustic characteristics, such multi-talker scenes are particularly challenging for speech denoising algorithms. Therefore, automatic speech separation also plays a crucial role in front-end speech processing [ 9 ] . Currently, deep neural network (DNN) based approaches are the state of the art for speech separation. Time-frequency based DNN speech separation methods operate on spectrogram representations of multi-talker mixtures [ 10 , 11 , 12 ] . Such approaches either extract masks of each talker in the mixture [ 13 , 14 ] or directly reconstruct the spectrogram of each talker [ 15 , 16 ] . However, time-frequency speech separation approaches have several drawbacks. Most notably, as phase information is discarded, only an approximated phase [ 17 ] or the phase of the mixed audio [ 18 ] can be used to reconstruct the sound wave for each talker. This introduces additional noise and limits the quality of the reconstructed, clean talkers [ 19 ] . More recent speech separation approaches therefore focus on separation in the time-domain, i.e., operating directly on the audio mixture’s waveform [ 20 , 21 , 22 ] . These approaches avoid the phase inversion problem and therefore introduce fewer artifacts in the reconstructed speech waveforms [ 22 ] . Moreover, several time-domain approaches were shown to be highly computationally efficient [ 22 , 23 , 24 ] , making them more suitable for small devices such as CIs and other assistive hearing technology. However, few approaches (either in the time or in the time-frequency domain) have been applied to real-world acoustic scenes that are spatial and include reverberation. Reverberation distorts the talkers’ speech and thereby increases the difficulty of the speech separation task significantly [ 23 ] , resulting in a drop in speech separation performance [ 25 , 26 , 27 ] . This poses a significant obstacle to the application of speech separation algorithms as a front-end processing step in assistive hearing technology such as CIs. Yet, the spatial dimension of real-world acoustic scenes can also be considered an additional source of information for optimizing speech separation: The spatial separation between talkers causes each talker to have different spatial cues, which can be utilised by speech separation algorithms for speech separation. Studies investigating to what degree incorporating spatial learning improves speech separation performance typically add spatial features as auxiliary input to the model. For example, inter-microphone phase differences (IPD) a priori [ 28 , 29 , 18 , 30 ] , level differences (ILD) [ 31 , 32 ] , time difference (ITD) [ 33 ] , or angle features (AF) [ 28 ] . The aforementioned approaches all show that auxiliary spatial features improve speech separation performance. However, these spatial features are typically extracted from time-frequency representations of the two-talker mixture, even if the model itself operates in the time domain (e.g. [ 30 , 32 ] ). This results in relatively long latency and a high computational cost. Moreover, introducing auxiliary features such as spatial features increases the number of model parameters, resulting in an even higher latency and computational cost. These approaches are thus sub-optimal for applications in devices such as CIs, which require energy-efficient and low-latency speech separation approaches. Furthermore, a recent study which introduced an efficient speech separation system utilising the spatial information that is present in the scene rather than adding explicit, auxiliary spatial features, showed a significant drop in performance in real-world scenes with reverberation [ 32 ] . A similar study [ 34 ] which also leveraged implicit cues for a single speaker and noise configuration shows the drop in performance experienced in the real-word even with the help of implicit cues. The present work therefore investigates to what extent the spatial cues directly available in the audio signals recorded by microphones on an assistive hearing device such as a CI can be utilised to efficiently enhance speech separation in real-world listening scenes. We assess the impact of such implicit spatial cues (that is, cues inherently present in the acoustic input) on speech separation and we evaluate how this relates to separation performance when intermicrophone phase differences (IPDs) are added as explicit, auxilliary spatial cues to the model input. Further, as