PAS-SE: Personalized Auxiliary-Sensor Speech Enhancement for Voice Pickup in Hearables
Mattes Ohlenbusch \sthanks This work was done during an internship at Bose, Mikolaj Kegler, Marko Stamenovic · 2025
Read PAS-SE: Personalized Auxiliary-Sensor Speech Enhancement for Voice Pickup in Hearables by Mattes Ohlenbusch \sthanks This work was done during an internship at Bose, Mikolaj Kegler, Marko Stamenovic (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2509-20875.txt.
Introduction
Hearing devices must lift the wearer's own voice out of surrounding noise and other talkers. Mattes Ohlenbusch, Mikolaj Kegler, and Marko Stamenovic compare two ways to do that in a 2025 paper accepted for ICASSP 2026. Both strategies give the enhancer extra context about which voice is the target.
One strategy personalizes the system with enrollment recordings of the user. The other uses in-ear microphones as auxiliary sensors. The authors evaluate both on two public datasets with different sensor arrays, probing how well each generalizes. They also propose training-time augmentations to improve robustness. The comparison matters for hearables, where single-channel methods alone struggle to tell the wearer's voice from a nearby speaker.
Catalog record
| CREATOR | Mattes Ohlenbusch \sthanks This work was done during an internship at Bose, Mikolaj Kegler, Marko Stamenovic |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 24 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2509-20875
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
PAS-SE: Personalized Auxiliary-Sensor Speech Enhancement for Voice Pickup in Hearables Mattes Ohlenbusch \sthanks This work was done during an internship at Bose, Mikolaj Kegler, Marko Stamenovic Abstract Speech enhancement for voice pickup in hearables aims to improve the user’s voice by suppressing noise and interfering talkers, while maintaining own-voice quality. For single-channel methods, it is particularly challenging to distinguish the target from interfering talkers without additional context. In this paper, we compare two strategies to resolve this ambiguity: personalized speech enhancement (PSE), which uses enrollment utterances to represent the target, and auxiliary‑sensor speech enhancement (AS-SE), which uses in‑ear microphones as additional input. We evaluate the strategies on two public datasets, employing different auxiliary sensor arrays, to investigate their cross-dataset generalization. We propose training‑time augmentations to facilitate cross-dataset generalization of AS-SE systems. We also show that combining PSE and AS-SE (PAS-SE) provides complementary performance benefits, especially when enrollment speech is recorded with the in‑ear microphone. We further demonstrate that PAS-SE personalized with noisy in-ear enrollments maintains performance benefits over the AS-SE system. † † address: 1 Bose Corporation, USA 2 Fraunhofer Institute for Digital Media Technology IDMT, Oldenburg Branch for Hearing, Speech and Audio Technology HSA, Germany 1 Introduction Hearable devices with one or more microphones can be used to capture the user’s own voice (i.e., target speech), but due to environmental noise and interfering talkers, the captured speech signal needs to be enhanced to facilitate communication. Although single-channel speech enhancement (SE) approaches based on deep neural networks can reduce environmental noise [ 1 , 2 , 3 , 4 ] , they tend to struggle with removing interfering talkers from the mixture while preserving the user’s voice. Modern hearables often include microphones placed inside or near the user’s occluded ear canal, typically employed in active noise reduction systems [ 5 ] . These in-ear microphones are acoustically shielded from environmental noise and interfering talkers by the device. At the same time, the user’s voice is also picked up by the in-ear microphone, predominantly through body-conduction [ 6 , 7 ] . These two effects lead to a substantial benefit in terms of signal-to-noise ratio (SNR) of the user’s voice at the in-ear microphone. However, this signal cannot be directly used for voice communication due to time- and user-varying band-limitation, nonlinearities and distortions introduced by body conduction [ 6 , 7 ] , and undesired additive body-produced noises [ 8 ] . Other auxiliary body-conduction sensors, such as accelerometers, have similar properties and trade-offs as in-ear microphones [ 9 ] . Due to their benefits, auxiliary sensors have been employed for own-voice speech enhancement [ 9 , 10 , 11 , 12 , 13 ] . The use of an auxiliary sensor as an additional input can substantially improve performance, especially in challenging SNRs [ 14 ] and scenarios with interfering talkers [ 9 ] . Notably, in [ 9 ] it was found that using a body-conduction sensor as additional input facilitated interferer suppression when tested with simulated data, assuming no noise and interferer transmission to the auxiliary sensor. We broadly refer to this approach, where the system uses both an outer microphone and an auxiliary sensor as input, as auxiliary sensor speech enhancement (AS-SE). Another way to extract speech from a target talker in a single-channel system is to condition the system with a feature vector obtained from enrollment utterances of the target talker [ 15 , 16 , 17 , 18 , 19 ] , commonly referred to as target speaker extraction (TSE). In this work, we consider a special case of TSE, personalized speech enhancement (PSE), where the target talker is always the device user. Using the enrollment utterance of the user, the PSE system can resolve the ambiguity between target and interfering talker. It should be noted that the methods mentioned above each come with their own unique trade-offs. PSE typically requires enrollment speech, which implies an additional setup procedure for the user prior to using the system. In contrast, AS-SE systems can be readily used by any user without an additional setup procedure. However, the resulting system may not generalize across different devices due to unique array properties or acoustic design of the hearable device. A systematic evaluation or integration of PSE and AS-SE methods has yet to be carried out. In this paper, we: 1. Benchmark PSE and AS-SE performance by systematically evaluating their denoising capabilities, suppression of interfering talkers, and generalization across datasets. 2. Explore and ablate different training-time augmentation configurations for training AS-SE systems with limited pre-recorded in-ear signals to facilitate interfering talker suppression and cross-dataset generalization. 3. Personalize AS-SE through enrollment-based conditioning, investigate the benefits of auxiliary-sensor enrollment to develop personalized AS-SE (PAS-SE) systems, and analyze their robustness to noise in enrollment utterances. Experimental evaluations carried out on two openly available in-ear microphone datasets demonstrate that interfering speech can be considerably reduced with either PSE or AS-SE, while a combination of both, which we term personalized auxiliary-sensor speech enhancement (PAS-SE), leads to further improvements. In particular, PAS-SE personalized with in-ear enrollments led to the best results, generalizing both within and across datasets tested. 2 Signal model We consider a hearable device equipped with an outer microphone and an in-ear microphone. The signals are denoted by subscripts o for the outer microphone (OM) and i for the in-ear microphone (IM). In the short-time Fourier transform (STFT) domain, S_{o}(k,l) and S_{i}(k,l) denote the own voice signals of the user (i.e., target speech) at each microphone, where k and l denote the frequency and the frame index. The outer and in-ear microphone signals are given by Y_{\{o,i\}}(k,l)=S_{\{o,i\}}(k,l)+N_{\{o,i\}}(k,l)+V_{\{o,i\}}(k,l), (1) respectively, where the noise components are denoted by N_{o}(k,l) and N_{i}(k,l) , and the interfering talker components are denoted by V_{o}(k,l) and V_{i}(k,l) . 3 System architecture In this paper, we explore various SE, PSE, AS-SE, and PAS-SE systems by modifying the FT-JNF architecture proposed in [ 20 ] . This architecture has previously been applied to the AS-SE task in [ 14 , 21 ] . Fig. 1 shows the personalized FT-JNF variant used in this paper, Figure 1: PAS-SE system architecture based on FT-JNF [ 20 ] . The system is personalized using multiplicative conditioning with a feature vector \mathbf{e} obtained from an enrollment utterance \tilde{Y}_{\text{enroll}} . which takes the magnitude of the noisy STFT coefficients of one or more microphones as input. The architecture consists of a unidirectional long short-term memory (LSTM) layer operating across the frequency dimension of the input (F-LSTM) with 512 hidden units, a causal, unidirectional LSTM layer operating across the time dimension (T-LSTM) with 128 hidden units, a linear layer, and a \tanh activation function. The output is a magnitude mask M(k,l) , which is applied to the noisy outer microphone signal Y_{o}(k,l) , i.e. \hat{S}_{o}(k,l)=M(k,l)\cdot Y_{o}(k,l) . The SE system (with one input) has 1.384 M parameters, and the AS-SE system has 1.386 M parameters. We add a conditioning mechanism to the architecture to investigate personalization for our PSE and PAS-SE systems by introducing a speaker encoder branch based on the time-domain Speakerbeam architecture [ 15 ] using the authors’ implementation in the Asteroid toolkit [ 22 ] . An utterance \tilde{Y}_{\text{enroll}}(k,l) from the same user but excluding