VibOmni: Towards Scalable Bone-conduction Speech Enhancement on Earables

Lixing He, Yunqi Guo, Haozheng Hou, Zhenyu Yan · 2025

Read VibOmni: Towards Scalable Bone-conduction Speech Enhancement on Earables by Lixing He, Yunqi Guo, Haozheng Hou, Zhenyu Yan (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2512-02515.txt.

Introduction

Lixing He and three co-authors work on clearer speech from earbuds and headsets used in noisy places. Tiny devices leave little room for microphones, yet calls and voice assistants must still hear the user well. A single all direction microphone struggles when other voices crowd the same space.

VibOmni adds bone conduction sensing, picking up voice through vibrations in the head itself. Those vibrations carry the wearer's speech while mostly ignoring outside noise. Blending both signals aims for clean voice pickup that scales across earables, a growing class of small devices people wear all day. Clear voice pickup is a must for calls made on a busy street or a loud train.

Catalog record

CREATORLixing He, Yunqi Guo, Haozheng Hou, Zhenyu Yan
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE60 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2512-02515

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

VibOmni: Towards Scalable Bone-conduction Speech Enhancement on Earables Lixing He, Yunqi Guo, Haozheng Hou, Zhenyu Yan Abstract Earables, such as True Wireless Stereo earphones and VR/AR headsets, are increasingly popular, yet their compact design poses challenges for robust voice-related applications like telecommunication and voice assistant interactions in noisy environments. Existing speech enhancement systems, reliant solely on omnidirectional microphones, struggle with ambient noise like competing speakers. To address these issues, we propose VibOmni, a lightweight, end-to-end multi-modal speech enhancement system for earables that leverages bone-conducted vibrations captured by widely available Inertial Measurement Units (IMUs). VibOmni integrates a two-branch encoder-decoder deep neural network to fuse audio and vibration features. To overcome the scarcity of paired audio-vibration datasets, we introduce a novel data augmentation technique that models Bone Conduction Functions (BCFs) from limited recordings, enabling synthetic vibration data generation with only 4.5% spectrogram similarity error. Additionally, a multi-modal SNR estimator facilitates continual learning and adaptive inference, optimizing performance in dynamic, noisy settings without on-device back-propagation. Evaluated on real-world datasets from 32 volunteers with different devices, VibOmni achieves up to 21% improvement in Perceptual Evaluation of Speech Quality (PESQ), 26% in Signal-to-Noise Ratio (SNR) and about 40% WER reduction with much less latency on mobile devices. A user study with 35 participants showed 87% preferred VibOmni over baselines, demonstrating its effectiveness for depolyment in diverse acoustic environments. I Introduction Earables are smart devices designed to be worn on users’ heads or ears, including products such as True Wireless Stereo (TWS) earphones, VR/AR headsets, and smart glasses. These devices are equipped with various sensors and support a range of applications, including virtual and augmented reality (VR/AR), motion recognition, and voice assistants. TWS earphones, like the Apple AirPods series, have significantly contributed to the growth of the earables market, which has become the largest category among all wearable devices, with an estimated shipment of over 273 million units worldwide in 2023 [ 2 ] . Manufacturers are continuously enhancing earables by adding new functionalities. For instance, many earphones and headphones now feature active noise cancellation (ANC) to improve the listening experience. Additionally, voice-related applications that utilize the microphones in earables are among the most commonly used features. One of the key applications of earables is making phone calls, which an increasing number of people are utilizing. Users can also interact with voice assistants, such as Siri and Alexa, through voice commands. However, the speech quality on earables often falls short due to several challenges: First, most earables use omnidirectional microphones that capture sound from all directions, which can result in unwanted environmental noise. Second, while many earables are equipped with multiple microphones to create an array for noise reduction via beamforming, the proximity of the microphones to one another can hinder their performance in effectively isolating the user’s voice. Third, the speech audio is significantly attenuated by the time it reaches the earables, as they are typically positioned far away from the user’s mouth. Various approaches have been developed for speech enhancement. Signal processing-based methods [ 3 ] remove noises based on their statistical models. However, these approaches cannot handle complex environments. Microphone beamforming [ 4 , 5 ] removes noises based on their directions. But they fail to distinguish the speaker’s voice and noise when the microphones are placed too close. Several research [ 6 , 7 , 8 ] adopt deep neural networks (DNNs) to improve speech quality. However, their performance varies when the domain changes. Except for the audio-only solution, several works leverage other modalities like contact sensors [ 9 ] , vibration sensor [ 10 ] , camera [ 11 ] , mm-Wave Radar [ 12 , 13 , 14 ] , ultrasonic sensing [ 15 , 16 ] , and Lidar [ 17 ] , which introduce additional hardware requirements or user overhead to existing earables. Motivated by the above works, we envision finding a modality that: 1) can be obtained with existing earables without significant modification. 2) has a close correlation with the user’s speech and may not be influenced by noises. Since earables are well connected to the user’s head, it becomes a desired position to obtain the vibration transmitted to the user’s head. The vibration, which is also known as bone conduction vibration, is less influenced by the ambient sound and mainly depends on the clear speech from the user’s vocal tract. Fortunately, the existing sensors of commercial earables include an Inertial Measurement Unit (IMU), whose original function is to track the head pose. At the same time, the IMU (including accelerometer and gyroscope) can also capture the subtle vibration on the head, which corresponds to the bone conduction vibration. Considering the mainstream sampling rate of IMU is about 1.6~\text{kHz} , the bandwidth ( 800\text{Hz} ) overlaps with the lower part of the frequency range of human speech. Fig. 1 : VibOmni enhances speech quality of head-mounted wearables by extracting the user’s clear voice from the bone-conducted vibrations. To enhance speech quality in earables, we propose VibOmni, a low-latency, end-to-end multi-modal speech enhancement system using bone-conducted vibrations and audio, as shown in Fig. 1 . Optimized for mobile devices, VibOmni leverages vibration’s noise-robustness and audio’s rich information, addressing three key challenges: • Multi-Modal Data Fusion : Microphone audio, with its higher sample rate, provides richer information than vibration data, risking DNN overfitting and neglect of vibration input. A specialized training strategy with dual loss functions is needed to balance both modalities. • Scarcity of Paired Data : Collecting paired audio-vibration datasets for head-mounted wearables is labor-intensive. Existing public datasets lack paired data, and large-scale collection involves significant overhead from diverse volunteers and extensive annotated recordings. • Real-World Deployment : Dynamic interferences complicate deploying deep learning speech enhancement. Limited and variable-quality user-collected data may hinder training a robust model, even with rapid data collection. First, unlike previous work like audio-only speech enhancement or target speaker extraction that relies on time-invariant speaker embeddings, our multi-modal speech enhancement network (§ IV-B ) employs two encoders to extract and concatenate features from both modalities, with a projection layer aligning frequency dimensions due to differing sampling rates. A DPRNN module separates speech at the feature level and utilizes two decoders to estimate the full-band and low-band speech (to avoid the collapse of multi-modal learning), which is trained by SISNR loss. The whole model is constructed by causal convolutions and unidirectional RNNs for frame-by-frame inference. Second, to overcome the scarcity of paired audio-vibration datasets, VibOmni employs an innovative pre-training strategy using Bone Conduction Functions (BCFs) (§ IV-C ). By estimating BCFs from limited recordings and applying them to public datasets like LibriSpeech [ 18 ] , we generate synthetic vibration data, achieving a low 4.5% error in spectrogram similarity. This approach, enhanced by cubic interpolation and Gaussian modeling, ensures diverse training data and improves model generalization across users and conditions, reducing the need for extensive user-specific fine-tuning. These advancements enable VibOmni to bridge the gap between training and real-world testing, particularly in noisy, dynamic environments. Lastly, to tackle real-world speech enhancement challenges, VibOmni integrates advanced continual learning and adaptive inference mechanisms, as detailed in Sections IV-D3 and IV-D4 . We propose a multi-modal SNR estimator to distinguish clean audio from noise, enabling a continual

Read the complete text