Real-Time System for Audio-Visual Target Speech Enhancement

2025

Read Real-Time System for Audio-Visual Target Speech Enhancement (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2509-20741.txt.

Introduction

Picking one voice out of a noisy room is easier when you can see the speaker's lips. T Aleksandra Ma and three co-authors demonstrate RAVEN, a real-time audio-visual speech enhancement system, in a 2025 paper for the WASPAA demo session. The system runs entirely on an ordinary CPU.

RAVEN uses pretrained visual features taken from a speech recognition model that also watches the speaker, capturing how the lips move. Those features guide the cleaning of a single audio channel. The authors report that the system holds up across different background noises and competing speakers. They present it as the first interactive demonstration of its kind on CPU hardware. The work points toward hearing aids and live communication tools that watch as well as listen.

Catalog record

YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE9 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2509-20741

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Real-Time System for Audio-Visual Target Speech Enhancement Abstract We present a live demonstration for RAVEN, a r eal-time a udio- v isual speech en hancement system designed to run entirely on a CPU. In single-channel, audio-only settings, speech enhancement is traditionally approached as the task of extracting clean speech from environmental noise. More recent work has explored the use of visual cues, such as lip movements, to improve robustness, particularly in the presence of interfering speakers. However, to our knowledge, no prior work has demonstrated an interactive system for real-time audio-visual speech enhancement operating on CPU hardware. RAVEN fills this gap by using pretrained visual embeddings from an audio-visual speech recognition model to encode lip movement information. The system generalizes across environmental noise, interfering speakers, transient sounds, and even singing voices. In this demonstration, attendees will be able to experience live audio-visual target speech enhancement using a microphone and webcam setup, with clean speech playback through headphones. 1 Introduction Speech enhancement is commonly defined as the removal of background noise [ 1 ] , often focused on environmental sounds. However, real-world acoustic environments are more complex, frequently involving transient noises such as sudden bangs, as well as interfering speakers. Traditional audio-only speech enhancement algorithms struggle with separating out clean speech from competing speakers, unless provided with enrollment audio from target speaker. These challenges prompt researchers to explore the use of additional modalities, such as visual information, to improve target speech enhancement performance in recent years, as computing power increases [ 2 ] . There has been a rise in audio-visual speech enhancement (AVSE) literature, from mask-based approaches [ 3 , 4 , 5 ] to synthesis-based approaches [ 6 , 7 , 8 ] , achieving better results than only using audio [ 9 , 10 ] . 1.1 Motivation Despite this increase in AVSE research, most existing systems operate in a non-causal setting [ 5 , 9 , 3 ] . While having access to future information may achieve strong performance, they are unsuitable for real-time deployment. Although some real-time AVSE systems have been proposed, their implementations and demonstrations are not made available, making reproduction difficult [ 11 , 10 , 12 ] . Motivated by this gap, we present a real-time AVSE algorithm and built an application that enables users to test the speech enhancement algorithm in realistic scenarios, as seen in Fig. 1 . To our knowledge, this is the first publicly available, real-time AVSE demonstration that can be run on a CPU. 1.2 Application & Problem Scenario Real-time AVSE systems are especially useful for video calls [ 13 ] , where the model can isolate and enhance the on-screen speaker’s voice. This also applies to in-car communication systems [ 14 ] , enabling clear speech pickup even in noisy cabins with multiple speakers or background music. Beyond video calls, the technology extends to wearables equipped with cameras, such as smart glasses or headphones, where it can provide users with an audio feed of the clean speech of the speaker in their field of view. This offers a potential solution to the longstanding cocktail party problem, which refers to the human ability to focus auditory attention on a single speaker in a noisy, multi-speaker setting [ 15 ] . In addition, AVSE algorithms also hold promise for application in advanced audio-visual hearing aids [ 16 , 17 ] . Conventional hearing aids typically amplify all incoming sounds indiscriminately, which can overwhelm users in noisy environments and limit intelligibility. In contrast, audio-visual hearing aids could identify and prioritize speech signal of interest by leveraging relevant visual cues. Fig. 1: Setup of our real-time audio-visual speech enhancement system. The audio-visual input is streamed into our Python application via the webcam and the microphones. Visitors can hear the clean speech through the playback headphones in real time. 2 System Design Our real-time AVSE system consists of two main components: (1) a causal AVSE model designed for real-time inference trained on VoxCeleb2 [ 18 ] , and (2) a real-time streaming pipeline that interfaces with live audio-visual inputs, processes them through the AVSE model, and outputs the enhanced speech signal. 1 1 1 The algorithm and training code are released through Interspeech 2025 [ 19 ] ; however, the real-time system has never been demonstrated live. 2.1 Model Architecture Fig. 2: Architecture of our mask-based late fusion method. A batch normalization layer is added after each CNN layer, and a ReLU layer is added to each CNN and fully connected layer. A Sigmoid function is applied to the magnitude mask to limit the mask values from 0 to 1. As shown in Fig. 2 , RAVEN uses a mask-based, late-fusion architecture. The visual stream first crops the mouth region from the input video and passes it through a pretrained visual encoder to extract lip movement embeddings. The pretrained visual encoder, Visual Speech Recognition in the Wild (VSRiW) [ 20 , 21 ] , is an end-to-end visual speech recognition model that integrates feature extraction with a hybrid CTC/attention back-end. The checkpoint that we used was trained on GRID, which results in a Word Error Rate of 4.8. Its visual front end uses a ResNet encoder with a 3D convolutional layer that has a kernel size of 5. It has a padding of 2 on both sides along the time axis, which results in a receptive field of 5 including a look-ahead of 2 video frames. The visual stream has a frame rate of 25 frames per second. The audio input mixture clips are synthesized to simulate our target task by combining the target audio with the audio from another randomly selected input video. Then the signal-to-noise ratio of the input audio mixture is constrained to a range of -5 dB to 5 dB. We obtain the time-frequency representations of these 5-second clips through Short-Time Fourier Transform (STFT). We use a Hann window of length 400, a hop size of 160, and 512 frequency bins (nfft), with a power-law compression rate (p) of 0.3. These spectrograms are passed into a 15-layer convolutional neural network (CNN). To align the audio and visual streams, the visual embeddings are upsampled to 100 fps before they are concatenated with the audio embeddings. The concatenated audio-visual features are subsequently fed into a uni-directional Long Short-Term Memory (LSTM) network, the output of which is passed through a series of three fully connected layers that produce a predicted magnitude mask. This mask is applied element-wise to the magnitude spectrogram of the noisy input mixture, yielding an estimated magnitude spectrogram of the clean speech. To reconstruct the enhanced speech waveform, the estimated clean magnitude spectrogram is combined with the phase of the original noisy mixture to form an estimated complex spectrogram. The resulting complex spectrogram is then transformed back into the time domain using the inverse Short-Time Fourier Transform (ISTFT). The model is trained using a phase-sensitive approximation (PSA) loss function, which minimizes the Mean Squared Error (MSE) between both the predicted and ground truth magnitude as well as complex spectrogram to mitigate for the loss of phase information during training, inspired by [ 22 , 23 ] . 2.2 Components & Implementation The system is implemented in Python. Audio

Read the complete text