Audio–Visual Speech Enhancement: Architectural Design and Deployment Strategies

Anis Hamadouche, Haifeng Luo, Mathini Sellathurai and Tharm Ratnarajah · 2025

Read Audio–Visual Speech Enhancement: Architectural Design and Deployment Strategies by Anis Hamadouche, Haifeng Luo, Mathini Sellathurai and Tharm Ratnarajah (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2508-08468.txt.

Introduction

Anis Hamadouche and co-authors build and field-test a system that cleans up speech using both sound and lip movement. Their 2025 paper deploys the service on a public 5G edge network, where computing sits near the user. Live enhancement for calls and media is easily ruined by delay, so placement matters.

A sound model, face feature extraction, and a fusion network that keeps timing coherent form the pipeline. Stress tests varied network load and media settings. Edge placement proved vital for real-time timing, and uplink capacity was the main bottleneck. Strong compression shrank payloads by up to eighty percent with little quality loss. Simpler models cut delay but hurt cleanup in noise, a trade-off the authors map carefully.

Catalog record

CREATORAnis Hamadouche, Haifeng Luo, Mathini Sellathurai and Tharm Ratnarajah
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE25 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2508-08468

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Audio–Visual Speech Enhancement: Architectural Design and Deployment Strategies Anis Hamadouche, Haifeng Luo, Mathini Sellathurai and Tharm Ratnarajah Abstract This paper introduce a new AI-based Audio-Visual Speech Enhancement (AVSE) and presents a comparative performance analysis of deployment architectures. The proposed AVSE system integrates convolutional neural networks (CNNs) for spectral feature extraction and long short-term memory (LSTM) networks for temporal modeling, enabling robust speech enhancement through multimodal fusion of audio and visual cues. We investigate multiple deployment scenarios, including cloud-based, edge-assisted, and standalone device implementations, evaluating their performance in terms of speech quality improvement, latency, and computational overhead. Real-world experiments are conducted across heterogeneous network conditions (Ethernet, Wi-Fi, 4G, and 5G) to assess the trade-offs between processing delay, communication latency, and perceptual speech quality. Results show that while cloud deployment achieves superior enhancement quality, edge-assisted architectures offer the best compromise between latency and intelligibility, meeting real-time requirements under 5G and Wi-Fi 6 conditions. The findings provide practical guidelines for selecting and optimizing AVSE deployment architectures in diverse application domains, from assistive hearing devices to telepresence and industrial communications. I INTRODUCTION Audio-Visual Speech Enhancement (AVSE) leverages the complementary nature of auditory and visual cues to improve speech perception in adverse acoustic conditions. In humans, this multimodal integration is an innate cognitive process, enabling effective communication in the presence of background noise, competing talkers, or reverberant environments. Neuroscientific studies have established the value of visual input in selective auditory attention. Golumbic et al. [ 13 ] demonstrated that visual access to a talker’s face significantly improves the auditory cortex’s ability to segregate and track the target speaker in multi-speaker environments, a phenomenon often referred to as the “Cocktail Party” effect. Likewise, McGurk and MacDonald [ 3 ] revealed that incongruent lip movements can alter auditory perception entirely, demonstrating that speech understanding is not purely an auditory process but is strongly influenced by visual context. Further, Partan and Marler [ 9 ] emphasized that multimodal communication signals, common in group-living species, improve clarity, robustness, and interpretation. Motivated by these findings, AVSE systems employ visual information—such as lip motion, facial dynamics, and articulatory patterns—to guide audio enhancement. The integration of these modalities allows for more accurate speaker separation, noise suppression, and reverberation reduction. Modern deep learning techniques have accelerated this field by enabling hierarchical feature extraction and cross-modal fusion at multiple abstraction levels. Convolutional Neural Networks (CNNs) are typically employed to extract discriminative visual features, while spectrogram-based CNNs or waveform-level encoders capture audio representations. Recurrent architectures, such as Long Short-Term Memory (LSTM) networks and Gated Recurrent Units (GRUs), are frequently used to model temporal dependencies, which are critical for capturing speech dynamics. The applications of AVSE extend beyond traditional hearing aids. In military communications, AVSE can enhance intelligibility in complex acoustic environments such as the battlefield. In consumer electronics, it improves voice-command recognition for smart assistants and in-car systems, even in noisy driving conditions [ 27 ] . In education and telepresence, AVSE can ensure intelligible communication in virtual classrooms or video conferencing, while in industrial and public-service robotics, it enables reliable interaction despite background noise. The healthcare sector can also benefit, as AVSE improves telemedicine consultations for patients with speech or hearing impairments [ 26 ] . While algorithmic design is central to AVSE performance, deployment architecture has a decisive role in real-world effectiveness. Processing location—whether cloud-based, edge-assisted, or entirely on-device—affects latency, speech quality, and system scalability. Communication bandwidth, device computational capability, and network stability determine whether a deployment can meet the real-time requirements for interactive speech enhancement. In particular, deployment trade-offs become critical when serving resource-constrained devices such as IoT endpoints, low-cost smart glasses, or wearable hearing-assist devices. In this paper, we present a complete AVSE system implemented within the COG-MHEAR framework. The system uses a CNN–LSTM-based fusion architecture, and we evaluate its performance across multiple deployment configurations: server-side cloud processing, edge-assisted processing, and local execution. The contributions of this work are threefold: 1. We propose an AVSE architecture that combines convolutional and recurrent feature extraction with intermediate fusion, enabling robust speech enhancement in challenging conditions. 2. We design and implement a flexible deployment framework capable of supporting heterogeneous device capabilities, from lightweight IoT hardware to high-performance computing platforms. 3. We conduct a comparative performance analysis under realistic network conditions, quantifying trade-offs between latency, intelligibility, and computational load across deployment scenarios. The remainder of this paper is organized as follows. Section II describes the acoustic signal modeling framework. Section III reviews the visual feature extraction and fusion strategies. Section IV presents the proposed AVSE architecture and deployment designs. Section V reports a comparative evaluation of deployment scenarios, and Section VI concludes with implications for future AVSE systems. II Acoustic Signal Modeling Central to the concept of AVSE is the nuanced modeling of acoustic signals. Here, each impulse response emanating from a spatial position of a target source to a microphone is represented as h_{s}[k] , where k signifies a discrete-time index. This impulse response is bifurcated into two distinct components: the early part h_{s}^{1}[k] , encompassing the direct sound and early reflections, and the late part h_{s}^{2}[k] , which pertains to the subsequent reverberations. Consequently, the observed acoustic mixture at any given instance k can be expressed in a mathematical formulation as: y[k]=\sum_{s=1}^{T}x_{s}[k]+d[k] (1) x_{s}[k]=x_{s}^{0}[k]\circledast h_{s}^{1}[k] (2) d[k]=\sum_{s=1}^{T}x_{s}^{0}[k]\circledast h_{s}^{2}[k]+\sum_{c=1}^{N}d_{c}[k] (3) In this formulation, x_{s}^{0}[k] represents the original speech signal emitted by the s -th target speaker. The term x_{s}[k] denotes the clean speech signal from the s -th target speaker, inclusive of low-order reflections as captured by the microphone. The component d_{c}[k] symbolizes the noise contribution from various sources as observed at the microphone. III Visual Information Integration The incorporation of visual data significantly augments the AVSE process. Studies, such as those mentioned in [ 25 ] , have demonstrated the critical role visual elements play in enhancing speech recognition systems. Visual features not only provide high-level information about speech patterns but also furnish fine-grained articulatory details. The visual processing initiates with face detection and tracking algorithms [ 11 ] , [ 17 ] , [ 10 ] , [ 4 ] , [ 7 ] , which are vital for delineating input dimensionality and discerning the count of speakers. This step significantly reduces the complexity of subsequent stages by focusing solely on the target faces. III-A Deep Learning in AVSE The actual fusion and processing of the audio and visual data in AVSE are achieved using an ensemble of deep neural network models. These models, by virtue of their design, can learn and synthesize representations of the acoustic and visual modalities at several levels of abstraction. Convolutional Neural Networks (CNNs) [ 6 ] , [ 16 ] , [ 18 ] , [ 19 ] , [ 24 ] , with their inherent characteristics of sparse connectivity and translation invariance, are typically employed to process visual frames and extract pertinent features. They also find applications in acoustic signal processing, handling either the spectrogram or the raw waveform. The training of these networks, often conducted using stochastic gradient descent (SGD) [ 2 ] , [ 1 ] , RmsProp [ 12 ] , and Adam [ 15 ] , plays a critical role in parameter optimization and model effectiveness. III-B Fusion Paradigms With Deep

Read the complete text