Beyond Lips: Integrating Gesture and Lip Cues for Robust Audio-visual Speaker Extraction

Zexu Pan, Xinyuan Qian, Shengkui Zhao, Kun Zhou, Bin Ma · 2026

Read Beyond Lips: Integrating Gesture and Lip Cues for Robust Audio-visual Speaker Extraction by Zexu Pan, Xinyuan Qian, Shengkui Zhao, Kun Zhou, Bin Ma (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2601-19130.txt.

Introduction

Zexu Pan and four co-authors pull one voice from a crowd using hands as well as lips. Most tools that isolate a target speaker lean on lip motion synced to the speech. But people also gesture as they talk, and those gestures line up with stressed words and beats.

Adding gesture cues gives the system a second clock to lock onto. That should help when lips are hidden, blurred, or turned away from the camera. Cleaner single voice tracks serve hearing aids, meeting tools, and any setting where many people talk at once.

Catalog record

CREATORZexu Pan, Xinyuan Qian, Shengkui Zhao, Kun Zhou, Bin Ma
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE24 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2601-19130

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Beyond Lips: Integrating Gesture and Lip Cues for Robust Audio-visual Speaker Extraction Zexu Pan, Xinyuan Qian, Shengkui Zhao, Kun Zhou, Bin Ma Abstract Most audio-visual speaker extraction methods rely on synchronized lip recording to isolate the speech of a target speaker from a multi-talker mixture. However, in natural human communication, co-speech gestures are also temporally aligned with speech, often emphasizing specific words or syllables. These gestures provide complementary visual cues that can be especially valuable when facial or lip regions are occluded or distant. In this work, we move beyond lip-centric approaches and propose SeLG , a model that integrates both lip and upper-body gesture information for robust speaker extraction. SeLG features a cross-attention-based fusion mechanism that enables each visual modality to query and selectively attend to relevant speech features in the mixture. To improve the alignment of gesture representations with speech dynamics, SeLG also employs a contrastive InfoNCE loss that encourages gesture embeddings to align more closely with corresponding lip embeddings, which are more strongly correlated with speech. Experimental results on the YGD dataset, containing TED talks, demonstrate that the proposed contrastive learning strategy significantly improves gesture-based speaker extraction, and that our proposed SeLG model, by effectively fusing lip and gesture cues with an attention mechanism and InfoNCE loss, achieves superior performance compared to baselines, across both complete and partial (i.e., missing-modality) conditions. † † address: 1 Tongyi Lab, Alibaba Group, Singapore 2 University of Science and Technology Beijing, China Fig. 1: Our proposed audio-visual speaker extraction model named SeLG , which estimates the target speech conditioned on the synchronized gesture sequence v_{g}(t) and the lip sequence v_{l}(t) of the target speaker. The symbols \oplus and \otimes represent element-wise addition and multiplication, respectively. 1 Introduction Speech is the most natural form of human communication. However, speech signals are often contaminated by interfering background speakers, posing a significant challenge for speech processing tasks such as automatic speech recognition, speaker verification, or localization. Humans have the innate ability to focus attention on a speaker of interest, i.e., the target speaker, in multi-talker environments [ 2 , 3 ] . Inspired by this capability, researchers have spent decades trying to mimic this behavior in machines, a challenge commonly referred to as the “cocktail party problem” [ 7 ] . Speech separation is one prominent direction of research aimed at addressing the cocktail party problem. It seeks to decompose a multi-talker speech signal into individual clean streams by leveraging the unique dynamics or prosody of each speaker’s voice [ 10 , 15 , 14 , 33 , 28 ] . However, the resulting separated streams lack explicit association with speaker identity, making it difficult to identify which output corresponds to the target speaker. Moreover, most methods require prior knowledge of the number of speakers for optimal performance, limiting their applicability in realistic, dynamic environments. Speaker extraction algorithms, in contrast, more closely mimic the human ability to selectively extract the speech of a target speaker from a mixture, by leveraging an auxiliary cue to guide attention. Examples include pre-enrolled reference utterances [ 37 , 32 , 8 ] , lip movements from video recordings [ 34 , 23 , 20 ] , directional cues [ 6 ] , subject’s neural signals [ 4 , 19 , 25 ] , or other speaker-unique characteristics [ 30 , 5 ] to disentangle a specific voice from overlapping speech. Among these, visual cues are especially popular in human-robot interaction scenarios, as onboard cameras can easily capture synchronized video without requiring a pre-enrollment or selection process. Among visual cues, the majority of existing work has focused on lip movements as the primary modality. However, the lips may often be occluded, or difficult to be detected and tracked reliably in low-resolution or distant-view scenarios. In contrast, the upper body remains more visible from a distance and is less prone to full occlusion. Embodied human communication involves a rich interplay between verbal (speech) and non-verbal behaviors, including body posture, hand gestures, and head nods [ 1 , 11 ] . Notably, co-speech gestures are tightly synchronized with speech and highly correlated with its conceptual content and prosodic structure. Neuroscience studies further suggest that these gestures play a functional role in both speech production and perception, enhancing comprehension for listeners [ 31 , 17 ] . The SEG model has utilized co-speech gestures as a cue for target speaker extraction [ 21 ] . However, the correlation between gesture and speech is less direct and weaker compared to the lip-speech relationship, leading to a performance gap. In this work, for the first time, we aim to leverage both lip and gesture cues for robust speaker extraction, which is particularly useful in scenarios where one modality is missing or degraded. The SeLG model (Speech extraction with Lip and co-speech Gesture cues) is proposed. We first introduce the cross-attention fusion method that uses each visual cue to query and select relevant speech features from the mixture as opposed to simple concatenation fusion in previous works [ 21 , 20 ] . We also propose to enhance the discriminative representation of gesture embeddings to better correlate with the speech dynamics, by aligning them with lip motion using an InfoNCE loss, which pulls gesture embeddings closer to corresponding lip embeddings. Experimental results on the “in the wild” YouTube Gesture Dataset (YGD) [ 36 ] , which includes TED talks, demonstrate that the proposed contrastive learning strategy significantly improves gesture-based extraction. Moreover, the attention-based fusion of lip and gesture cues outperforms the concatenation baseline, with the best performance achieved when combined with the InfoNCE loss. Improvements are observed in both two- and three-speaker mixtures, as well as across scenarios with and without missing modalities. 2 Methodology Let x(\tau) 1 1 1 Variables indexed by \tau denote time-domain signals, while those indexed by t denote frame-based embeddings. be a multi-talker mixture speech signal, composed of the target speech signal s(\tau) and interference speech signal b(\tau) . The audio-visual speaker extraction (AVSE) network we studied in this work estimates the target speech signal \hat{s}(\tau) to approximate s(\tau) , conditioned on the synchronized gesture sequence v_{g}(t) , or the combination of v_{g}(t) and the lip sequence v_{l}(t) of the target speaker. 2.1 Model architecture Our proposed model is shown in Fig. 1 , which consists of a gesture encoder, a lip encoder, a speech encoder, a speech decoder, and an ensemble separator. The separator comprises lip and gesture attention layers along with a mask estimator, as shown in the dotted box. 2.1.1 Gesture encoder The input to the gesture encoder v_{g}(t) is a sequence of human upper-body poses, represented as 3-dimensional (3D) coordinates of ten spine-centered joints: head, neck, nose, spine, left/right (L/R) shoulders, L/R elbows, and L/R wrists. The 3D coordinates are obtained by first applying a gesture estimation algorithm to the raw video to extract 2D pose sequences [ 36 ] , which are then lifted to 3D using a pose estimator [ 26 ] . The resulting 3D pose sequence v_{g}(t) is passed through the gesture encoder, which is an R_{g} -layer Bidirectional Long Short-Term Memory (BLSTM), that models the temporal dynamics of gestures, yielding the encoded representation V_{g}(t) that is in sync with the target’s

Read the complete text