Understanding Co-speech Gestures in-the-wild

Sindhu B Hegde, K R Prajwal, Taein Kwon, Andrew Zisserman · 2025

Read Understanding Co-speech Gestures in-the-wild by Sindhu B Hegde, K R Prajwal, Taein Kwon, Andrew Zisserman (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2503-22668.txt.

Introduction

Sindhu Hegde and co-authors present a 2025 paper on hand and arm gestures that go with speech. Gestures carry meaning that words alone miss. The team sets three new tasks to test how well a model grasps the link among gesture, speech, and text. Tasks cover finding a clip from a gesture, spotting a word in a gesture stream, and picking the speaker.

Their model learns a shared space for video, speech, and text. It trains with both broad phrase matches and fine word-level links. The learned gesture features beat prior methods and large vision-language models. Speech and text each carried distinct clues about gesture. All data, code, and the model are free online for others.

Catalog record

CREATORSindhu B Hegde, K R Prajwal, Taein Kwon, Andrew Zisserman
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE48 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2503-22668

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Understanding Co-speech Gestures in-the-wild Sindhu B Hegde, K R Prajwal, Taein Kwon, Andrew Zisserman Abstract Co-speech gestures play a vital role in non-verbal communication. In this paper, we introduce a new framework for co-speech gesture understanding in the wild. Specifically, we propose three new tasks and benchmarks to evaluate a model’s capability to comprehend gesture-text-speech associations: (i) gesture-based retrieval, (ii) gestured word spotting, and (iii) active speaker detection using gestures. We present a new approach that learns a tri-modal speech-text-video-gesture representation to solve these tasks. By leveraging a combination of global phrase contrastive loss and local gesture-word coupling loss, we demonstrate that a strong gesture representation can be learned in a weakly supervised manner from videos in the wild. Our learned representations outperform previous methods, including large vision-language models (VLMs), across all three tasks. Further analysis reveals that speech and text modalities capture distinct gesture-related signals, underscoring the advantages of learning a shared tri-modal embedding space. The dataset, model, and code are available at: https://www.robots.ox.ac.uk/~vgg/research/jegal . 1 Introduction Humans gesture when they talk – gesturing is an integral part of human communication, together with speech and facial expressions. Gestures can vary from beats – two phase hand movements (up/down, left/right etc) that emphasize particular words or phrases and match the rhythm of the speech, but do not carry semantic content – to iconic and deictic gestures that are representational and illustrate the content of the speech [ 9 , 34 ] . For example, hands and arms moving apart can accompany a speech segment indicating that something is “huge”, or as illustrated in Fig 1 , an inward pointing gesture to depict the uttered word “my”. Non-verbal communication accounts for 55\% of overall communication 1 1 1 https://online.utpb.edu/about-us/articles/communication/how-much-of-communication-is-nonverbal/ , highlighting the need for machines to understand non-verbal gestural elements in order to have a holistic understanding of human communication. A clear application is enriching human-computer interaction (HCI) through gestures, and this requires machines to comprehend the semantics of the user’s hand gestures. Another application is to detect if a person is speaking based on their gestures, or spot specific words or phrases in a video based on gestures alone. More generally, being able to recognize gestures and determine their semantic and temporal alignment with speech enables human communication to be studied at scale [ 26 ] . In this paper, our objective is to learn and evaluate co-speech gesture representations. To this end, we propose three tasks and evaluation benchmarks that act as a proxy for assessing real world applications: (1) gesture-based cross-modal retrieval, (2) gesture word spotting, and (3) active speaker detection via gestures. We perform large-scale training featuring \approx 7000 speakers and evaluate on in-the-wild videos from the AVSpeech dataset [ 20 ] . Figure 1 : Co-speech gestures supplement the spoken language – we show examples for three phrases here, with common words. Learning to associate gestures with the uttered phrases is essential for a holistic understanding of human communication. All three tasks require models to learn to associate gesture clips with speech segments or their corresponding textual transcripts. For this, we propose a model termed J oint E mbedding space for G estures, A udio, and L anguage ( JEGAL ) 2 2 2 JEGAL, known as Zhuge Liang in Chinese, was a prominent historical figure from China’s Three Kingdoms period and is regarded as a symbol of wisdom. that facilitates matching gestures to words and phrases in the accompanying speech. The matches can be based on the style of the speech (intonation, stress, prosody) or the semantic content of the phrase or particular words. However, learning a rich joint gesture-audio-language embedding space is a very challenging task. The associations between gestures and speech are typically sparse and ambiguous, with a high degree of variability across speakers. Usually, only a few of the spoken words are clearly gestured. Additionally, the same sentence can be gestured very differently in different contexts and by different people. Gestures also depend on the speaker’s emotion, culture, and social scenario (formality, private vs. public, with friends or strangers, etc.). Furthermore, some types of gestures, such as beat gestures, carry no semantic information, resulting in no direct mapping from the gesture to words. The sparse and weak cross-modal correlations makes gesture representation learning a very unique research problem. We make three key design choices in our approach that result in a strong tri-modal gesture representation. To start with, we learn gesture video representations from large-scale weak cross-modal supervision. The supervision is weak because we only use phrase-level speech audio and transcripts – since we do not have any information on which words in the speech are gestured for videos collected ‘in-the-wild’. Second, we obtain cross-modal supervision in the form of both audio and the corresponding text transcript (in Section 6.1 , we demonstrate that speech and text modalities capture complementary gesture-related signals). Third, we introduce a new gesture-word alignment and spotting loss that strongly encourages learning of word-level correspondences. To summarize, we make the following contributions: (i) we propose a new framework for co-speech gesture understanding with three new tasks and evaluation benchmarks; (ii) we learn a joint tri-modal embedding space in a weakly-supervised manner with a combination of global phrase-level objective, and a local word-level gesture coupling loss; (iii) we demonstrate that the learned JEGAL representation outperforms previous vision-language foundation models on the three gesture-centric tasks and is useful for practical applications. 2 Related Work Figure 2 : The JEGAL architecture. The three input modalities (video, text, speech) are each encoded with a modality-specific encoder, followed by a fusion block to merge speech and text representations. The encoder outputs are average-pooled to obtain global (phrase-level) gesture and speech-text embeddings. During training, these provide the inputs for the global ‘phrase contrastive loss’. The gesture alignment module aggregates the relevant video frames to obtain a local word-level gesture embedding for each speech-text word. During training, these provide the inputs for the local ‘gesture-word coupling loss’. The two losses encourage the learning of global and local correspondences between the three modalities. Spectrum of Human Gestures. Gestures can be classified broadly into four major classes [ 34 ] . Emblematics convey a clear symbolic meaning, e.g. a “thumbs-up”. Iconic gestures are used to convey meaning co-occurring with the articulated speech, e.g. “revolving door” is accompanied with the hands moving in a circular motion. Deictic gestures are pointing gestures using the index finger. The most common are the “beat” gestures which are co-speech gestures that are temporally aligned with the prosodic characters of human speech. They are prominent on lexically stressed syllables. Past works [ 55 , 26 ] have studied how these gesture classes relate to the speech. Several papers have been attempted to recognize hand, head, and facial emblematic gestures [ 22 , 18 ] . Recognizing deictic gestures can help in identifying the referring object being pointed to in a conversation, an essential part of human-robot interaction [ 33 ] . The other two gesture classes, i.e., beat and iconic, are quite diverse and are difficult to associate with a well-defined set of words or hand movements. This work takes a data-driven approach to learn gesture representations with the help of speech and

Read the complete text