SemGes : Semantics-aware Co-Speech Gesture Generation using Semantic Coherence and Relevance Learning

Lanmiao Liu Esam Ghaleb Aslı Özyürek, Zerrin Yumak · 2025

Read SemGes : Semantics-aware Co-Speech Gesture Generation using Semantic Coherence and Relevance Learning by Lanmiao Liu Esam Ghaleb Aslı Özyürek, Zerrin Yumak (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2507-19359.txt.

Introduction

Liu and co-authors build an avatar that gestures in line with what it says. Past systems made beat gestures that matched rhythm but not meaning. SemGes adds meaning at both the word level and the whole-speech level. It first learns a motion code with a vector-quantized autoencoder. A second stage then makes gestures from speech, text meaning, and speaker identity.

Two modules keep the gestures tied to the speech content. One checks local fit, and one checks broad sense across the clip. Tests and user studies beat prior methods on two benchmarks. Gains showed in both hard scores and viewer ratings. Code, data, and trained models are posted on the project site.

Catalog record

CREATORLanmiao Liu Esam Ghaleb Aslı Özyürek, Zerrin Yumak
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE36 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2507-19359

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

SemGes : Semantics-aware Co-Speech Gesture Generation using Semantic Coherence and Relevance Learning Lanmiao Liu Esam Ghaleb Aslı Özyürek, Zerrin Yumak Abstract Creating a virtual avatar with semantically coherent gestures that are aligned with speech is a challenging task. Existing gesture generation research mainly focused on generating rhythmic beat gestures, neglecting the semantic context of the gestures. In this paper, we propose a novel approach for semantic grounding in co-speech gesture generation that integrates semantic information at both fine-grained and global levels. Our approach starts with learning the motion prior through a vector-quantized variational autoencoder. Built on this model, a second-stage module is applied to automatically generate gestures from speech, text-based semantics and speaker identity that ensures consistency between the semantic relevance of generated gestures and co-occurring speech semantics through semantic coherence and relevance modules. Experimental results demonstrate that our approach enhances the realism and coherence of semantic gestures. Extensive experiments and user studies show that our method outperforms state-of-the-art approaches across two benchmarks in co-speech gesture generation in both objective and subjective metrics. The qualitative results of our model, code, dataset and pre-trained models can be viewed at https://semgesture.github.io/ . 1 Introduction Figure 1 : SemGes integrates audio, text-based semantics, and speaker identity to produce both contextually relevant (discourse-level) and fine-grained (local) gestures. A semantic coherence module aligns text and motion embeddings. The multimodal consistency loss synchronizes the quantized multimodal representations to match the quantized learned motion features for final speech-driven semantics-aware gesture generation. The semantic relevance loss selectively emphasizes gestures with semantic annotations. Human language is inherently multimodal, with gestures and speech complementing each other to convey pragmatic and semantic information [ 35 , 21 ] . Co-speech gestures are non-verbal cues that are uniquely related to co-occurring speech, pragmatically, semantically, and temporally. For example, representational iconic gestures that visually express the semantic content of speech and interact with spoken language [ 21 , 19 , 41 , 14 , 13 ] . A long-standing goal in Computer Vision is to create digital humans that use non-verbal cues in sync with speech. Gesture generation—synthesizing movements from co-occurring speech, masked motion, or speaker identity—has advanced to enhance AI agents’ expressiveness and realism [ 29 ] . However, much of the focus has gone into generating rhythmic beat gestures with limited semantic information, leaving representational gestures that convey semantic messages (e.g., iconic) less explored [ 29 , 40 ] . Generating spontaneous and semantically rich gestures from speech comes with multiple challenges. First, it requires capturing global discourse-level information and local fine-grained details (e.g., salient words) to generate speech-driven gestures that reflect the intended meaning and align with speech temporally and semantically. Second, existing methods often generate repetitive and short sequences that do not span the full range of expressive motions required for natural communication. To leverage semantics when generating gestures, researchers have attempted to align motion with speech representations at a global level, e.g., by leveraging pre-trained semantic representations such as CLIP [ 58 ] or focusing on semantically important keywords [ 6 , 57 ] . Nonetheless, they often fail to (i) unify global and local semantic modelling within a single framework and (ii) exploit the relevance of the semantic information in guiding gesture generation [ 27 ] . At the same time, raw audio features and speaker identity are relevant to the timing and style of gestures. In this paper, we address these limitations by integrating speech, speech semantics, and gesturing style, exploiting semantic information at different levels. Specifically, we propose a two-stage framework, namely, SemGes, that integrates speech, text-based semantics, and speaker identity into a unified gesture-generation model (see Figure 1 ). In stage 1, we build motion prior of holistic gestures ( i.e . , body and hands) by training a vector-quantized variational autoencoder (VQ-VAE) to learn an efficient, compositional motion latent space. This stage results in a robust motion encoder & decoder and quantized codebooks that can reconstruct naturalistic gestures while allowing the reuse of learned codebook entries. Stage 2 leverages the learned motion priors to drive gesture synthesis by fusing three modalities using a cross-modal Transformer encoder: (i) text-based semantics, (ii) raw-audio speech features, and (iii) speaker identity for style consistency. We impose a semantic coherence loss that aligns text-based embeddings with the VQ-VAE motion latent space and a semantic relevance loss that emphasises representational gestures ( e.g . iconic and metaphoric gestures). A multimodal consistency objective ensures the fused multimodal representations are compatible with the learned motion codebooks, enabling the generation of gestures that are both semantically rich and visually natural. Finally, we introduce a simple but effective long-sequence inference strategy that smoothly combines overlapping motion clips for extended durations. To summarize our contributions, • We introduce a novel framework, SemGes , that first learns a robust VQ-VAE motion prior for body and hand gestures, and then generates gestures driven by fused speech audio, text-based semantics, and speaker identity in a cross-modal transformer. • Our method jointly captures discourse-level context via a semantic coherence loss and fine-grained representational gestures (e.g., iconic, metaphoric) via a semantic relevance loss. • We propose an overlap-and-combine inference algorithm that maintains smooth continuity over extended durations. • Extensive experiments on two benchmarks, namely, the BEAT [ 27 ] and TED Expressive [ 33 ] datasets show that our method outperforms recent baselines in both objective metrics (e.g., Fréchet Gesture Distance (FGD), diversity, semantic alignment) and user judgment of generated gestures. 2 Related Work Data-driven Co-Speech Gesture Generation. Current gesture generation approaches are based on generative deep neural networks. These approaches use advanced models such as Transformers [ 28 ] , Generative Adversarial Networks [ 23 ] , Normalizing Flows [ 18 , 30 ] , Vector Quantized Variational Autoencoder(VQ-VAE) [ 16 ] and Denoising Diffusion Probabilistic Models [ 47 ] . In addition, researchers have explored the impact of different model inputs on the naturalness and appropriateness of generated gestures. Various modal inputs have been used, such as text [ 15 ] , audio [ 59 , 50 ] , image [ 32 , 39 ] , and speaking style [ 3 ] . For a comprehensive survey, we refer to Nyatsanga et al. 2023 . Although there have been significant improvements in this field, current methods fall short in generating semantically grounded gestures at a fine-grained level. In other words, while the generated motions look convincing at first glance, they do not match well with the meaning of the text, or they mostly focus on beat-type gestures. Semantics-aware Co-Speech Gesture Generation. A group of work focused on semantics-aware gesture generation where the semantic information is handled in two ways: global semantics and local semantics. Methods that focus on global semantic information [ 58 , 10 , 22 ] align gestures with text or audio, but they fall short in generating gesture types matching the semantic context, such as iconic, metaphoric and deictic gestures. To capture a wider range of semantic gestures, works like [ 5 , 6 , 27 ] adopt local semantic-aware modelling by integrating the semantic salient words to the neural network. However, these approaches often fail to ensure that the generated gestures align with both the broader audio or textual context and a combination

Read the complete text