GestureLSM: Latent Shortcut based Co-Speech Gesture Generation with Spatial-Temporal Modeling

Pinxin Liu, Luchuan Song, Junhua Huang, Haiyang Liu, Junfan Zhu, Chenliang Xu · 2025

Read GestureLSM: Latent Shortcut based Co-Speech Gesture Generation with Spatial-Temporal Modeling by Pinxin Liu, Luchuan Song, Junhua Huang, Haiyang Liu, Junfan Zhu, Chenliang Xu (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2501-18898.txt.

Introduction

Digital figures that move as they speak look more natural, but making a full body move in time with speech is slow and hard. Pinxin Liu and co-authors present GestureLSM, a 2025 method built for speed and smooth motion.

Older tools treat the body, legs, and hands apart, so moves can look split and out of sync. GestureLSM handles all parts together, with steps that track links across the body and across time. It also uses a faster way to sample motion, trained with a shortcut trick that cuts the steps needed. On a standard speech and gesture test set, it matched the best quality while running much faster than prior tools.

Catalog record

CREATORPinxin Liu, Luchuan Song, Junhua Huang, Haiyang Liu, Junfan Zhu, Chenliang Xu
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE39 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2501-18898

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

GestureLSM: Latent Shortcut based Co-Speech Gesture Generation with Spatial-Temporal Modeling Pinxin Liu, Luchuan Song, Junhua Huang, Haiyang Liu, Junfan Zhu, Chenliang Xu Abstract Generating full-body human gestures based on speech signals remains challenges on quality and speed. Existing approaches model different body regions such as body, legs and hands separately, which fail to capture the spatial interactions between them and result in unnatural and disjointed movements. Additionally, their autoregressive/diffusion-based pipelines show slow generation speed due to dozens of inference steps. To address these two challenges, we propose GestureLSM , a flow-matching-based approach for Co-Speech Gesture Generation with spatial-temporal modeling. Our method i) explicitly model the interaction of tokenized body regions through spatial and temporal attention, for generating coherent full-body gestures. ii) introduce the flow matching to enable more efficient sampling by explicitly modeling the latent velocity space. To overcome the suboptimal performance of flow matching baseline, we propose latent shortcut learning and beta distribution time stamp sampling during training to enhance gesture synthesis quality and accelerate inference. Combining the spatial-temporal modeling and improved flow matching-based framework, GestureLSM achieves state-of-the-art performance on BEAT2 while significantly reducing inference time compared to existing methods, highlighting its potential for enhancing digital humans and embodied agents in real-world applications. Project Page: https://andypinxinliu.github.io/GestureLSM Figure 1 : GestureLSM. We present Gesture Latent Shortcut Model, a method that generates full-body human gestures from speech and text scripts with high quality and real-time speed. It explicitly models the body regions interactions, e.g. , the interactions between body and hands, to achieve coherent gesture motions. Besides, it is also capable of real-time generation based on shortcut sampling. 1 Introduction Speech is often accompanied by gestures that serve to convey emotions and enhance comprehension [ 8 , 50 ] within daily conversations. These non-verbal cues play a crucial role in effective communication [ 2 ] , with gesture generation serving as a fundamental component of human-computer interaction. As artificial intelligence continues to advance, endowing virtual avatars with realistic gestural capabilities will be essential for fostering immersive interactive experiences and digital avatar construction [ 53 , 49 , 20 , 46 , 48 , 47 ] . Many recent works [ 58 , 33 , 29 , 60 ] have conducted preliminary studies on full-body co-speech gesture generation. These methods typically treat different body regions as separated motion features, represented using either continuous [ 17 ] or discrete VQ tokens [ 29 , 60 ] . While these approaches have demonstrated plausible gesture patterns for individual body regions, they often produce coherent full-body movements. This issue stems from their separated motion features learning process, which overlooks the nuanced interactions between different body parts. For example, when expressing the sentence “I completely agree” , natural gestures involve intricate coordination. The fingers may point or emphasize, the arms may extend outward, and the torso may subtly shift, all contributing to the reinforcement of affirmation. However, separated motion decoding constraints models from learning such fine-grained interactions, leading to uncoordinated and unnatural gesture patterns. These limitations not only hinder the realism of generated gestures but also restrict their applications in industry-grade animations. Moreover, many existing models [ 5 , 4 ] employ either diffusion-based approaches or auto-regressive generation methods [ 60 ] . The diffusion-based approaches operate through iterative denoising processes, whereas auto-regressive models generate gestures sequentially. Both approaches introduce substantial computational overhead, thereby hindering real-time inference and posing significant challenges for real-world deployment. To address the above issues, we propose GestureLSM : a flow-matching-based approach for Co-Speech Gesture Generation with spatial-temporal modeling.. Our method explicitly models the interactions between different body parts by segmenting the human body into the upper body, hands, lower body, and facial expressions. To fully capture spatial-temporal dependencies, we first apply spatial attention to ensure coherence between body regions at each time step, followed by temporal attention to model motion progression. This sequential design enables our model to learn both intra-frame structural relationships and inter-frame motion dynamics. In addition, we introduce the flow matching to enable more efficient sampling by explicitly modeling the latent velocity space. Even though flow matching is potentially faster and easier for learning in theory, we found the vanilla flow matching yields suboptimal results’ quality and speed, i.e. , underperform the diffusion-based models. To solve this, we adopt latent shortcut learning and beta distribution time stamp sampling during training to enhance gesture synthesis quality and accelerate inference, which are the keys to enable flow matching based model outperform diffusion based model on our task. Combining the spatial-temporal modeling and improved flow matching framework, GestureLSM achieves state-of-the-art performance on BEAT2 while significantly reducing inference time compared to existing methods. In summary, our primary contributions are: • We present the GestureLSM , a framework that achieves high quality and real-time co-speech human gesture generation. • We develop the spatial-temporal modeling that enables cohernet gesture generation by capturing and integrating interactions across different body regions. • We improve the flow matching-based to outperform diffusion-based models for gesture generation, in particular, we introduce the latent shortcut model and optimize time-sampling strategies during training to accelerate inference speed and benefit the gesture synthesis. Figure 2 : Our GestureLSM achieves significant generation quality improvement over baseline methods with fastest inference speed. The inference time is computed on one NVIDIA A100 while the generation quality is from FGD on BEAT2. 2 Related Works Co-speech Gesture Generation Existing works on co-speech gesture generation mostly employ skeleton- or joint-level pose representations. [ 15 ] use an adversarial framework to predict hand and arm poses from audio, and leverage conditional generation [ 3 ] based on pix2pixHD [ 56 ] for videos. Some recent works [ 34 , 9 , 59 , 28 , 63 , 33 , 51 , 32 , 31 ] learn the hierarchical semantics or leverage contrastive learning to obtain joint audio-gesture embeding to assist the gesture pose generation. HA2G [ 34 ] construct high and low level audio-motion embedding for gesture generation. TalkShow [ 60 ] estimates SMPL [ 43 ] poses, and models the body and hand motions for talk-show videos. CaMN [ 30 ] and EMAGE [ 29 ] propose a large scale conversational and speech datasets for joint face and body modeling with diverse style control, with GPT-style decoding for gesture generation. MambaTalk [ 58 ] speeds up the generation process with an efficient mamba structure. Semantic Gesticulator [ 64 ] enhances gesture generation with semantic annotation of gesture types and triggers for retrieval. DiffSHEG [ 5 ] and SynTalker [ 4 ] build up a diffusion-based gesture generation pipeline. However, none of these works considers the interactions of different body regions conditioned on speech input and achieves fast and real-time gesture generations. Fast Diffusion Sampling Diffusion models [ 18 , 38 , 24 , 39 , 40 ] have demonstrated impressive generation quality across various modalities [ 44 , 1 , 42 ] , but suffer from slow inference speeds due to their iterative sampling process. To address these efficiency challenges, several approaches have been proposed, such as Consistency Models [ 52 , 41 ] and Diffusion Distillations [ 45 , 61 ] . However, these methods still face limitations in terms of training speed and flexibility. One

Read the complete text