HoloGest: Decoupled Diffusion and Motion Priors for Generating Holisticly Expressive Co-speech Gestures
Yongkang Cheng, Shaoli Huang · 2025
Read HoloGest: Decoupled Diffusion and Motion Priors for Generating Holisticly Expressive Co-speech Gestures by Yongkang Cheng, Shaoli Huang (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2503-13229.txt.
Introduction
Virtual speakers look stiff when their hands and faces barely move with their words. Yongkang Cheng and Shaoli Huang present a system called HoloGest to fix that in 2025. Past systems leaned too hard on the audio track, which made motion look unnatural. Their design splits the task: one part learns how bodies move, and one part adds speech timing.
Large motion data sets taught the system a strong base of real movement, down to the fingers. Built-in joint rules and shape limits keep the output smooth and fast to make. A shared space links gestures with the words they match, so moves fit meaning. User tests rated the output close to real human motion. Code, the model, and a demo were shared in public.
Catalog record
| CREATOR | Yongkang Cheng, Shaoli Huang |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 37 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2503-13229
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
HoloGest: Decoupled Diffusion and Motion Priors for Generating Holisticly Expressive Co-speech Gestures Yongkang Cheng, Shaoli Huang Abstract Animating virtual characters with holistic co-speech gestures is a challenging but critical task. Previous systems have primarily focused on the weak correlation between audio and gestures, leading to physically unnatural outcomes that degrade the user experience. To address this problem, we introduce HoleGest , a novel neural network framework based on decoupled diffusion and motion priors for the automatic generation of high-quality, expressive co-speech gestures. Our system leverages large-scale human motion datasets to learn a robust prior with low audio dependency and high motion reliance, enabling stable global motion and detailed finger movements. To improve the generation efficiency of diffusion-based models, we integrate implicit joint constraints with explicit geometric and conditional constraints, capturing complex motion distributions between large strides. This integration significantly enhances generation speed while maintaining high-quality motion. Furthermore, we design a shared embedding space for gesture-transcription text alignment, enabling the generation of semantically correct gesture actions. Extensive experiments and user feedback demonstrate the effectiveness and potential applications of our model, with our method achieving a level of realism close to the ground truth, providing an immersive user experience. Our code, model, and demo are are available at https://cyk990422.github.io/HoloGest.github.io/. Figure 1 : A comparison of three methods: DSG, a diffusion-based co-speech gesture generation method using DDPM (stiff limbs, slow inference, physically unnatural); EMAGE, an autoregressive generation method using VAE (motion artifacts, global flipping, physically unnatural); and our proposed generation method (rich movements, lively fingers, physically natural). The transition from past frames to the current frame (every 10 frames) is represented by the gradient in virtual human color, from light to dark. 1 Introduction Animating virtual characters with holistic co-speech gestures is a challenging but critical task in various fields such as entertainment, education, and telecommunication. These gestures play a vital role in enhancing the naturalness and appeal of virtual characters, as they convey non-verbal information and improve the overall communication experience. However, generating holistic co-speech gestures that accurately represent the complex interplay between audio and body motion remains a challenging task, primarily due to the weak correlation between audio and both global motion trajectory and finger movements. This weak correlation often leads to physically unnatural outcomes, such as jittering in the global motion trajectory and poor expressiveness of finger motions, significantly reducing the effect of virtual characters. Previous co-speech gesture generation methods can be divided into two categories: VAE-based or VQ-VAE-based generation systems [ 24 , 26 ] and diffusion-based generation systems [ 40 , 49 ] . The former maps weakly correlated gesture-audio pairs to a low-dimensional latent space and learns a continuous probability distribution, from which new latent vectors are sampled and decoded to obtain co-speech gestures. However, due to the VAE’s reconstruction loss [ 21 ] based on joint-level errors and the ambiguity in the latent space, the generated gestures often appear overly smooth or unnatural. In contrast, diffusion-based methods model gesture generation as a gradual diffusion process, where the mapping relationship between audio and gestures is gradually established through a series of noise diffusion steps. Compared to VAE, this method can generate gesture sequences with rich details while maintaining audio synchrony. However, the high computational density and the resulting time cost limit the further development of diffusion methods. Furthermore, both approaches lack consideration of prior knowledge in motion, focusing only on the weakly correlated mapping between audio and upper-body gestures, and neglecting the physical laws of overall movement, such as continuity, stability, and rationality. Therefore, the generated holistic gestures may exhibit unnatural sliding, hovering phenomena, and monotonous finger movement, leading to a lack of overall naturalness and expressiveness, as shown in Figure 1 . To address these challenges, we introduce HoloGest, a novel diffusion-based framework for automatically synthesizing high-fidelity holistic co-speech gesture sequences from audio. Our system posits that in holistic co-speech gestures, limb movements are correlated with audio, global trajectories are related to limbs and independent of audio, and fingers are associated with both arm movements and audio information. Based on this, we learn a global trajectory diffusion generative prior model guided by limb movements on a large-scale human motion dataset. Simultaneously, we learn a finger diffusion generative prior model guided by the arms on a mixed sign language and gesture dataset, leaving audio and semantic features blank for subsequent fine-tuning. The former provides our system with strong locomotion prior, overcoming long-standing issues of unnatural sliding and jittering, while the latter offers more diverse finger movements, assisting in generating more vivid and high-fidelity gesture results. Unlike previous methods that model the denoising process of the whole body as a single distribution, our system decouples the upper limbs, lower body, and fingers into three smaller and simpler subproblems, breaking down holistic co-speech gestures. During the denoising process, each sub-model focuses more on the distribution of specific body parts, thereby improving the generation quality of each part. However, the parallel diffusion denoising processes for the three parts further reduce the generation efficiency. To break free from this constraint, we employ a semi-implicit constraint [ 37 ] , modeling large-stride complex multimodal distributions between adjacent denoising steps to significantly reduce the required number of denoising steps, thus achieving acceleration. Predicting gestures from speech is a challenging multimodal mapping task. A single speech segment can correspond to multiple gestures, making the association between the semantics intended to be conveyed in speech and gestures non-intuitive. Our system adopts the JEPA strategy [ 13 , 6 ] to learn a gesture-speech joint embedding space. We first introduce wav2vec2 [ 5 ] for text transcription, then extract textual features and map them with gestures to a shared low-dimensional space based on a variational autoencoder. Finally, we introduce a predictor layer to further extract semantic features, aligning these abstract semantic features with the low-dimensional latent variables of gestures in this space. This approach maintains semantic alignment while generating natural gestures closely related to speech. To demonstrate the inspirational value of motion priors in our system for the human motion generation domain, we further fine-tune our framework on the music-to-dance task, addressing the physical naturalness of generated results and showcasing its powerful generalization capabilities. To the best of our knowledge, our system represents the first audio-whole body gesture generation model considering motion priors, capable of generating high-fidelity, diverse, and physically natural holistic co-speech gesture sequences based on arbitrary user-provided audio (speech or music). We showcase our approach on multiple publicly available audio-motion datasets, and extensive experimental results indicate that, compared to VAE systems, our method generates more diverse and higher-quality results, while maintaining the naturalness of overall motion compared to diffusion systems, significantly reducing time costs and providing users with a novel experience. The importance of algorithmic design is also validated through ablation experiments. 2 Related Work Audio-to-motion Generation. Initial data-driven methods ( [ 26 , 15 ] ) aimed to learn gesture matching from human demonstrations but lacked diversity. With the increasing interest in these action reconstruction methods [ 9 , 22 , 44 ] , the training datasets have gradually become more abundant. Subsequent works ( [ 15 , 41 , 36 , 10 , 11 ] ) improved model diversity and introduced unique,