Cosh-DiT: Co-Speech Gesture Video Synthesis via Hybrid Audio-Visual Diffusion Transformers
Yasheng Sun, Zhiliang Xu, Hang Zhou, Jiazhi Guan, Quanwei Yang, Kaisiyuan Wang, Borong Liang, Yingying Li, Haocheng Feng, Jingdong Wang, Ziwei Liu, Koike Hideki · 2025
Read Cosh-DiT: Co-Speech Gesture Video Synthesis via Hybrid Audio-Visual Diffusion Transformers by Yasheng Sun, Zhiliang Xu, Hang Zhou, Jiazhi Guan, Quanwei Yang, Kaisiyuan Wang, Borong Liang, Yingying Li, Haocheng Feng, Jingdong Wang, Ziwei Liu, Koike Hideki (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2503-09942.txt.
Introduction
Sun and a large team present Cosh-DiT in twenty twenty-five, a system that makes videos of people moving as they speak. Good gesture videos must match the beat of speech and still look real. Hands, face, and upper body all move in linked ways. Past methods often made stiff or blurred results.
The system works in two stages with two model types. The first stage turns speech sound into body motion, beat by beat. The second stage turns that motion into video frames, using space and time cues. Tests showed lifelike videos with smooth gestures and clear facial moves. The authors say the split design is the key to the gain.
Catalog record
| CREATOR | Yasheng Sun, Zhiliang Xu, Hang Zhou, Jiazhi Guan, Quanwei Yang, Kaisiyuan Wang, Borong Liang, Yingying Li, Haocheng Feng, Jingdong Wang, Ziwei Liu, Koike Hideki |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 46 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2503-09942
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Cosh-DiT: Co-Speech Gesture Video Synthesis via Hybrid Audio-Visual Diffusion Transformers Yasheng Sun, Zhiliang Xu, Hang Zhou, Jiazhi Guan, Quanwei Yang, Kaisiyuan Wang, Borong Liang, Yingying Li, Haocheng Feng, Jingdong Wang, Ziwei Liu, Koike Hideki Abstract Co-speech gesture video synthesis is a challenging task that requires both probabilistic modeling of human gestures and the synthesis of realistic images that align with the rhythmic nuances of speech. To address these challenges, we propose Cosh-DiT , a Co-s peech gesture video system with h ybrid Di ffusion T ransformers that perform audio-to-motion and motion-to-video synthesis using discrete and continuous diffusion modeling, respectively. First, we introduce an audio Diffusion Transformer (Cosh-DiT-A) to synthesize expressive gesture dynamics synchronized with speech rhythms. To capture upper body, facial, and hand movement priors, we employ vector-quantized variational autoencoders (VQ-VAEs) to jointly learn their dependencies within a discrete latent space. Then, for realistic video synthesis conditioned on the generated speech-driven motion, we design a visual Diffusion Transformer (Cosh-DiT-V) that effectively integrates spatial and temporal contexts. Extensive experiments demonstrate that our framework consistently generates lifelike videos with expressive facial expressions and natural, smooth gestures that align seamlessly with speech. keywords Audio-Visual Learning, Co-Speech Video Synthesis, Diffusion Transformers † † equal-contributors: These authors contributed equally to this work. † † equal-contributors: These authors contributed equally to this work. 1 Introduction Driving a portrait with audio is of significant importance in a variety of applications Seeger et al. (2021) ; Pizzi et al. (2023) ; Adam et al. (2021) ; Roesler et al. (2021) ; Seitz et al. (2022) , such as digital human creation, human-computer interaction, and live-streaming e-commerce. Many works Zhou et al. (2021) ; Thies et al. (2020) ; Li et al. (2021b) ; Park et al. (2022) ; Cheng et al. (2022) ; Ye et al. (2023) ; Fan et al. (2022) on this domain focus on facial animation around head region, which limits their applications. It is essential to synthesize gesture movements that conform to human speech to present a comprehensive virtual human solution. For the task of co-speech gesture synthesis, most studies Ahuja et al. (2020a) ; Ahuja et al. (2020b) ; Li et al. (2021a) ; Liu et al. (2022c) ; Ao et al. (2023) ; Zhu et al. (2023) only target on intermediate 3D motion representations such as 3D key points and 3D parametric models Pavlakos et al. (2019b) ; Loper et al. (2015) . Despite advancing AR/VR and gaming Fu et al. (2022) ; Farouk (2022) , these approaches are less suited for tasks like virtual anchor creation Huang et al. (2024) , where realistic videos provide a more vivid and immersive experience. However, previous co-speech gesture studies cannot be easily transformed into the real-world domain. Specifically, videos with co-speech gestures normally record only the upper body of a target person, making 3D reconstruction methods Pavlakos et al. (2019b) ; Cai et al. (2023) unstable. Particularly when the target is moving, without capturing the legs and feet, 3D reconstruction systems would find it hard to render reasonable results Yi et al. (2023) . On the other hand, leveraging 2D or 3D key points cannot provide enough details for human hands and expression. As a result, previous gesture representations are not suitable off-the-shelf preliminary tools for the co-speech gesture video synthesis task. While researchers also explored directly generating co-speech gesture videos Liu et al. (2022b) ; He et al. (2024) , their visual generator is not sufficient for producing realistic results. Very recently, VLOGGER Corona et al. (2024) leverages a two-stage system and uses 3D representations, but they only produce results with subtle movements. The task of achieving realistic co-speech gesture video synthesis is still under-explored and remains an open problem. Figure 1: Reconstruction with Varied Representations. Holistic 3D reconstruction via SMPL-X often exhibits inaccuracies in limb and joint positioning ( red box). While 2D pose estimation (e.g., DwPose) provides accurate body joint localization, it struggles to represent complex hand gestures effectively ( yellow box). Combination of them leads to precise overly. Figure 2: Illustration of Cosh-DiT System. Given a human speech input and an arbitrary reference image, our framework consistently generates lifelike videos with synchronized gestures. The synthesized videos feature natural, rhythmic movements and expressive facial and hand gestures, capturing vivid details for a realistic portrayal. To address this problem, we propose Cosh-DiT , a system illustrated in Fig. 2 , which synthesizes realistic Co-s peech gesture videos using h ybrid audio-visual Di ffusion T ransformers. Our insight is to identify a suitable intermediate motion representation, on top of which audio-visual Diffusion Transformers are designed for realistic video synthesis . We argue that such representation should precisely deliver human motions and can be easily employed by video generation models. As a result, we first focus more on whether the representation can be well-overlayed in videos as shown in Fig. 1 . Specifically, leveraging parametric hand models Pavlakos et al. (2024) not only conveys the depth difference and occlusion information but also effortlessly constrains finger shape and length. In contrast, the upper body movements can be well depicted through 2D skeletons. Additionally, both the Mano hand and body poses can be precisely and robustly detected by off-the-shelf models Pavlakos et al. (2024) ; Yang et al. (2023b) , facilitating the usage of a large number of raw videos as high-quality training data. To learn the coordination of such mixed representations, we propose a discrete audio-driven Diffusion Transformer, Cosh-DiT-A , for motion synthesis. It takes a novel path that operates directly on encoded token indices. This approach differs from prior latent motion diffusion methods Chen et al. (2023) by offering a more abstract state space and enabling stronger motion priors Gu et al. (2022) . Another advantage is that such diffusion-based strategies iteratively take into account the whole temporal sequence, naturally avoiding issues like uni-directional bias and cumulative prediction errors of step-by-step generation from previous studies Yi et al. (2023) ; Jiang et al. (2023) . However, modeling such hybrid motion representation is non-trivial, especially connecting the 2D upper-body poses and 3D hand parametric models. Thus, we first extend the upper-body poses with the projected key points of hand motions, both of which are encoded into tokens and jointly modeled. Then a Geometric-Aware Alignment Module is carefully designed to optimize the hand position on the fly by projection restriction, seamlessly ensuring the holistic movements of hybrid motion representations. Synthesizing realistic video frames conforming to pose motions requires not only exploiting the appearance information of speaker identity but also spatially aligning with the temporal movements. Although extending the UNet diffusion backbones Blattmann et al. (2023) ; Zhang et al. (2024) ; Corona et al. (2024) with temporal attention has shown its effectiveness on realistic video synthesis, such approaches still struggle on detail synthesis such as hand or facial parts, which severely affects the visual aesthetics. In our work, we take advantage of recent advances in Diffusion Transformers (DiTs) Peebles and Xie (2023) ; Esser et al. (2024) for video generation Gao et al. (2024) ; Yang et al. (2024) . A visual Diffusion Transformer network, Cosh-DiT-V , is devised to jointly model temporal-spatial association, providing higher model capacity and thus achieving improvement in terms of image