Contextual Gesture: Co-Speech Gesture Video Generation through Context-aware Gesture Representation Pinxin Liu, Pengfei Zhang, Hyeongwoo Kim, Pablo Garrido, Ari Shapiro, Kyle Olszewski Abstract . Co-speech gesture generation is crucial for creating lifelike avatars and enhancing human-computer interactions by synchronizing gestures with speech. Despite recent advancements, existing methods struggle with accurately identifying the rhythmic or semantic triggers from audio for generating contextualized gesture patterns and achieving pixel-level realism. To address these challenges, we introduce Contextual Gesture, a framework that improves co-speech gesture video generation through three innovative components: (1) a chronological speech-gesture alignment that temporally connects two modalities, (2) a contextualized gesture tokenization that incorporate speech context into motion pattern representation through distillation, and (3) a structure-aware refinement module that employs edge connection to link gesture keypoints to improve video generation. Our extensive experiments demonstrate that Contextual Gesture not only produces realistic and speech-aligned gesture videos but also supports long-sequence generation and video gesture editing applications, shown in Fig. 1 . Conference: Proceedings of the 33rd ACM International Conference on Multimedia; October 27–31, 2025; Dublin, Ireland Proceedings of the 33rd ACM International Conference on Multimedia (MM ’25), October 27–31, 2025, Dublin, Ireland DOI: 10.1145/3746027.3755140 ISBN: 979-8-4007-2035-2/2025/10 CCS: Computing methodologies Computer vision CCS: Computing methodologies Procedural animation Figure 1 . Contextual Gesture achieves various fine-grained control over video-level gesture motion. Left : We can generate 30s to 1 min speech conditioned gesture videos. Mid : We modify the gestures for intermediate frames of a video by providing a new audio segment. Right : Different people present the same gesture patterns for a given audio. † † cc-license: by 1. Introduction In human communication, speech is often accompanied by gestures that enhance understanding and convey emotions ( De Ruiter et al., 2012 ) . These non-verbal cues play a crucial role in effective interaction ( Burgoon et al., 1990 ) , making gesture generation a key component of natural human-computer interaction. Equipping virtual avatars ( Song et al., 2024a ; Liu et al., 2024a ; Zhang et al., 2025 ) with realistic gesture capabilities is therefore essential for creating engaging interactive experiences for industry production ( Tang et al., 2025 ) . Since the era of deep learning paradigm ( Zhu et al., 2024b ; Lu et al., 2022 ; Gao et al., 2025 ; Lu et al., 2023 ; Lu et al., 2024 ; Ning et al., 2024 ) , many recent works tried to model the relationship between the semantic and emotional content of speech, the associated gestures ( Yi et al., 2023 ; Liu et al., 2023 ; Luo et al., 2022 ; Liu et al., 2022c ; Liu et al., 2022a ; Liu et al., 2025b ) . They simplify the problem by generating coarse 3D motion representations—typically keypoints of joints and body parts—that plausibly align with a given speech sample. These simplified representations can be rendered via standard pipelines and effectively capture basic motion patterns. However, they often disregard the speaker’s visual appearance and fine-grained motions, leading to a lack of realism that limits the expressive power and communicative effectiveness of the generated gestures. Other works, such as ANGIE ( Liu et al., 2022b ) and S2G-diffusion ( He et al., 2024 ) , employ image-warping techniques for video generation, guided by motion representations in the form of keypoints obtained from optical-flow-based deformations. While promising, these approaches face several critical limitations. First, their broad and unconstrained motion representations fail to capture the contextual triggers of gestures—particularly the semantic and emotional cues embedded in speech. This disconnect limits the model’s ability to learn the nuanced relationship between speech and gesture, making it difficult to generate expressive motions that convey the speaker’s intent or metaphoric meaning. Second, the predicted keypoints primarily encode coarse, large-scale transformations, resulting in unstructured motion patterns that are overly sensitive to pronounced movements. Consequently, the generated outputs often suffer from noise and imprecision, especially in fine-grained areas such as the hands and shoulders. These issues degrade the realism and coherence of the generated video content. To address these challenges, we introduce Contextual Gesture , a framework designed to generate speech-aligned gesture motions and afterwards high-fidelity speech video. To uncover the intrinsic temporal connections between gestures and speech, we employ chronological contrastive learning to align these two modalities into time-sensitive joint representation of speech-contextual and gesture motion, which captures the triggers of gesture patterns influenced by speech. We then use knowledge distillation to incorporate speech-contextual features into the tokenization process of gesture motions, aiming to infuse the implicit intentions of gestures conveyed in the speech. This integration creates a clear chronological linkage between the gestures and the speech, enabling the generation of gestures that reflect the speaker’s intentions. For motion generation, we employ a masking-based, Transformer gesture generator that produces motion tokens and refines their alignment with the speech signal through bidirectional mask pretraining. Finally, for uplifting the gesture generation into 2D animations, we propose a structure-aware image refinement module that generates heatmaps of edge connections from keypoints, providing image-level supervision to improve the quality of body regions with large motion. Extensive experiments demonstrate that our method outperforms the existing approaches in both quantitative and qualitative metrics and achieves long sequence generation and editing applications. In summary, our primary contributions are: (1) Contextual Gesture , a framework that achieves time-sensitive joint representation of both speech-contextual and gesture motion, and video generation with various application support; (2) a Contextual-aware gesture representation obtained through knowledge distillation from the chronological gesture-speech aligned features from chronological contrastive learning; (3) a Structural-aware Image refinement module , with edge heatmaps as supervision to improve video fidelity. 2. Related Work Figure 2 . We extract keypoints and leverage chronological alignment with distillation to achieve contextualized gesture motion representation. We leverage a masking-based generator conditioned on audio to generate gesture keypoint sequences and apply image warping with refinement based on edge heatmaps for final gesture video generation. Co-speech Gesture generation Co-speech gesture generation is essential for digital avatar production ( Song et al., 2024b ; Kumar et al., 2023 ) and functions as the prior for realistic rendering ( Luo et al., 2024 ; Song et al., 2024c ; Song et al., 2023 ) . ( Ginosar et al., 2019 ) use an adversarial framework to predict hand and arm poses from audio, and leverage conditional generation ( Chan et al., 2019 ) based on pix2pixHD ( Wang et al., 2018 ) for videos. Some recent works ( Liu et al., 2022c ; Deichler et al., 2023 ; Xu et al., 2023 ; Ao et al., 2022 ; Song et al., 2021a ; Song et al., 2021b ; Liu et al., 2025a ; Song et al., 2024d ) learns the hierarchical semantics or leverages contrastive learning to obtain joint audio-gesture embedding from linguistic theory to assist the gesture pose generation. TalkShow ( Yi et al., 2023 ) estimates SMPL ( Pavlakos et al., 2019 ) poses, and models the body and hand motions for talk-shows. CaMN ( Liu et al., 2022d ) , EMAGE ( Liu et al., 2023 ) and Diffsheg ( Chen et al., 2024 ) use large conversational and speech datasets for joint face and body modeling with diverse style control. ANGIE ( Liu et al., 2022b ) and S2G-Diffusion ( He et al., 2024 ) use image-warping features based on MRAA ( Siarohin et al., 2021 ) or TPS ( Zhao and Zhang, 2022 ) to model body motion and achieve speech driven animation by learning correspondence between audio and image-warping features. However, none of these works produce structure- and speech-aware motion patterns suitable for achieving natural and realistic gesture rendering. Conditional Video Generation Conditional Video Generation has undergo significant progress with diffusion models ( Huang et al., 2025 ; Tang et al., 2024a ; Karras et al., 2023 ; Huang et al., 2024a ; Tang et al., 2024b ) . AnimateDiff ( Guo et al., 2024b ) presents an efficient low-rank adaptation ( Hu et al., 2022 ) (LoRA) to adapt image diffusion model for video motion generation. AnimateAnyone ( Hu et al., 2023 ) construct referencenet for fine-grained control based on skeleton. Make-Your-Anchor ( Huang et al., 2024b ) and Champ ( Zhu et al., 2024a ) improve avatar video generation through face and body based on SMPL-X conditions. EMO ( Tian et al., 2024 ) and EchoMimic ( Meng et al., 2024 ) and Tango ( Liu et al., 2024b ) leverages audio as control signal for talking head and upper body generation. However, these methods are slow in inference speed and ignore the gesture patterns or rhythmic or semantic signals from audio. 3. Contextual Gesture Shown in Fig. 2 , our framework targets at generating co-speech photo-realistic human videos with contextualized gestures. To achieve this goal, we first learn time-sensitive contextual-aware gesture representation through knowledge distillation based on chronological gesture-speech alignment (Sec. 3.1 ). We then leverage a Masking-based Gesture Generator for gesture motion generation. (Sec. 3.2 ) To improve the noisy hand and shoulder movement during the transfer of latent motion to pixel space, we propose a structure-aware image refinement through edge heatmaps for guidance. (Sec. 3.3 ). 3.1. Contextualized Gesture Representation Generating natural and expressive gestures requires capturing fine-grained contextual details that conventional approaches often overlook. Consider a speaker emphasizing the word "really" in the sentence "I really mean it" - while existing methods might generate a generic emphatic gesture, they typically fail to capture the subtle, context-specific body movements that make human gestures truly expressive. This limitation stems from relying solely on motion quantization, which often loses the nuanced relationship between speech and corresponding gestures. To address this challenge, we propose a novel approach that integrates both audio and semantic information into the motion quantizer’s codebook. This integration requires solving two fundamental problems. First, we need to understand how gestures align with speech not just at a high level, but in terms of precise temporal correspondence - when specific words or phrases trigger particular movements, and how the rhythm of speech influences gesture timing. To capture these temporal dynamics, we develop a chronological gesture-speech alignment framework using specialized contrastive learning. Second, we leverage knowledge distillation to incorporate this learned temporal alignment information into the gesture quantizer, enabling our system to generate gestures that are synchronized with speech both semantically and rhythmically. Figure 3 . For gesture generation, we tokenize gesture keypoint sequences using RVQ-VAE (Sec. 3.1 PART II), and use Gesture Generator to generate the gesture tokens based on masked input (Sec 3.2). The gesture tokens are then used to query the RVQ-VAE in order to generate the gestures. For the RVQ-VAE training, we distill the contextual-aware feature from gesture content encoder (Sec 3.1 PART II), which we trained using chronological contrastive learning (Sec 3.1 PART I), in order to distill speech-motion alignment information into the RVQ-VAE codebook. Feature Representation. We utilize 2D poses extracted from images to formulate gestures by facial and body movements. We represent a gesture motion sequence as G=[F;B]=[f_{t};b_{t}]_{t=1}^{T} , where T denotes the length of the motion, f represents the 2D facial landmarks, and b denotes the 2D body landmarks. For speech representation, we extract audio embeddings from WavLM ( Chen et al., 2022 ) and Mel spectrogram features ( Rabiner and Schafer, 2010 ) and beat information using librosa ( McFee et al., 2015 ) . For text-semantics, we extract embedding from RoBERTa ( Liu et al., 2019 ) . These features are concatenated to form the speech representation. Chronological Speech-Gesture Alignment. Traditional approaches to modality alignment ( Ao et al., 2022 ; Liu et al., 2022c ; Deichler et al., 2023 ) rely on global representations through class tokens or max pooling, which overlook the fine-grained temporal dynamics between speech and gestures. We address this limitation by introducing chronological modality alignment. Vanilla Contrastive Alignment. To align gesture motion patterns with the content of speech and beats, we first project both speech and gesture modalities into a shared embedding space to enhance the speech content awareness of gesture features. As illustrated in Fig. 3 Middle, we separately train two gesture content encoders, \mathcal{E}_{f} for face motion and \mathcal{E}_{b} for body motion, alongside two speech encoders, \mathcal{E}_{S_{f}} and \mathcal{E}_{S_{b}} , to map face and body movements and speech signals into this joint embedding space. For simplicity, we represent the general gesture motion sequence as G . We then apply mean pooling to aggregate content-relevant information to optimize the following loss ( van den Oord et al., 2019 ) : (1) \begin{split}\mathcal{L}_{\text{NCE}}&=-\frac{1}{2N}\sum_{i}\left(\log\frac{\exp{S_{ii}/\tau}}{\sum_{j}\exp{S_{ij}/\tau}}+\log\frac{\exp{S_{ii}/\tau}}{\sum_{j}\exp{S_{ji}/\tau}}\right),~\end{split} where S computes the cosine similarities for all pairs in the batch, defined as S_{ij}=\text{cos}(z^{s}_{i},z^{g}_{j}) and \tau is the temperature. Chronological Negative Examples. While vanilla Contrastive Learning builds global semantical alignment, we further propose to address the temporal correspondence between speech and gesture. As shown in fig. 3 Left, consider a speaker saying, "After watching that video, you realize how much, more than any other president…". In this case, the gesture sequence involves "knocking at the table" when saying "more than any other," serving as a visual emphasis for "how much" to highlight the point. To encourage the model understand both semantic and rhythmic alignment between two modalities, we shuffle the words and their corresponding phonemes. By shuffling the sequence to "you realize how much, after watching that video," the semantic intention of the speech is preserved, but the rhythmic correspondence between speech and gesture is disrupted. We use Whisper-X ( Bain et al., 2023 ) to detect temporal segments in the raw sequences. We cut the audio and shuffle these segments, creating these augmented samples as additional chronological negative examples within a batch during contrastive learning. Gesture Quantization with Distillation. To construct context-aware motion representations, we encode alignment information into the gesture motion codebook. This allows the semantics and contextual triggers from speech to be directly fused into the motion embedding, and enables the generator to easily identify the corresponding motion representation in response to speech triggers. To achieve this goal, we leverage gesture content encoder as the teacher and distill knowledge to codebook latent representation, shown in Fig. 3 Middle. We maximize the cosine similarity over time between the RVQ quantization output and the representation from the gesture content encoder: (2) \mathcal{L}_{\text{distill}}=\sum_{t=1}^{T}\cos\left(p(Q_{R})^{t},Es(G)^{t}\right) where p denotes a linear projection layer, Q_{R} is the final quantized output from the RVQ-VAE, Es(G) represents the output from the gesture content encoder, and T is the total time frames. The overall training objective is defined as: (3) \mathcal{L}_{\text{rvq}}=\left\lVert x-\hat{x}\right\rVert^{2}+\alpha\sum_{r=1}^{R}\left\lVert e_{r}-\text{sg}\left(z_{r}-e_{r}\right)\right\rVert^{2}+\beta\mathcal{L}_{\text{distill}} where \mathcal{L}_{\text{rvq}} combines a motion reconstruction loss, a commitment loss ( van den Oord et al., 2018 ) for each layer of quantizer with a distillation loss, with \alpha and \beta weighting the contributions. 3.2. Speech-conditioned Gesture Generation For Motion Generator, we adopt a similar masking-based generation procedure as in MoMask ( Guo et al., 2024a ) . We only include the generator data flows in the main paper but defer the training and inference strategy in the Appendix. Mask Gesture Generator. As shown in Fig. 3 Right, during training, we derive gesture tokens by processing raw gesture sequences through both body and face tokenizers. The gesture token corresponding to the source image acts as the conditioning for all subsequent frames. For speech control, we initialize the audio content encoder from alignment pre-training as described in Sec. 3.1 . This pre-alignment of gesture tokens with audio encoder features enhances the coherence of gesture generation. We employ cross-attention to integrate audio information and apply Adaptive Instance Normalization (AdaIN) ( Huang and Belongie, 2017 ) , enabling gesture styles based on the speaker’s identity. Residual Gesture Generator. The Residual Gesture Generator shares a similar architecture with the Masked Gesture Generator, but it includes R separate embedding layers corresponding to each RVQ residual layer. This module iteratively predicts the residual tokens from the base layers, ultimately producing the final quantized output. Please see MoMask ( Guo et al., 2024a ) for further details. 3.3. Structure-Aware Image Generation Converting generated gesture motions into realistic videos presents significant challenges, particularly in handling camera movement commonly found in videos. While recent 2D skeleton-based animation methods ( Hu et al., 2023 ) offer a potential solution, our empirical analysis (detailed in the Appendix) reveals that these approaches struggle with background stability in the presence of camera motion. To address this limitation, we draw inspiration from optical-flow based image warping methods ( Zhao and Zhang, 2022 ; Siarohin et al., 2021 ; Siarohin et al., 2019 ) , which have shown promise in handling deformable motion. We adapt these approaches by replacing their unsupervised keypoints with our generated gesture keypoints from Sec. 3.2 for more precise foreground optical flow estimation. However, this estimation still presents uncertainties, resulting in blurry artifacts around hands and shoulders, particularly when speakers make rapid or large movements. To address the uncertainties by optical-flow-based deformation during this process, we propose a Structure-Aware Generator. Auto-Link ( He et al., 2023 ) demonstrates that the learning of keypoint connections for image reconstruction aids the model in understanding image semantics. Based on this, we leverage keypoint connections as semantic guidance for image generation. Edge Heatmaps. Using the gesture motion keypoints, we establish linkages between them to provide structural information. To optimize computational efficiency, we limit the number of keypoint connections to those defined by body joint relationships ( Wan et al., 2017 ) , rather than all potential connections in ( He et al., 2023 ) . For two keypoints {\bm{k}}_{i} and {\bm{k}}_{j} within connection groups, we create a differentiable edge map {\bm{S}}_{ij} , modeled as a Gaussian function extending along the line connecting the keypoints. The edge map {\bm{S}}_{ij} for keypoints ({\bm{k}}_{i},{\bm{k}}_{j}) is defined as: (4) {\bm{S}}_{ij}({\bm{p}})=\exp\left(v_{ij}({\bm{p}})d^{2}_{ij}({\bm{p}})/\sigma^{2}\right), where \sigma is a learnable parameter controlling the edge thickness, and d_{ij}({\bm{p}}) is the L_{2} distance between the pixel {\bm{p}} and the edge defined by keypoints {\bm{k}}_{i} and {\bm{k}}_{j} : (5) {\bm{d}}_{ij}({\bm{p}})=\left\{\begin{aligned} &\lVert{\bm{p}}-{\bm{k}}_{i}\rVert_{2}&\text{if}~~t\leq 0,\\ &\lVert{\bm{p}}-((1-t){\bm{k}}_{i}+t{\bm{k}}_{j})\rVert_{2}&\text{if}~~0 10s 2.356 11.956 0.994 BEAT-X \leq 10s 4.747 13.14 7.323 > 10s 4.650 13.55 7.370 (f) Long Seq Generation Quality. Refine VQA A \uparrow VQA T \uparrow FVD \downarrow w/o refine 92.15 5.43 494.35 + UNet 93.86 5.51 478.54 + skeleton 95.79 5.65 473.34 + heatmap 96.53 6.12 466.43 (g) Image-refinement strategies. Figure 6 . Ablations. Left: motion by unsupervised keypoints or 2d poses; Middle: RVQ-based gesture representation and generation; Right: image-refinement helps hand generation. Figure 7 . Our model supports multiple video gesture generation end editing applications. 4.4. Application Long Sequence Generation. As in fig. 7 , to generate long sequences, we begin with the initial frame and corresponding target audio, segmented into smaller windows. After generating the first segment, the last few frames of the output serve as the new starting condition for the next segment, enabling iterative outpainting. Video Gesture Editing. For editing, we extract keypoints from the video, tokenize face and body movements into motion tokens, and insert mask tokens where edits are needed. By changing the speech audio or speaker identity, we can create new gesture patterns and re-render the video. Gesture Pattern Transfer. With different identity embeddings, we generate unique gesture patterns for the same audio input. See the demo videos in the Appendix. Speech-Gesture Retrieval. With chronological speech-gesture alignment, the model is capable of retrieving the best gesture motion correponding to the given speech audio in a batch of data. See additional details in the Appendix. 5. Conclusion We present Contextual Gesture , a framework for generating realistic co-speech gesture videos. To ensure the gestures cohere well with speech, we propose speech-content aware gesture motion representation though knowledge distillation from the gesture-speech aligned features. Our structural-aware image generation module improves the transformation of latent motions into realistic animations for large-scale body motions. We hope this work encourage further exploration of the relationship between gesture patterns and speech context for better video generations in the future. Contextual Gesture: Co-Speech Gesture Video Generation Through Context-Aware Gesture Representation Supplementary Material Appendix A Overview The supplementary material is organized into the following sections: • Section B : Additional Related Works • Section C : Dataset Details and Preprocessing • Section D : Additional Implementation Details • Section E : Gesture Motion Generation • Section F : Gesture Speech Retrieval • Section G : Additional Experiments • Section H : Time and Resource Consumption • Section I : User Study Details • Section J TPS-based Image Warping • Section K Ethical Ethical Considerations • Section L : Limitations For more visualization, please see the additional demo videos. Appendix B Additional Related Works Masked Representation Learning for Generation Masked Representation Learning has been demonstated an effective representation learning for various modalities. ( Devlin, 2018 ; He et al., 2022 ) Some works explored the generation capabilities using this paradigm. MAGE ( Li et al., 2023 ) achieves high-quality image generation through iterative remasking. Muse ( Chang et al., 2023 ) extends this idea to leverage language with region masking for image editing and achieve fine-grained control. Recent Masking Models ( Pinyoanuntapong et al., 2024 ; Guo et al., 2024a ; Wang, 2023 ; Mao et al., 2024 ; Zhang et al., 2025 ) bring this strategy to the motion and gesture domain and improves the motion generation speed, quality, and editing capability. Inspired by these work, we propose the masked gesture generation conditioned the audio to learn the gesture-speech correspondence during generation. Appendix C Dataset Details and Preprocessing C.1. Preprocessing We found that many videos used in ANGIE ( Liu et al., 2022b ) and S2G-Diffusion ( He et al., 2024 ) , particularly for the subject Jon , are no longer available. To address this, we replaced Jon with Noah . We utilized the PATS ( Ginosar et al., 2019 ) metadata to download videos from YouTube and preprocess them. After filtering, we obtained 1080 videos for Oliver , 1080 for Kubinec , 1080 for Seth , and 988 for Noah . For the testing dataset, we collected 120 videos for Oliver , 120 for Kubinec , 120 for Seth , and 94 for Noah . During the dataset preprocessing, while for image-generation we use the whole video preprocessed as above, for for the speech-gesture alignment and gesture pattern generation modules, we further preprocess the data by slicing them into smaller chunks following S2G-Diffusion ( He et al., 2024 ) . Specifically, based on the source training dataset, the keypoint sequences and audio sequences are clipped to 80 frames (3.2s) with stride 10 (0.4s) for training. We obtain 85971 overlapping training examples and 8867 testing examples for gesture pattern modeling. C.2. Feature Representation Gesture Keypoints. We utilize RTMPose ( Jiang et al., 2023 ) from MMPose ( OpenMMLab, 2020 ) for whole-human-body keypoint identification. The keypoint definition is based on by 133 CoCo human pose estimation. Due to the PATS ( Ginosar et al., 2019 ) only contains the upper body, we select 68 face landmarks for face motion modeling, 3 for left shoulder, 3 for right shoulder, 21 for left hand and 21 for right hand separately, which results in flattened face feature with dim of 136 and body feature with dim of 96. Audio Features. The audio features are pre-extracted WavLM features (dim of 1024) with additional low-level mel-spectrum and beat information with dimension of 34. We concatenate them channel-wise as the speech feature. C.3. Dataset License. The video data within PATS dataset include personal identity information, and we strictly adhere to the data usage license “CC BY - NC - ND 4.0 International,” which permits non-commercial use. Appendix D Additional Implementation Details We jointly train the framework on three speakers. The following sections provide the details for each module’s training. Optimizer Settings. All modules utilize the Adam Optimizer ( Kingma, 2014 ) during training, with a learning rate of 1\times 10^{-4} , \beta_{1}=0.5 , and \beta_{2}=0.999 . Speech-Gesture Alignment. For aligning speech with facial and bodily gestures, we implement two standard transformer blocks for encoding each modality. The latent dimension is configured to 384, accompanied by a feedforward size of 1024. We calculate the mean features for both modalities and project them using a two-layer MLP in a contrastive learning framework, with a temperature parameter set to 0.7. Residual Vector Quantization (RVQ) Tokenization. The overall training objective for the RVQ-VAE is defined as: (9) \mathcal{L}_{\text{rvq}}=\mathbb{E}_{x\sim p(x)}\left[\left\lVert x-\hat{x}\right\rVert^{2}\right]+\alpha\sum_{r=1}^{R}\mathbb{E}_{z_{r}\sim q(z_{r}|x)}\left[\left\lVert e_{r}-\text{sg}\left(z_{r}-e_{r}\right)\right\lVert^{2}\right]+\beta\mathcal{L}_{\text{distill}} where \mathcal{L}_{\text{rvq}} combines a motion reconstruction loss, a commitment loss ( van den Oord et al., 2018 ) for each layer of quantizer with a distillation loss, with \alpha and \beta weighting the contributions. We employ six layers of codebooks for residual vector quantization ( Lee et al., 2022 ) for both face and body modalities, each comprising 1024 codes. To address potential collapse issues during training, we implement codebook resets. The RVQ encoder and decoder are built with two layers of convolutional blocks and a latent dimension of 128. We avoid temporal down-sampling to ensure the latent features maintain the same temporal length as the original input sequences. During RVQ training, we set \alpha=1 and \beta=0.5 to balance gesture reconstruction with speech-context distillation. Mask Gesture Generator. The generator takes sequences of discrete tokens for both face and body, derived from the RVQ codebook. This module includes two layers of audio encoders for face and body, initialized based on the Speech-Gesture Alignment. The latent dimension is again set to 384, with a feedforward dimension of 1024, and it features eight layers for both modalities. A two-layer MLP is utilized to project the latent space to the codebook dimension, and cross-entropy is employed for model training. We calculate reconstruction and acceleration loss by feeding the predicted tokens into the RVQ decoder. A reconstruction loss of 50 is maintained during training, and the mask ratio is uniformly varied between 0.5 and 1.0. For inference, a cosine schedule is adopted for decoding. We uniformly mask between 50% and 100% of the tokens during training. Following the BERT ( Devlin, 2018 ) , when a token is selected for masking, we replace it with a [MASK] token 80% of the time, a random token 10% of the time, and leave it unchanged 10% of the time. The Mask Gesture Generator is trained over 250 epochs, taking approximately 1 days to complete. Residual Gesture Generator. The Residual Gesture Generator is designed similarly to the Mask Gesture Generator but utilizes only six layers for the generator. It features four embedding and classification layers corresponding to the RVQ tokenization scheme for residual layers. During training, we randomly select a quantizer layer j\in[1,R] for learning. All tokens from the preceding layers t^{0:j-1} are embedded and summed to form the token embedding input. After generating the base layer predictions of discrete tokens from the Masked Gesture Generator, these tokens are fed into the Residual Gesture Generator. This module iteratively predicts the tokens from the base layers, ultimately producing the final quantized output. This module is trained for an additional 500 epochs, requiring about 0.5 days to finalize. Image Warping. For pixel-level motion generation, we utilize Thin Plate Splines (TPS) ( Zhao and Zhang, 2022 ) . Our framework tracks 116 keypoints (68 for the face and 48 for the body). The number of TPS transformations K is set to 29, with each transformation utilizing N=4 paired keypoints. In accordance with TPS methodologies, both the dense motion network and occlusion-aware generators leverage 2D convolutions to produce 64\times 64 weight maps for optical flow generation, along with four occlusion masks at various resolutions (32, 64, 128, and 256) to facilitate image frame synthesis. Structure-aware Image-Refinement. We use the UNet similar to S2G-Diffusion ( He et al., 2024 ) to restore missing details, further improve the hand and shoulder areas. We keep the training loss to be the same except the added conditional adversarial loss based on edge heatmap. For the network design difference, we add the multi-level edge heatmap as additional control for different resolutions (32, 64 and 128). Each correponds to a SPADE ( Park et al., 2019 ) block to inject the semantic control into the current generation. Appendix E Gesture Motion Generation Inference. While existing works ( Liu et al., 2023 ; Yi et al., 2023 ; Chen et al., 2024 ) leverage auto-regressive next-token prediction or diffusion-based generation process, these strategies hinder the fast synthesis for real-time applications. To resolve this problem, as in Fig. 3 , we employ an iterative mask prediction strategy to decode motion tokens during inference. Initially, all tokens are masked except for the first token from the source frame. Conditioned on the audio input, the Mask Gesture Generator predicts probabilities for the masked tokens. In the l -th iteration, the tokens with the lowest confidence are re-masked, while the remaining tokens stay unchanged for subsequent iterations. This updated sequence continues to inform predictions until the final iteration, when the base-layer tokens are fully generated. Upon completion, the Residual Gesture Generator uses the predicted base-layer tokens to progressively generate sequences for the remaining quantization layers. Finally, all tokens are transformed back into motion sequences via the RVQ-VAE decoder. Training Objective. To train our gesture generation models, \mathcal{L}_{mask} , and \mathcal{L}_{res} functions for two generaors respectively by minimizing the categorical cross-entropy loss, as illustrated below: (10) \mathcal{L}_{mask}=\sum_{i=1}^{T}-\log p_{\phi}(t_{i}|Es(S),\mathrm{MASK}),\quad\mathcal{L}_{res}=\sum_{j=1}^{V}\sum_{i=1}^{T}-\log p_{\phi}(t_{i}^{j}|t_{i}^{1:j-1},Es(S),j). In this formulation, \mathcal{L}_{mask} predicts the masked motion tokens t_{i} at each time step i based on the input audio and the special [\mathrm{MASK}] token. Conversely, \mathcal{L}_{res} focuses on learning from multiple quantization layers, where t_{i}^{j} represents the motion token from quantizer layer j and t_{i}^{1:j-1} includes the tokens from preceding layers. We also feed the predicted tokens into the RVQ decoder for gesture reconstructions, with velocity and acceleration losses ( Tevet et al., 2022 ; Siyao et al., 2022 ) . Appendix F Speech-Gesture Retrieval To validate the effectiveness of Speech-Gesture Alignment, inspired by TMR ( Petrovich et al., 2023 ) we propose the Speech-Gesture Retrieval as the evaluation benchmark. Evaluation Settings. The retrieval performance is measures under recall at various ranks, R@1, R@2, etc.Ṙecall at rank k indicates the percentage of times the correct label is among the top k results; therefore higher is better. We define two settings. The retrieval is based on the sliced clips, with each lasting for 4 seconds and 120 frames, in total 8176 samples. (a) All test set samples for face and body motions. This set is problematic because the speech and gesture motion should not be of one-to-one mapping relationship. (b) Small batch size of 32 speech-gesture pairs are randomly picked, reporting average performance. Evaluation Result. Shown in table 4 , the gesture patterns and speech context do not present one-to-one mapping relationship, leading to the significantly low performance of retrieval. However, based on setting (b), within a small batch size of 32, the model achieves significantly higher performance, indicating the alignment provides the discrimination over different speech context and the motion. In addition, chronological negative examples during contrastive training enhances the robustness of retrieval. Table 4 . Speech-to-Gesture Motion retrieval benchmark on PATS: We establish two evaluation settings as described in Section F . Setting Speech-Face retrieval Face-Speech retrieval R@1 \uparrow R@2 \uparrow R@3 \uparrow R@5 \uparrow R@10 \uparrow R@1 \uparrow R@2 \uparrow R@3 \uparrow R@5 \uparrow R@10 \uparrow (a) All 0.181 0.350 0.485 0.722 1.343 0.226 0.361 0.429 0.677 1.207 (a) + chrono 0.231 0.372 0.501 0.734 1.696 0.323 0.398 0.454 0.712 1.332 (b) Small batches 26.230 45.318 59.330 77.019 89.858 24.977 44.822 59.894 77.775 90.264 (b) + chrono 27.437 47.552 63.193 74.343 89.996 26.451 46.432 61.727 79.779 91.373 Setting Speech-Body retrieval Body-Speech retrieval R@1 \uparrow R@2 \uparrow R@3 \uparrow R@5 \uparrow R@10 \uparrow R@1 \uparrow R@2 \uparrow R@3 \uparrow R@5 \uparrow R@10 \uparrow (a) All 0.102 0.237 0.327 0.587 1.230 0.158 0.271 0.406 0.654 1.320 (a) + chrono 0.132 0.257 0.373 0.603 1.340 0.178 0.289 0.443 0.671 1.404 (b) Small batches 25.542 43.660 57.954 77.471 90.309 24.052 43.874 58.495 76.986 89.745 (b) + chrono 28.732 48.569 59.958 79.321 90.003 22.671 45.737 57.669 79.565 90.672 Appendix G Additional Experiments In the main paper, we have shown our method achieves promising joint gesture motion and video generation. To understand the disentangled gesture and video avatar generation separately, we further conduct Video Avatar Animation experiments separately to compare our method with the corresponding representative works. Figure 8 . Comparison of Video Avatar Animation Though presented with worse hand structure reconstruction, we achieve better identity preserving and significantly better background motion. G.1. Video Avatar Animation Experiment Settings. We select PATS dataset as in main paper for avatar rendering comparison. We processed the videos into 512x512 for Diffusion-bassed model AnimateAnyone ( Hu et al., 2023 ) . We extract the 2D poses by MMPOse ( OpenMMLab, 2020 ) for pose guidance for the Diffusion Model, and maintain all the training details as in AnimateAnyone for consistency. Experiment Results. We compare the gesture generation module of our work with representative AnimateAnyone ( Hu et al., 2023 ) . As shown in fig. 8 , though AnimateAnyone achieves better video generation quality for hand structure of the speaker centering in the video, it fails to maintain the speaker identity, making the avatar less similar to the source image compared with our method. In addition, due to the entanglement of camera motions and speaker gesture motions within the dataset, AnimateAnyone fails to separate two types of motions from the source training video, thus leading to significant background changes over time and dynamic inconsistency. Unlike completely relying on human skeletons as conditions in AnimateAnyone, our method benefits from Warping-based method, which has the capability of resolving the background motions in addition to the speaker motion. We defer visual comparisons in the Appendix videos. Appendix H Time and Resource Consumption In table 5 , we present a comparison of training and inference times against existing baseline methods. For audio-gesture generation, our model’s training time is comparable, albeit slightly slower, than that of ANGIE ( Liu et al., 2022b ) and S2G-Diffusion ( He et al., 2024 ) , primarily due to the inclusion of additional modules. However, it is considerably faster than MM-Diffusion ( Ruan et al., 2023 ) . Notably, our method excels in inference speed, outperforming all other baselines. While the training of image-warping and image refinement requires a lot of time, our method leads to a substantial reduction in overall time and resource usage compared to MM-Diffusion and other stable-diffusion-based video generation approaches. Furthermore, the generative masking paradigm we employ significantly cuts down inference times when compared to diffusion-based models like S2G-Diffusion or the autoregressive generations in ANGIE. We further compared image-warping based method computation requirements with Stable Diffusion-based models like AnimateAnyone ( Hu et al., 2023 ) in table 6 . Table 5 . Time consumption comparison of training (1 NVIDIA A100 GPU) and inference (1 NVIDIA GeForce RTX A6000 GPU). Name Training Training Breakdown Inference (video of \sim 10 sec) ANGIE \sim 5d Motion Repr. \sim 3d + Quantize \sim 0.2d + Gesture GPT \sim 1.8d \sim 30 sec MM-Diffusion \sim 14d Generation \sim 9d + Super-Resolution \sim 5d \sim 600 sec S2G-Diffusion \sim 5d Motion Decouple \sim 3d + Motion Diffusion \sim 1.5d + Refine \sim 0.5d \sim 35 sec Ours \sim 6d Quantize \sim 0.2d + Mask-Gen \sim 1.5d + Res-Gen \sim 0.5d + Img-warp & Refine \sim 3.5d \sim 3 sec Table 6 . Resource consumption comparison with Stable-Diffusion-based Image-Animation models (1 NVIDIA A100 GPU), * means our re-implementation on PATS dataset. Methods Training \downarrow Batch Size Resolution Memory \downarrow Training Task Inference \uparrow AnimateAnyone* 10 days 4 512 44 GB Pose-2-Img - AnimateAnyone* 5 days 4 512 36GB Img-2-Vid 15s Ours 2.5 days 64 256 64 GB Img-Warp \leq 1s Ours 1 day 64 256 48GB Img-Refine \leq 1s Ours 3.5 days 32 512 60GB Img-Warp \leq 1s Ours 1 day 32 512 40GB Img-Refine \leq 1s Figure 9 . Screenshot of user study website. Appendix I User Study Details For user study, we recruited 20 participants with good English proficiency. To conduct the user study, we randomly select 80 videos from EchoMimicV2 ( Meng et al., 2024 ) , ANGIE ( Liu et al., 2022b ) , S2G-Diffusion ( He et al., 2024 ) and ours. Each user works on 20 videos, with 4 videos from each of the aforementioned methods. The users are not informed of the source of the video for fair evaluations. A visualization of the user study is shown in fig. 9 . Appendix J TPS-based Image-Warping In this paper, we utilize Thin Plate Splines (TPS) ( Zhao and Zhang, 2022 ) to model deformations based on human poses for image-warping. Here, we provide additional details on this approach. The TPS transformation accepts N pairs of corresponding keypoints (p^{\mathbf{D}}_{i},p^{\mathbf{S}}_{i}) for i=1,2,\ldots,N (referred to as control points) from a driving image \mathbf{D} and a source image \mathbf{S} . It outputs a pixel coordinate mapping \mathcal{T}_{tps}\left(\cdot\right) , which represents the backward optical flow from \mathbf{D} to \mathbf{S} . This transformation is founded on the principle that 2D warping can be effectively modeled through a thin plate deformation mechanism. The TPS transformation seeks to minimize the energy associated with bending this thin plate while ensuring that the deformation aligns accurately with the control points. The mathematical formulation is as follows: (11) \begin{split}\min&\iint_{\mathbb{R}^{2}}\left(\left(\frac{\partial^{2}\mathcal{T}_{tps}}{\partial x^{2}}\right)^{2}+2\left(\frac{\partial^{2}\mathcal{T}_{tps}}{\partial x\partial y}\right)^{2}+\left(\frac{\partial^{2}\mathcal{T}_{tps}}{\partial y^{2}}\right)^{2}\right)\,dxdy,\\ &\text{s.t.}\quad\mathcal{T}_{tps}(p_{i}^{\mathbf{D}})=p_{i}^{\mathbf{S}},\quad i=1,2,\ldots,N,\end{split} where p_{i}^{\mathbf{D}} and p_{i}^{\mathbf{S}} denote the i^{th} keypoints in \mathbf{D} and \mathbf{S} respectively. As shown in ( Zhao and Zhang, 2022 ) , it can be demonstrated that the TPS interpolating function satisfies eq. 11 : (12) \mathcal{T}_{tps}(p)=A\left[\begin{array}[]{l}p\\ 1\end{array}\right]+\sum_{i=1}^{N}w_{i}U\left(\left\lVert p^{\mathbf{D}}_{i}-p\right\rVert_{2}\right), where p=(x,y)^{\top} represents the coordinates in \mathbf{D} , and p^{\mathbf{D}}_{i} is the i^{th} keypoint in \mathbf{D} . The function U(r)=r^{2}\log r^{2} serves as a radial basis function. Notably, U(r) is the fundamental solution to the biharmonic equation ( Selvadurai and Selvadurai, 2000 ) , defined by: (13) \Delta^{2}U=\left(\frac{\partial^{2}}{\partial x^{2}}+\frac{\partial^{2}}{\partial y^{2}}\right)^{2}U\propto\delta_{(0,0)}, where the generalized function \delta_{(0,0)} is characterized as: (14) \displaystyle\delta_{(0,0)}=\begin{cases}\infty,&\text{if }(x,y)=(0,0)\\ 0,&\text{otherwise}\end{cases},\quad\text{and}\iint_{\mathbb{R}^{2}}\delta_{(0,0)}(x,y)\,dxdy=1, indicating that \delta_{(0,0)} is zero everywhere except at the origin, where it integrates to one. We denote the i^{th} keypoint in image \mathbf{X} (either \mathbf{D} or \mathbf{S} ) as p^{\mathbf{X}}_{i}=(x^{\mathbf{X}}_{i},y^{\mathbf{X}}_{i})^{\top} , and we define: (15) r_{ij}=\left\lVert p^{\mathbf{D}}_{i}-p^{\mathbf{D}}_{j}\right.\rVert,\quad i,j=1,2,\ldots,N. Next, we construct the following matrices: (16) \displaystyle K= \displaystyle{\displaystyle\left[\begin{array}[]{cccc}0&U\left(r_{12}\right)&\cdots&U\left(r_{1N}\right)\\ U\left(r_{21}\right)&0&\cdots&U\left(r_{2N}\right)\\ \vdots&\vdots&\ddots&\vdots\\ U\left(r_{N1}\right)&U\left(r_{N2}\right)&\cdots&0\end{array}\right]},\quad P=\left[\begin{array}[]{ccc}1&x^{\mathbf{D}}_{1}&y^{\mathbf{D}}_{1}\\ 1&x^{\mathbf{D}}_{2}&y^{\mathbf{D}}_{2}\\ \vdots&\vdots&\vdots\\ 1&x^{\mathbf{D}}_{N}&y^{\mathbf{D}}_{N}\end{array}\right], (17) \displaystyle L \displaystyle=\left[\begin{array}[]{cc}K&P\\ P^{T}&0\end{array}\right],\quad Y=\left[\begin{array}[]{ccccccc}x^{\mathbf{S}}_{1}&x^{\mathbf{S}}_{2}&\cdots&x^{\mathbf{S}}_{N}&0&0&0\\ y^{\mathbf{S}}_{1}&y^{\mathbf{S}}_{2}&\cdots&y^{\mathbf{S}}_{N}&0&0&0\\ \end{array}\right]^{\top}. We can then determine the affine parameters A\in\mathcal{R}^{2\times 3} and the TPS weights w_{i}\in\mathcal{R}^{2\times 1} by solving the following equation: (18) \displaystyle\left[w_{1},w_{2},\cdots,w_{N},A\right]^{\top}=L^{-1}Y. In eq. 12 , the first term A\left[\begin{array}[]{l}p\\ 1\end{array}\right] represents an affine transformation that aligns the paired control points (p^{\mathbf{D}}_{i},p^{\mathbf{S}}_{i}) in linear space. The second term \sum_{i=1}^{N}w_{i}U\left(\left\lVert p^{\mathbf{D}}_{i}-p\right\rVert_{2}\right) accounts for nonlinear distortions that enable the thin plate to be elevated or depressed. By combining both linear and nonlinear transformations, the TPS framework facilitates precise deformations, which are essential for accurately capturing motion while preserving critical appearance details within our framework. Appendix K Ethical Considerations While this work is centered on generating co-speech gesture videos, it also raises important ethical concerns due to its potential for photo-realistic rendering. This capability could be misused to fabricate videos of public figures making statements or attending events that never took place. Such risks are part of a broader issue within the realm of AI-generated photo-realistic humans, where phenomena like deepfakes and animated representations pose significant ethical challenges. Although it is difficult to eliminate the potential for misuse entirely, our research offers a valuable technical analysis of gesture video synthesis. This contribution is intended to enhance understanding of the technology’s capabilities and limitations, particularly concerning details such as facial nuances and temporal coherence. In addition, we emphasize the importance of responsible use. We recommend implementing practices such as watermarking generated videos and utilizing synthetic avatar detection tools for photo-realistic images. These measures are vital in mitigating the risks associated with the misuse of this technology and ensuring ethical standards are upheld. Appendix L Limitations While our method have achieved significant improvements over existing baselines, there are still two limitations of the current work. First, the generation quality still exhibit blurries and flickering issues. The intricate structure of hand hinders the generator in understanding the complex motions. In addition, PATS dataset is sourced from in-the-wild videos of low quality. Most frames extracted from videos demonstrate blurry hands, limiting the network learning. Thus, it is important to collect the high-quality gesture video dataset with clearer hands to further enhance the generation quality. Second, when modeling the whole upper-body, it is hard to achieve synchronized lip movements aligned with the audio. Even though we explicit separate the face motion and body motion to deal with this problem, there is no regularization on lip movement. We would like to defer this problem to the future works that models disentangled and fine-grained motions for each face and body region.