M A G : M ulti-Modal Aligned A utoregressive Co-Speech G esture Generation without Vector Quantization
Binjie Liu*, Lina Liu*, Sanyi Zhang, Songen Gu, Yihao Zhi, Tianyi Zhu, Lei Yang, Long Ye · 2025
Read M A G : M ulti-Modal Aligned A utoregressive Co-Speech G esture Generation without Vector Quantization by Binjie Liu*, Lina Liu*, Sanyi Zhang, Songen Gu, Yihao Zhi, Tianyi Zhu, Lei Yang, Long Ye (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2503-14040.txt.
Introduction
Binjie Liu and co-authors generate gestures that match speech, without chopping motion into discrete tokens, in a 2025 paper. Token based gesture models lose detail and look less real. Their system first learns a smooth motion code aligned with sound and text, using embeddings from a pretrained audio text model. A masked generator then builds motion step by step in that smooth space, steered by a block that fuses voice and words.
On two benchmark sets, the method topped rivals in both scores and human judgment, the authors report, yielding varied, lifelike full body gestures. They plan to release the code.
Catalog record
| CREATOR | Binjie Liu*, Lina Liu*, Sanyi Zhang, Songen Gu, Yihao Zhi, Tianyi Zhu, Lei Yang, Long Ye |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 35 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2503-14040
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
M A G : M ulti-Modal Aligned A utoregressive Co-Speech G esture Generation without Vector Quantization Binjie Liu*, Lina Liu*, Sanyi Zhang, Songen Gu, Yihao Zhi, Tianyi Zhu, Lei Yang, Long Ye Abstract This work focuses on full-body co-speech gesture generation. Existing methods typically employ an autoregressive model accompanied by vector-quantized tokens for gesture generation, which results in information loss and compromises the realism of the generated gestures. To address this, inspired by the natural continuity of real-world human motion, we propose MAG , a novel multi-modal aligned framework for high-quality and diverse co-speech gesture synthesis without relying on discrete tokenization. Specifically, (1) we introduce a motion-text-audio-aligned variational autoencoder ( MTA-VAE ), which leverages pre-trained WavCaps’ text and audio embeddings to enhance both semantic and rhythmic alignment with motion, ultimately producing more realistic gestures. (2) Building on this, we propose a multi-modal masked autoregressive model ( MMAG ) that enables autoregressive modeling in continuous motion embeddings through diffusion without vector quantization. To further ensure multi-modal consistency, MMAG incorporates a hybrid granularity audio-text fusion block, which serves as conditioning for diffusion process. Extensive experiments on two benchmark datasets demonstrate that MAG achieves state-of-the-art performance both quantitatively and qualitatively, producing highly realistic and diverse co-speech gestures. The code will be released to facilitate future research. Figure 1 : On the left , inspired by the natural continuity of human motion, we observe that VQ-VAE discretizes motion data, while VAE preserves a continuous latent space that better reflects real motion. On the right , motivated by this, we propose MAG, a framework that enables autoregressive modeling in continuous motion embeddings through diffusion, eliminating vector quantization. Given speech audio and text transcripts as conditionings, our model generates motion embeddings, which are decoded into realistic gestures via a Motion VAE decoder. † † footnotetext: * Equal contribution. † † footnotetext: \dagger Corresponding author. 1 Introduction Full-body co-speech gesture generation aims to create realistic gestures that seamlessly complement verbal communication, which can serve as vital tools for conveying information in human communication [ 50 , 19 , 7 , 23 , 26 ] . Automated generation of full-body co-speech gestures is a crucial technology for developing lifelike avatars in diverse applications, such as virtual characters in the metaverse, human-computer interaction (HCI), and gaming to robot assistants [ 61 , 43 ] . However, there still exists two key challenges: 1) how to generate realistic and diverse gestures that seamlessly complement verbal communication; 2) how to generate semantically and rhythmically aligned gestures. Researchers have conducted substantial solutions in the field of full-body co-speech gesture generation. Traditional rule-based methods [ 8 , 9 , 29 , 25 , 30 , 46 , 37 ] can integrate linguistic rules well, but the generated gestures lack flexibility and adaptability. In contrast, deep learning based approaches own favourable ability to handle complex multi-modal inputs. A wide range of strategies have been effectively explored for this task, including: normalizing flows [ 55 , 27 ] , autoregressive models based on VQ-VAE [ 2 , 39 , 57 , 33 , 52 , 47 , 10 , 40 , 11 ] , GANs [ 20 , 45 , 17 , 16 ] , and diffusion models [ 1 , 3 , 54 , 64 , 12 ] . This paper follows the learning-based solution upon diffusion model. Many existing learning-based approaches rely on autoregressive models with vector quantization (VQ) [ 2 , 39 , 57 , 33 , 52 , 47 , 11 ] to tokenize motion data. However, as illustrated on the left side of Fig. 1 , real-world human motion usually exists in a continuous space. Discretizing it into categorical labels disrupts numerical magnitude relationships, leading to information loss and reducing the model’s ability to capture motion complexity. In addition, multi-modal information has not been effectively utilized to generate semantically and rhythmically aligned gestures. Methods such as MotionCLIP [ 48 ] , GestureDiffCLIP [ 3 ] , and LivelySpeaker [ 63 ] primarily focus on contrastive learning between text and motion, effectively capturing semantic content but neglecting the role of audio in rhythmic coordination. The joint alignment of motion with both text and audio embeddings remains an open challenge for generating more natural and synchronized gestures. To tackle the above limitations, motivated by the continuity of real-world human motion, we propose MAG, a multi-modal aligned autoregressive co-speech gesture generation framework, as illustrated on the right side of Fig. 1 . To maintain the high consistency between the multi-modal input and generated gestures, we simultaneously aggregate the motion, text, and audio into an unified learning module to enhance semantic and rhythmic coherence and generate more realistic gestures. Specifically, taking the motion, text and audio as input, we introduce a motion-text-audio-aligned variational autoencoder (MTA-VAE) supervised through contrastive learning, which leverages pre-trained WavCaps [ 38 ] text and audio embeddings to enhance both semantic and rhythmic alignment with motion. To meet the reliability distribution of real-world continuous motion space, we introduce to model with continuous motion embeddings by utilizing diffusion processes to avoid information loss in discretized representation. Specifically, we propose a multi-modal masked autoregressive model (MMAG) that facilitates autoregressive modeling without VQ through diffusion. Furthermore, a hybrid audio-text fusion block is incorporated by MMAG and served as conditioning for diffusion process. For final gesture generation, the continuous motion representations are predicted by diffusion network, and decoded by motion VAE decoder to generate more realistic gestures. Our contributions are summarized as follows: • We propose a novel MTA-VAE , the first to align motion, text and audio through contrastive learning to construct a unified multi-modal latent space, enabling accurate semantically aware and rhythmically aligned gesture generation. • We introduce MMAG , a multi-modal masked autoregressive gesture generation framework without VQ that enables autoregressive modeling in continuous motion embeddings through diffusion, avoiding information loss in discretized representation. • Experiments on two benchmark datasets confirm that MAG achieves state-of-the-art performance. User studies further validate its effectiveness in generating realistic, synchronized, and diverse co-speech gestures. 2 Related work Realistic Co-speech Gesture Generation. As outlined above, co-speech gesture generation aims to produce realistic and diverse gestures that align with speech. Early rule-based methods [ 8 , 9 , 29 , 25 , 30 , 46 , 37 ] mapped linguistic rules to predefined gesture sequences but lacked flexibility. Deep learning has since become the dominant approach, with early neural models such as RNNs [ 5 ] and CNNs [ 42 ] improving gesture synthesis by capturing temporal dependencies. More recently, masked autoregressive Transformers have been widely adopted for speech-aligned gesture generation. EMAGE [ 33 ] employs VQ-VAE to discretize body motion, enabling autoregressive Transformers to learn motion representations. However, these methods [ 2 , 39 , 57 , 33 , 52 , 47 , 10 , 40 , 11 ] rely on discrete tokenization techniques like VQ-VAE, which introduce quantization errors, ultimately degrading motion quality and diversity. To mitigate quantization errors, recent research has refined quantization techniques. ProbTalk [ 36 ] integrates PQ-VAE [ 51 ] for holistic motion modeling, improving gesture representation through a two-stage refinement. Similarly, SynTalker [ 40 ] replaces VQ-VAE with RVQ-VAE to enhance precision and diversity. However, discretizing continuous motion data remains inherently