MoSa: Mo \mathrm{Mo} tion Generation with S \mathrm{S} calable A \mathrm{A} utoregressive Modeling PubID: pubid: 0000–0000/00$00.00 © 2025 IEEE
Mengyuan Liu, Sheng Yan, Yong Wang, Yingjie Li, Gui-Bin Bian, Hong Liu · 2025
Read MoSa: Mo \mathrm{Mo} tion Generation with S \mathrm{S} calable A \mathrm{A} utoregressive Modeling PubID: pubid: 0000–0000/00$00.00 © 2025 IEEE by Mengyuan Liu, Sheng Yan, Yong Wang, Yingjie Li, Gui-Bin Bian, Hong Liu (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2511-01200.txt.
Introduction
Mengyuan Liu and colleagues introduce MoSa, a system that generates three dimensional human motion from text descriptions. It builds on vector quantization guided generative transformers, where motion is compressed into discrete tokens and a transformer predicts them. Earlier systems predict one token at a time, which is slow.
MoSa generates coarse to fine instead. A hierarchical quantization scheme with a multi scale token preservation strategy keeps tokens at several scales. A scalable autoregressive model predicts whole scale levels per step, needing only as many steps as there are quantization layers. A lightweight convolution attention autoencoder guards reconstruction quality against the frequent interpolation used. On the Motion-X data set, MoSa beat a leading prior method on fidelity while cutting inference time by about a quarter.
Catalog record
| CREATOR | Mengyuan Liu, Sheng Yan, Yong Wang, Yingjie Li, Gui-Bin Bian, Hong Liu |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 43 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2511-01200
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
MoSa: Mo \mathrm{Mo} tion Generation with S \mathrm{S} calable A \mathrm{A} utoregressive Modeling PubID: pubid: 0000–0000/00$00.00 © 2025 IEEE Mengyuan Liu, Sheng Yan, Yong Wang, Yingjie Li, Gui-Bin Bian, Hong Liu Abstract We introduce MoSa, a novel hierarchical motion generation framework for text-driven 3D human motion generation that enhances the Vector Quantization-guided Generative Transformers (VQ-GT) paradigm through a coarse-to-fine scalable generation process. In MoSa, we propose a Multi-scale Token Preservation Strategy (MTPS) integrated into a hierarchical residual vector quantization variational autoencoder (RQ-VAE). MTPS employs interpolation at each hierarchical quantization to effectively retain coarse-to-fine multi-scale tokens. With this, the generative transformer supports Scalable Autoregressive (SAR) modeling, which predicts scale tokens, unlike traditional methods that predict only one token at each step. Consequently, MoSa requires only 10 inference steps, matching the number of RQ-VAE quantization layers. To address potential reconstruction degradation from frequent interpolation, we propose CAQ-VAE, a lightweight yet expressive convolution-attention hybrid VQ-VAE. CAQ-VAE enhances residual block design and incorporates attention mechanisms to better capture global dependencies. Extensive experiments show that MoSa achieves state-of-the-art generation quality and efficiency, outperforming prior methods in both fidelity and speed. On the Motion-X dataset, MoSa achieves an FID of 0.06 (versus MoMask’s 0.20) while reducing inference time by 27%. Moreover, MoSa generalizes well to downstream tasks such as motion editing, requiring no additional fine-tuning. The code is available at https://mosa-web.github.io/MoSa-web I Introduction Text-driven 3D human motion generation is a novel and significant branch of human analysis [ 1 , 2 , 3 , 4 , 5 , 6 , 7 , 8 ] , which boasts a wide range of commercial applications. Our showcased program 1 1 1 https://huggingface.co/spaces/MoSa-web/MoSa shows an intuitive example: game designers can perform character modeling without relying on complex motion capture equipment. This approach greatly reduces labour and resource costs. Consequently, motion generation has attracted considerable research interest [ 9 , 10 , 11 , 12 , 13 , 14 , 15 , 16 , 17 , 18 , 19 , 20 ] . Earlier works like TEMOS [ 17 ] and MotionCLIP [ 16 ] aimed to fit the distribution between semantics and motion. Following the success of diffusion models [ 21 ] , numerous studies [ 22 , 23 , 24 , 25 , 26 ] shifted towards diffusion-based motion generation, such as the representative MLD [ 19 ] , which conducts diffusion within the latent space. In parallel, another paradigm, combining motion vector quantization [ 27 , 28 , 29 ] and generative transformers [ 30 , 31 , 32 ] (called VQ-GT) in a two-stage framework [ 2 , 12 , 11 , 13 ] , has achieved competitive performance. e.g. , T2M-GPT [ 14 ] quantizes motion into specialized discrete tokens and utilizes a transformer to generate continuous human motion. However, the VQ process inherently introduces approximation errors, which led the latest state-of-the-art method, MoMask [ 15 ] , to improve generation precision by introducing a hierarchical residual vector quantization variational autoencoder (RQ-VAE) [ 33 ] to preserve fine-grained details (see Fig. 1 (a) MoMask’s VQ). During the GT process, MoMask separates the hierarchical tokens into base (first-layer) and residual parts, and employs two independent masked transformers [ 34 ] to model them. While this approach offers stronger representation power compared to single-layer VQ-VAE, each layer of tokens is generated independently, leading explicit cross-layer misalignment (see Fig. 1 (c) MoMask’s GT). As a result, if the residual transformer fails to capture the structural context of the input tokens, it may lead to incoherent detail refinement. To better exploit intermediate representations, we propose a novel framework, MoSa, that enhances the VQ-GT paradigm through a coarse-to-fine scalable generation process. In MoSa, we first introduce a Multi-scale Token Preservation Strategy (MTPS) integrated into the RQ-VAE. MTPS leverages interpolation at each hierarchical quantization level to effectively retain multi-scale tokens from coarse to fine. With this, our autoregressive transformer is capable of jointly modeling all intermediate representations during the GT phase, thereby mitigating the aforementioned cross-layer misalignment and enhancing the global modeling capacity of the transformer. Fig. 1 : Comparison between state-of-the-art method MoMask [ 15 ] and our MoSa in the VQ-GT processes: (a) MoMask’s VQ. (b) Our VQ maintains a multi-scale token set via our proposed MTPS, which employs interpolation (downsample/upsampling) at each hierarchical quantization. (c) MoMask’s GT process relies on two independent transformers, leading to cross-layer misalignment. (d) Our GT process with a scalable autoregressive transformer shows cross-layer alignment. Precisely, in VQ, as illustrated in Fig. 1 (b), unlike previous methods [ 15 , 35 ] that directly preserve all same-scale intermediate tokens, our Multi-scale Token Preservation Strategy 2(MTPS) maintains a hierarchical token set across multiple scales. Starting from the first layer of the residual quantizer, motion sequences downsampling into coarse-scale representations. These are then quantized, with upsampling to align with the original motion length. As the hierarchy deepens, finer scales are progressively introduced, with tokens from each scale retained until reaching the final granularity ( i.e. , 49 tokens corresponding to motion latent length). MTPS enables us to extend the classic autoregressive (AR) modeling paradigm into a Scalable-Autoregressive (SAR) modeling. As shown in Fig. 1 (d), during training, the transformer jointly learns from the entire multi-scale token set by scanning from coarse to fine scales. At inference time, instead of generating one token at a time as in conventional AR, the model predicts multiple tokens of the next scale in parallel. Consequently, the total number of inference steps is reduced to the number of RQ-VAE quantization layers, which is set to 10 in our experiments, allowing the generation process to be completed in just 10 steps. Although MTPS brings significant improvements in inference speed, its frequent interpolation operations ( i.e. , downsampling and upsampling) introduce detail distortions during motion reconstruction. To address this critical issue, we further optimize the encoder-decoder architecture. Specifically, we propose CAQ-VAE, a lightweight yet expressive convolution-attention hybrid VQ-VAE. CAQ-VAE enhances the design of residual blocks and incorporates attention mechanisms to better capture global dependencies. Notably, the model size of CAQ-VAE remains comparable to prior methods. Experimental results show that MoSa achieves approximately a 27% improvement in inference speed while maintaining state-of-the-art generation quality. For example, on the latest and largest Motion-X dataset, MoSa achieves an FID of 0.06 (vs. 0.20 by MoMask), demonstrating superior overall performance. In addition, we explore the extensibility of MoSa. We demonstrate that MoSa can also be applied to motion editing tasks such as motion inpainting and outpainting without any additional fine-tuning. The model achieves strong qualitative results in these settings as well. In summary, our main contributions are as follows: • We propose MoSa, which introduces a Multi-scale Token Preservation Strategy (MTPS) to retain motion tokens across different scales. This strategy enables Scalable-Autoregressive (SAR) modeling, which jointly models all intermediate representations and generates motions in a coarse-to-fine manner. • We propose CAQ-VAE, a lightweight yet expressive convolution-attention hybrid VQ-VAE that mitigates detail distortion during motion reconstruction. • We explore the extensibility of MoSa and show that it generalizes well to motion editing tasks without requiring additional fine-tuning. Fig. 2 : Our MoSa framework overview. (a) Multi-scale Token Preservation Strategy (MTPS) integrated into a hierarchical RQ-VAE. MTPS employs interpolation