MARRS: Masked Autoregressive Unit-based Reaction Synthesis
Yabiao Wang *, Shuo Wang *, Jiangning Zhang, Jiafu Wu, Qingdong He, Yong Liu · 2025
Read MARRS: Masked Autoregressive Unit-based Reaction Synthesis by Yabiao Wang *, Shuo Wang *, Jiangning Zhang, Jiafu Wu, Qingdong He, Yong Liu (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2505-11334.txt.
Introduction
Wang and five co-authors built a tool in twenty twenty-five that makes a person's reaction moves to a partner's acts in a dance-like duet. Past tools cut moves into codes, which loses fine detail and wastes code slots. Their tool codes body and hand parts apart in smooth form.
A mask step hides some reaction parts and fills them from the act parts that stay. A swap step lets body and hand parts shape each other as they form. Small nets guess noise for each part in a spread step. Tests show top scores past past tools, they report.
Catalog record
| CREATOR | Yabiao Wang *, Shuo Wang *, Jiangning Zhang, Jiafu Wu, Qingdong He, Yong Liu |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 50 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2505-11334
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
MARRS: Masked Autoregressive Unit-based Reaction Synthesis Yabiao Wang *, Shuo Wang *, Jiangning Zhang, Jiafu Wu, Qingdong He, Yong Liu Abstract This work aims at a challenging task: human action-reaction synthesis, i.e. , generating human reactions conditioned on the action sequence of another person. Currently, autoregressive modeling approaches with vector quantization (VQ) have achieved remarkable performance in motion generation tasks. However, VQ has inherent disadvantages, including quantization information loss, low codebook utilization, etc . In addition, while dividing the body into separate units can be beneficial, the computational complexity needs to be considered. Also, the importance of mutual perception among units is often neglected. In this work, we propose MARRS, a novel framework designed to generate coordinated and fine-grained reaction motions using continuous representations. Initially, we present the Unit-distinguished Motion Variational AutoEncoder (UD-VAE), which segments the entire body into distinct body and hand units, encoding each independently. Subsequently, we propose Action-Conditioned Fusion (ACF), which involves randomly masking a subset of reactive tokens and extracting specific information about the body and hands from the active tokens. Furthermore, we introduce Mutual Unit Modulation (MUM) to facilitate interaction between body and hand units by using the information from one unit to adaptively modulate the other. Finally, for the diffusion model, we employ a compact MLP as a noise predictor for each distinct body unit and incorporate the diffusion loss to model the probability distribution of each token. Both quantitative and qualitative results demonstrate that our method achieves superior performance. Project page: https://aigc-explorer.github.io/MARRS/ . I Introduction In the field of generative computer vision, human-human motion generation holds substantial importance for computer animation [ 1 , 2 , 3 ] , game development [ 4 , 5 , 6 ] , and robotic control [ 7 , 8 , 9 , 10 ] . In this work, we focus on generative models for human action-reaction synthesis, i.e. , generating human reactions based on the action sequence of another individual as conditions, which can significantly reduce the workload for animators by enabling them to design an acting character and automatically generate meaningful motions for a reacting character. Although autoregressive modeling approaches have recently achieved remarkable performance in motion generation tasks [ 11 ] , it is sub-optimal to apply these methods to human action-reaction synthesis. For example, our experiments demonstrate that the reconstruction and generation performance of VQ-VAE are unsatisfactory. There are two challenges for this: first, vector quantization (VQ) exhibits inherent limitations. Mapping continuous motion data to a constrained set of discrete tokens inevitably results in a loss of information. Although Momask [ 12 ] utilizes successive stages of residual quantization to minimize quantization errors incrementally, complex multistage training does not fundamentally eliminate quantization errors. Second, the notorious codebook collapse [ 13 , 14 ] makes optimizing the code vectors in the existing VQ-VAE not entirely trivial. Without relying on VQ-VAE, some methods [ 15 ] have tried autoregressive diffusion in motion generation, but their cumbersome training and inference processes do not achieve stunning performance in single-segment motion generation. Moreover, units splitting has been shown to be an effective strategy in motion generation. However, as the number of units increases, the computational effort of the model also increases. And the importance of mutual perception among units is often neglected. Thus, it is necessary to explore a more rational units splitting as well as to design an efficient and effective units perception method. Fig. 1: Left: Paradigm comparison of different frameworks. (a) and (b) present the structures of the VQ-VAE-based and Diffusion-based methods, respectively, while (c) shows the framework of our proposed MARRS. \mathcal{L}_{CE} is cross entropy loss, \mathcal{L}_{Diff} is diffusion loss. Right: result comparison among our method and other methods on eight metrics. Recently, some works [ 16 ] have explored autoregressive image generation without vector quantization, providing new insight into autoregressive generation methods. In this paper, we propose MARRS, a novel framework that generates synchronized and fine-grained reaction movements in continuous representations. The framework has two advantages. First, compared with VQ-based frameworks, our motion representation operates in a continuous space, which helps to avoid the quantization errors associated with vector quantization(VQ). Second, diffusion-based frameworks typically model the joint distribution of all tokens within a single denoising process, whereas our framework factorizes the problem and models the conditional distribution of one token at a time. This decomposition empirically eases optimization and is more conducive to effective learning. As shown in Fig. 1 , we provide a paradigm comparison of different frameworks and a result comparison among different methods. During the training phase, MARRS follows a two-stage paradigm. In the first stage, for the network to learn the concept of the body and hands, we propose Unit-distinguished Motion Variational AutoEncoder (UD-VAE), which splits the whole-body into body and hands units and then encodes them independently using a variational autoencoder. In the second stage, we propose Action-Conditioned Fusion (ACF), which involves randomly masking out a portion of the reactive tokens and then extracting the information of body and hands from the active tokens separately using the transformer. Next, in order to avoid the lack of mutual perception between the two units (body and hands), we propose Mutual Unit Modulation (MUM), which utilizes information from the body or hands to modulate the other unit adaptively and enables interaction between the body and hands generators to jointly produce whole-body motion. Finally, we utilize a compact MLP to serve as a noise predictor for each distinct unit, modeling individual tokens separately and integrating the diffusion loss [ 16 ] to assess the probability of each token. Our contributions can be summarized as follows: • we propose MARRS, a novel framework that generates synchronized and fine-grained reaction movements. To the best of our knowledge, this is the first successful application of masked autoregressive generation to the field of action-reaction synthesis. • We introduce the Unit-distinguished Motion Variational AutoEncoder (UD-VAE) to enable the network to grasp the concepts of the body and hands. Next, we propose Action-Conditioned Fusion (ACF), which could extract information about the body and hands of the reactor from the active tokens. Furthermore, to prevent the body and hand units from lacking awareness of each other, we propose Mutual Unit Modulation (MUM). Finally, we employ a compact MLP as a noise predictor and integrate the diffusion loss to estimate the probability of each token. • We conduct extensive experiments on the human action-reaction synthesis datasets, NTU120-AS and Chi3D-AS. The results in both online and offline settings demonstrate the effectiveness of our proposed methods. II Related Work II-A Motion Generation Framework TEMOS [ 17 ] and T2M [ 18 ] use a Transformer-based VAE combined with a text encoder to create motion sequences from textual descriptions. T2M-GPT [ 19 ] introduces a framework based on VQ-VAE and Generative Pretrained Transformer (GPT) for motion generation. MMM [ 11 ] proposes a conditional masked motion architecture based on VQ-VAE. To mitigate quantization errors, MoMask [ 12 ] introduces a hierarchical quantization scheme. MotionDiffuse [ 20 ] and MDM [ 21 ] are groundbreaking frameworks for text-driven motion generation using diffusion models. Instead of using a diffusion model to connect raw motion sequences with conditional inputs, MLD [ 22 ] employs a latent