Disentangled Hierarchical VAE for 3D Human-Human Interaction Generation

Zichen Geng, Zeeshan Hayder, Bo Miao, Jian Liu, Wei Liu, Ajmal Mian, [4pt] Department of CSSE, The University of Western Australia, Crawley WA 6009, Australia · 2026

Read Disentangled Hierarchical VAE for 3D Human-Human Interaction Generation by Zichen Geng, Zeeshan Hayder, Bo Miao, Jian Liu, Wei Liu, Ajmal Mian, [4pt] Department of CSSE, The University of Western Australia, Crawley WA 6009, Australia (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2603-00144.txt.

Introduction

Geng and colleagues present DHVAE, a latent-diffusion system for making realistic 3D human-human interaction that users can steer. It splits the big-picture interaction setting from single motion patterns into two linked latent parts via a CoTransformer module. Contrastive learning rules sharpen the latent interaction space toward sharp, true-to-life contacts, curbing penetration and missed contact. A DDIM-based diffusion pass with a skip-connected AdaLN-Transformer denoiser then builds high-fidelity interaction, beating rivals on motion fidelity, text match and physical plausibility with less compute.

Catalog record

CREATORZichen Geng, Zeeshan Hayder, Bo Miao, Jian Liu, Wei Liu, Ajmal Mian, [4pt] Department of CSSE, The University of Western Australia, Crawley WA 6009, Australia
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE67482
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2603.00144

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Disentangled Hierarchical VAE for 3D Human-Human Interaction Generation Zichen Geng, Zeeshan Hayder, Bo Miao, Jian Liu, Wei Liu, Ajmal Mian, [4pt] Department of CSSE, The University of Western Australia, Crawley WA 6009, Australia Abstract Generating realistic 3D Human-Human Interaction (HHI) requires coherent modeling of the physical plausibility of the agents and their interaction semantics. Existing methods compress all motion information into a single latent representation, limiting their ability to capture fine-grained actions and inter-agent interactions. This often leads to semantic misalignment and physically implausible artifacts, such as penetration or missed contact. We propose Disentangled Hierarchical Variational Autoencoder (DHVAE) based latent diffusion for structured and controllable HHI generation. DHVAE explicitly disentangles the global interaction context and individual motion patterns into a decoupled latent structure by employing a CoTransformer module. To mitigate implausible and physically inconsistent contacts in HHI, we incorporate contrastive learning constraints with our DHVAE to promote a more discriminative and physically plausible latent interaction space. For high-fidelity interaction synthesis, DHVAE employs a DDIM-based diffusion denoising process in the hierarchical latent space, enhanced by a skip-connected AdaLN-Transformer denoiser. Extensive evaluations show that DHVAE achieves superior motion fidelity, text alignment, and physical plausibility with greater computational efficiency. 1 Introduction Humans naturally coordinate with each other through timed and spatially aligned actions, such as shaking hands, dancing, playing sports, or passing objects. Representing and generating such human-human interactions (HHIs) in 3D is a core challenge in embodied AI, with broad impact on virtual character animation, human-robot collaboration, and embodied communication. Given a simple natural language prompt like “Person A hands an object to Person B,” generating motion sequences for both agents that are semantically aligned, temporally coherent, and physically plausible remains an open challenge. Early progress in motion generation largely focused on single-agent synthesis, where approaches like T2M-GPT 41 , MotionGPT 16 , and MDM 35 successfully model long-range temporal dynamics conditioned on text. However, extending these techniques to multi-agent interactions is non-trivial. HHI generation presents unique challenges: it requires modeling synchronized dynamics between multiple agents, capturing both mutual awareness and individual autonomy, and handling diverse interaction semantics ranging from high-level coordination to fine-grained local motions. Recent advances in generative modeling, particularly latent diffusion models (LDMs) 28 , have shown impressive performance in synthesizing high-dimensional data across domains, including human motion. By operating in a compressed latent space, these models enable efficient learning and scalable generation. MLD 3 is the first to adopt this paradigm for single-agent motion generation, achieving good results through temporal-aware latent diffusion. Extending LDMs to human-human interaction, however, remains underexplored. A few recent attempts, such as InterLDM 19 , have begun applying this idea to HHI. As illustrated in Fig. 1 (a), these approaches encode both agents into a flat, unified latent representation and apply diffusion in this joint space. While such a design enables synchronized modeling, it entangles agent identity with interaction context, leading to limited expressivity and degraded realism, particularly in cases requiring fine-grained coordination or distinct agent behaviors. Moreover, as illustrated in Fig. 1 (b), models such as InterMask 13 encode motion within a unified latent space shared across agents. While this approach has achieved state-of-the-art (SOTA) performance, it lacks explicit modeling of global interactions between agents, often resulting in physically implausible outcomes, e.g., hand penetration or missed contact in tasks like “two people shake hands.” These failure cases underscore two fundamental limitations in current methods: the absence of structured interaction representations and limited control over contact realism. Figure 1: (a) InterLDM 19 , (b) InterMask 13 encode all motion information into a single latent. (c) Our encodes individual motions and interactions into separate disentangled latents. We propose a new paradigm for text-conditioned human-human interaction generation: Disentangled Hierarchical Variational Autoencoder (DHVAE) paired with structured latent diffusion. Our key innovation is to explicitly disentangle the HHI representation into three levels: (1) \mathbf{z}_{a} , modeling Person A’s individual motion; (2) \mathbf{z}_{b} , modeling Person B’s individual motion; and (3) \mathbf{z}_{o} , a shared latent capturing the global interaction context. To fully leverage this hierarchy, we introduce a CoTransformer module that jointly encodes mutual awareness and preserves individual agent identity. Beyond architectural design, we further enhance the learning of the interaction representation \mathbf{z}_{o} using a contrastive learning strategy. We introduce a simple yet effective approach to construct positive and negative HHI pairs, imposing prior-based supervision on \mathbf{z}_{o} to encourage it to encode meaningful and physically plausible interactions. This design directly addresses the lack of prior-based modeling for physically realistic contact interactions, a key limitation in prior works. The structured latents \{\mathbf{z}_{o},\mathbf{z}_{a},\mathbf{z}_{b}\} are passed through an AdaLN 26 Transformer-based denoiser trained using a Denoising Diffusion Implicit Model (DDIM) 33 process in the latent space. To address the scale imbalance and structural heterogeneity between \mathbf{z}_{o} , \mathbf{z}_{a} , and \mathbf{z}_{b} , we introduce segment positional encoding (SPE) to reflect each token’s role in the interaction, and token scaling to calibrate feature magnitudes across latent groups. We also adopt skip connections for the AdaLN Transformer to stabilize training while allowing flexible conditioning. Overall, we propose a disentangled and controllable framework for generating HHI motion from natural language with high alignment to text and strong physical plausibility. Evaluations on the popular InterHuman 21 and InterX 39 benchmarks demonstrate that our model significantly outperforms SOTA counterparts across all major metrics, including FID, R-Precision, and Multimodal Distance. Ablation studies confirm the value of our architectural decisions. Our main contributions include: • We propose a disentangled hierarchical VAE which separates the latent representation of human-human interactions into three components: the individual motion components, and the global interaction component, enabling controllable and personalized generation. • We propose a contrastive learning strategy over the global interaction latent \mathbf{z}_{o} to enable prior-based modeling of interaction semantics and improved physical plausibility, especially for contact-sensitive regions. • Our model is the lightest and fastest, and sets new SOTA performance on the InterHuman and InterX benchmarks on multiple metrics, demonstrating superior text-motion alignment and fidelity. 2 Related Work Human Motion Generation. Recent advances in human motion generation have been fueled by large-scale motion capture datasets and the emergence of generative models. Early methods relied on aligning latent spaces between text and motion using objectives like Kullback-Leibler divergence 11 ; 10 , or directly learning motion embeddings from paired descriptions 1 . With the advent of Transformer models, autoregressive approaches such as T2M-GPT 41 and MotionGPT 16 represent motion as discrete tokens and generate them sequentially. Despite producing coherent results, these methods often struggle with long-term dependencies and lack bidirectional modeling. To address these limitations, non-autoregressive models have gained popularity, including masked motion transformers such as MoMask 8 and MMM 27 , which leverage bidirectional context prediction via masked token reconstruction. In parallel, diffusion-based frameworks like MDM 35 , MotionDiffuse 42 , and FLAME 18 have achieved promising results in generating temporally consistent and realistic motion. However, these methods are designed for single-person motion and struggle to model complex multi-agent interactions. Human-Human Interaction Generation. Human-human interaction modeling extends the challenges of motion generation by requiring coherent inter-personal dynamics. Prior works can be broadly categorized into reaction-based and joint generation paradigms. Reaction models such as 4 ; 7 ; 40 ; 14 generate one agent’s motion conditioned on another’s actions, but often lack

Read the complete text