Social Structure Matters in 3D Human-Human Interaction Generation

Zhongju Wang, Beier Wang, Yatao Bian, Pichao WANG, Zhi Wang, Daoyi Dong, Hongdong Li, Huadong Mo, Zhenhong Sun · 2026

Read Social Structure Matters in 3D Human-Human Interaction Generation by Zhongju Wang, Beier Wang, Yatao Bian, Pichao WANG, Zhi Wang, Daoyi Dong, Hongdong Li, Huadong Mo, Zhenhong Sun (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2606-24255.txt.

Introduction

Wang and colleagues generate two-person motion from text by modelling the social script before the movement. Text-to-motion systems handle solo movers well, but two interacting people must follow an unfolding social structure: phases progress, roles are assigned, and each actor's motion answers the partner's. Their 2026 study first probes what large language models can and cannot do for this task.

Language models, the analysis shows, can recover phase decompositions and partner-aware roles, thinking through an interaction's structure convincingly. They cannot, however, emit dynamic, physically plausible, partner-aware motion directly. The resulting design splits labour: a language-model planner converts hidden interaction semantics into motion-aligned supervision, decomposing phases and assigning roles, while a dedicated motion executor renders continuous three-dimensional movement. Wang's planner-executor paradigm, thinking with the language model and moving with motion skills, grounds social structure into bodily coordination between the two actors.

Catalog record

CREATORZhongju Wang, Beier Wang, Yatao Bian, Pichao WANG, Zhi Wang, Daoyi Dong, Hongdong Li, Huadong Mo, Zhenhong Sun
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE61 KB
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2606.24255

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Social Structure Matters in 3D Human-Human Interaction Generation Zhongju Wang, Beier Wang, Yatao Bian, Pichao WANG, Zhi Wang, Daoyi Dong, Hongdong Li, Huadong Mo, Zhenhong Sun Abstract Although text-to-motion generation has achieved strong progress in synthesizing realistic single-person motions from language, extending it to text-driven 3D human-human interaction (HHI) remains non-trivial, as HHI requires modeling the underlying social structure that governs phase progression, actor roles, and inter-actor coordination. In this paper, we formulate HHI generation as a social structure modeling and grounding problem: the model must first infer how an interaction unfolds and how the two actors coordinate their roles, and then realize this structure as continuous, physically plausible, and partner-aware 3D motion. To study how such structure should be modeled, we first examine the capability boundary of large language models (LLMs) for HHI generation. Our analysis shows that LLMs can think by recovering phase decompositions and partner-aware roles, but cannot directly move , as they fail to generate dynamic, physically plausible, and interaction-aware motion. This motivates our planner-executor paradigm, Think with LLM, Move with Motion Skill . The LLM planner converts implicit interaction semantics into motion-aligned social supervision by decomposing interactions into phases, assigning partner-aware actor roles, and aligning them with motion sequence. The motion executor then grounds the planned social structure into coordinated two-person motion by adapting a pretrained solo motion model with LoRA, previous-phase self-conditioning, and ego-relative partner conditioning. Together, our Solo-to-Social framework bridges social organization and motion realization, producing 3D HHI with improved phase consistency, role alignment, and partner-aware coordination. Code: https://github.com/EngineeringAI-LAB/SocialStructureHHI 1 Introduction Text-to-motion generation has made rapid progress in synthesizing realistic single-person 3D motion from natural language [ 36 , 31 , 6 ] . Recent large-scale motion generators further show that pretrained motion backbones can learn strong atomic motion priors from broad single-person motion data [ 39 , 2 ] . As generative models move toward embodied AI [ 21 ] , multi-agent collaboration [ 44 ] , and social robotics [ 23 ] , the research focus is naturally shifting from individual motion synthesis to text-driven 3D human-human interaction (HHI) generation. In this setting, a model is expected to generate two coordinated human motions from a global interaction description, such as one person approaching another person, hugging them, and then releasing the hug. Compared with single-person motion generation, HHI generation requires not only realistic individual motion, but also coherent coordination between two actors over time. Existing HHI generation methods have made promising progress by jointly modeling two-person motion, composing individual motion priors, or introducing interaction-aware generation mechanisms [ 28 , 29 , 32 , 20 , 15 , 41 ] . However, treating HHI as a direct extension from one actor to two overlooks a key property of HHI: it is not merely a spatial combination of two plausible individual motions, but is organized by an underlying social structure , as shown in Fig. 1 . We define social structure as the latent interaction organization that governs how an interaction unfolds over time and how two actors coordinate their roles with respect to each other. It contains two essential dimensions: phase progression , which describes the temporal stages of an interaction, such as approach, contact, release, or in-place coordination; and partner-aware coordination , which captures the asymmetric but coupled responsibilities of the two actors in each phase, such as initiator and receiver, attacker and defender, or giver and taker. Without such structure, generated motions may appear plausible for each actor but fail as an interaction: the actors may approach at inconsistent times, miss the contact moment, face the wrong direction, or execute incompatible roles. Figure 1: Social structure of text-driven HHI generation. (a) Solo motion execution provides strong intra-personal motion priors but lacks interaction-level coordination. (b) HHI requires social structure along two dimensions: phase-level temporal organization and partner-aware coordination. (c) LLMs offer social planning abilities to make such structure explicit for interaction motion execution. This observation suggests that the core difficulty of text-driven HHI generation lies not only in motion realism, but also in recovering the social structure that guides two actors into coherent interactions. We therefore formulate HHI generation as a social structure modeling and grounding problem: the model must infer the interaction organization implied by a global language description, including phase decomposition and partner-aware role assignment, and ground it into continuous, physically plausible, and partner-aware 3D motion. In this view, phase consistency and partner awareness are not merely evaluation properties, but direct outcomes of correctly grounded social structure. Large Language Models (LLMs) are natural tools for making such structure explicit, given their strong language understanding [ 38 ] , commonsense reasoning [ 24 ] , and structured planning [ 55 ] abilities. Yet whether they can serve as complete HHI generators remains unclear. We therefore conduct a diagnostic study by representing Skinned Multi-Person Linear (SMPL) [ 22 ] parametric human motion in a token-like form and asking an LLM to generate interaction sequences from text. The results reveal a clear separation between thinking and moving : the LLM can infer plausible phase decompositions and partner-aware roles, but fails to reliably produce continuous, dynamic, and physically plausible 3D interaction motion. Thus, LLMs are suitable as social structure planners, but not as direct motion executors. Based on this insight, we propose a planner-executor paradigm for social-structure-centered HHI generation: Think with LLM, Move with Motion Skill . The LLM acts as a social structure planner, while a pretrained motion model serves as an executable motion skill. Instead of using the LLM as a direct motion generator, we use it to convert implicit interaction semantics into explicit social structure supervision. Given a global interaction prompt and paired motion sequence, we decompose the interaction into phase-level units, assign partner-aware roles to both actors, and align each phase with its motion segment. This converts coarse HHI text-motion pairs into fine-grained, motion-aligned social annotations, making social structure a trainable bridge between language intent and motion execution. To ground this structure into continuous two-person motion, we introduce a Solo-to-Social (S2S) motion execution framework, which adapts a pretrained solo motion backbone into an interaction motion executor rather than training from scratch. S2S preserves the atomic motion prior learned from single-person data while adding the social coordination ability required by HHI. Phase-wise self motion conditioning uses previous-phase motion prefixes to encourage smooth transitions and long-range consistency, while partner-aware motion conditioning injects the partner’s latest motion into the actor’s ego-centric frame to model relative position, orientation, and interaction geometry. With parameter-efficient Low-Rank Adaptation (LoRA), these mechanisms turn a solo motion model into a socially aware executor for coherent two-person motion. Experiments on standard HHI benchmarks show improved text-motion alignment, phase consistency, and partner-aware coordination over existing baselines, with qualitative results showing clearer phase progression, more role-consistent behaviors, and more plausible inter-person geometry. Overall, our framework treats social structure as the central abstraction of HHI generation, planned by LLM reasoning and grounded through motion skill adaptation. The main contributions of this paper are summarized as follows: \bullet We identify social structure as a central abstraction for text-driven 3D HHI generation, and formulate HHI generation as a social structure modeling and grounding problem rather than a direct extension of solo motion generation. \bullet

Read the complete text