StructBiHOI: Structured Articulation Modeling for Long-Horizon Bimanual Hand–Object Interaction Generation

Zhi Wang, Liu Liu, Ruonan Liu, Dan Guo, Meng Wang · 2026

Read StructBiHOI: Structured Articulation Modeling for Long-Horizon Bimanual Hand–Object Interaction Generation by Zhi Wang, Liu Liu, Ruonan Liu, Dan Guo, Meng Wang (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2603-08390.txt.

Introduction

Wang and colleagues present DuetHOI, a framework for bimanual articulated hand-object motion. Doing this means tracking object articulation, hand coordination, and contact. Older methods pack object and hand motion into one sequence, blurring task progress and local pose. DuetHOI starts with ContactVAE predicting contact intent from language and geometry, then ArtPlanner supplies the plan. DualFormer builds it, and ProxiRefine fixes the object path while fixing both hands from proximity. Tests across articulated and rigid objects beat three baselines on contact metrics, with the best edge in bimanual articulated interaction.

Catalog record

CREATORZhi Wang, Liu Liu, Ruonan Liu, Dan Guo, Meng Wang
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE36903
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2603.08390

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

StructBiHOI: Structured Articulation Modeling for Long-Horizon Bimanual Hand–Object Interaction Generation Zhi Wang, Liu Liu, Ruonan Liu, Dan Guo, Meng Wang Abstract Recent progress in 3D hand–object interaction (HOI) generation has primarily focused on single-hand grasp synthesis, while bimanual manipulation remains significantly more challenging. Long-horizon planning instability, fine-grained joint articulation, and complex cross-hand coordination make coherent bimanual generation difficult, especially under multimodal conditions. Existing approaches often struggle to simultaneously ensure temporal consistency, physical plausibility, and semantic alignment over extended sequences. We propose StructBiHOI, a Structured articulation modeling framework for long-horizon Bimanual HOI generation. Our key insight is to structurally disentangle temporal joint planning from frame-level manipulation refinement. Specifically, a jointVAE models long-term joint evolution conditioned on object geometry and task semantics, while a maniVAE refines fine-grained hand poses at the single-frame level. To enable stable and efficient long-sequence generation, we incorporate a state-space-inspired diffusion denoiser based on Mamba, which models long-range dependencies with linear complexity. This hierarchical design facilitates coherent dual-hand coordination and articulated object interaction. Extensive experiments on bimanual manipulation and single-hand grasping benchmarks demonstrate that our method achieves superior long-horizon stability, motion realism, and computational efficiency compared to strong baselines. Figure 1 : Given a continuous action instruction, our framework generates a long-horizon sequence of continuous actions. The method hierarchically disentangles long-term joint planning and frame-level hand articulation, producing coherent and physically plausible manipulation motions. 1 Introduction Generating human-like Hand–Object Interaction (HOI) conditioned on object geometry and language instructions is a fundamental problem for dexterous robotics [ 11 ] , virtual character animation [ 30 ] , and embodied AI [ 21 ] . Beyond achieving physically stable grasps, realistic manipulation must reflect task intent, semantic understanding, and temporally coherent motion evolution. Human manipulation is inherently goal-driven and unfolds as a continuous interaction process shaped jointly by object structure, semantic intent, and long-horizon planning. Recent advances in single-hand HOI generation [ 12 , 24 ] have demonstrated promising results in grasp synthesis and manipulation planning under multimodal conditions. Leveraging large-scale motion data and generative models, these approaches can produce physically plausible and semantically consistent motions for diverse objects. However, unlike single-hand grasping synthesis, bimanual interaction involves coordinated control of two articulated kinematic chains, where mutual constraints, contact dependencies, and asymmetric role assignment introduce significantly higher structural complexity. This results in the fact that simply extending single-hand modeling paradigms to bimanual HOI manipulation is far from trivial. Current bimanual HOI generation works can be summarized into physics-based reinforcement learning and data-driven generative modeling ways, where the former aims to learn manipulation policies in simulation considering physically plausible behaviors but relying heavily on simulator fidelity and reward engineering, while the latter models interaction dynamics from large-scale multimodal data with sequence-based articulation architectures. Despite these substantial progresses, these works often fail in long-horizon (over 150 frames) bimanual HOI generation and suffer from three fundamental challenges: (1) Modeling long-range temporal dependencies becomes increasingly computationally demanding as sequence length grows, particularly in diffusion-based frameworks that repeatedly apply the sequence backbone during iterative denoising. (2) Fine-grained joint-level articulation and high-level manipulation semantics are tightly coupled, making it difficult to achieve both stable long-term planning and accurate local pose refinement. (3) Bimanual coordination requires consistent cross-hand dynamics, where errors in one hand may propagate temporally and spatially, leading to unstable or unrealistic motion evolution. To address these issues, we propose StructBiHOI , a Struct ured articulation modeling framework for long-horizon Bi manual H and– O bject I nteraction generation. Our key insight is that bimanual manipulation exhibits strong temporal continuity and state-dependent evolution, which can be more naturally modeled through structured state-space dynamics rather than global attention. Instead of treating long-sequence generation and fine-grained pose synthesis as a monolithic problem, we introduce a hierarchical and structurally disentangled modeling strategy. Specifically, we first map multimodal conditional information—including language intent, object geometry, and temporal context—into a unified latent space. On top of this representation, we design two conditional variational autoencoders (cVAE): 1) JointVAE, which focuses on long-horizon joint motion planning and captures temporally coherent interaction dynamics; 2) ManiVAE, which operates at the single-frame level to refine detailed hand poses conditioned on the current state and semantic intent. This hierarchical decomposition separates global motion evolution from local pose refinement, reducing modeling complexity and improving stability in long sequences. To further enable efficient long-range dependency modeling, we incorporate a Mamba-based selective state space model [ 7 , 5 ] as the denoising network within a latent diffusion framework. The linear-complexity state-space formulation allows scalable long-horizon modeling while preserving temporal continuity. Combined with our hierarchical disentanglement strategy, the proposed framework generates globally coherent and semantically consistent motion sequences for both hands and articulated object joints. We conduct extensive evaluations of StructBiHOI on the challenging bimanual manipulation benchmark ARCTIC [ 6 ] . Experimental results demonstrate that our method consistently outperforms state-of-the-art approaches across a comprehensive set of metrics, including generation efficiency, motion realism, and coordination quality. Notably, the performance gains become increasingly significant in complex long-horizon bimanual scenarios, where precise inter-hand coordination and articulated object reasoning are required. These results highlight the effectiveness of our structured latent formulation and Mamba-based diffusion denoiser in modeling temporally consistent and articulation-aware interactions. Furthermore, we validate StructBiHOI on a single-hand grasping benchmark to assess its generalization capability beyond bimanual settings. Despite being designed for structured bimanual interaction modeling, StructBiHOI achieves superior performance compared to existing single-hand methods, demonstrating the robustness and scalability of our hierarchical generative framework. In summary, our main contributions are three-fold: 1) We present StructBiHOI, an efficient generative framework for long-horizon bimanual hand–object interaction, designed to address the scalability and stability challenges in long-sequence HOI generation. 2) We introduce a hierarchical and structurally disentangled modeling strategy, decomposing long-term joint motion planning and single-frame fine-grained hand pose refinement via jointVAE and maniVAE, enabling semantically consistent and temporally stable generation. 3) We incorporate a Mamba-based selective state space denoiser within a latent diffusion framework, achieving linear-complexity long-range dependency modeling and significantly improving efficiency, motion realism, and bimanual coordination quality over state-of-the-arts. 2 Related Works 2.1 Bimanual Hand Object Interaction HOI generation methods can generally be categorized into physics-based and data-driven approaches. Physics-based methods [ 28 , 29 , 23 , 3 ] leverage reinforcement learning within simulation environments to learn manipulation policies that ensure physical plausibility. These approaches reduce reliance on large-scale motion datasets and can produce dynamically consistent behaviors. However, their performance heavily depends on simulator fidelity and carefully engineered reward functions, which often limits generalization to unseen objects and interaction scenarios. Data-driven approaches [ 24 , 16 , 14 , 4 , 19 , 25 , 32 ] instead model HOI dynamics directly from large-scale multimodal motion data using generative frameworks such as autoencoder based [ 12 , 9 , 10 , 17 , 20 ] and diffusion-based models [ 13 , 1 ] . By learning joint motion distributions in a data-driven manner, these methods achieve realistic and semantically aligned interaction synthesis. Nevertheless, most existing approaches focus primarily on short-horizon or single-hand settings, and often treat joint-level articulation implicitly within end-to-end sequence models. As a result, structured modeling of long-horizon articulation dynamics and coherent bimanual coordination remains insufficiently explored. 2.2 Long-Horizon Motion Modeling Recent advances in motion diffusion models have achieved

Read the complete text