OpenHOI: Open-World Hand-Object Interaction Synthesis with Multimodal Large Language Model

Zhenhao Zhang, Ye Shi, Lingxiao Yang, Suting Ni, Qi Ye, Jingya Wang · 2025

Read OpenHOI: Open-World Hand-Object Interaction Synthesis with Multimodal Large Language Model by Zhenhao Zhang, Ye Shi, Lingxiao Yang, Suting Ni, Qi Ye, Jingya Wang (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2505-18947.txt.

Introduction

Zhenhao Zhang and co-authors generate hand and object scenes in a 2025 computer vision paper. Realistic hand action matters for virtual worlds and dexterous robots. Older methods cope with known objects and set tasks but fail on new objects or free form commands. Their system, OpenHOI, claims a first for open world synthesis, building long manipulation runs for novel objects from plain language.

A three dimensional multimodal model, tuned for the job, finds where on an object a hand should act, like handles and buttons. It also breaks a command into steps. Joining grounding and planning in one model may let virtual hands handle the long tail of daily objects, not just a training set.

Catalog record

CREATORZhenhao Zhang, Ye Shi, Lingxiao Yang, Suting Ni, Qi Ye, Jingya Wang
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE60 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2505-18947

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

OpenHOI: Open-World Hand-Object Interaction Synthesis with Multimodal Large Language Model Zhenhao Zhang, Ye Shi, Lingxiao Yang, Suting Ni, Qi Ye, Jingya Wang Abstract Understanding and synthesizing realistic 3D hand-object interactions (HOI) is critical for applications ranging from immersive AR/VR to dexterous robotics. Existing methods struggle with generalization, performing well on closed-set objects and predefined tasks but failing to handle unseen objects or open-vocabulary instructions. We introduce OpenHOI, the first framework for open-world HOI synthesis, capable of generating long-horizon manipulation sequences for novel objects guided by free-form language commands. Our approach integrates a 3D Multimodal Large Language Model (MLLM) fine-tuned for joint affordance grounding and semantic task decomposition, enabling precise localization of interaction regions (e.g., handles, buttons) and breakdown of complex instructions (e.g., “Find a water bottle and take a sip”) into executable sub-tasks. To synthesize physically plausible interactions, we propose an affordance-driven diffusion model paired with a training-free physics refinement stage that minimizes penetration and optimizes affordance alignment. Evaluations across diverse scenarios demonstrate OpenHOI’s superiority over state-of-the-art methods in generalizing to novel object categories, multi-stage tasks, and complex language instructions. Our project page at https://openhoi.github.io 1 Introduction Figure 1: Motivation: OpenHOI introduces an open-world framework for generating HOI sequences that demonstrates strong generalization across seen and unseen objects, high-level instructions, and long-horizon tasks. Hand-object interaction (HOI) involves jointly modeling hand articulation and object dynamics to generate and interpret realistic manipulation sequences [ 9 , 17 , 25 , 26 , 15 , 47 , 5 ] . This reflects one of the most pervasive human behaviors, deeply embedded in daily activities. Generating and understanding 3D HOI sequences is critical for advancing machine capabilities in human-centric applications. In augmented/virtual reality (AR/VR), realistic HOI modeling enables immersive digital experiences, allowing users to manipulate virtual objects naturally. For robotics, it provides the foundation for dexterous, feedback-driven manipulation in unstructured environments. Generating HOI sequences from natural language instructions remains a significant challenge in 3D interaction research. While traditional methods [ 19 , 40 ] rely on handcrafted motion priors, recent diffusion-based approaches [ 2 , 3 ] directly map text to action sequences, eliminating the need for manual motion design. However, due to limited data and modeling capacity, these models only deal with closed datasets and struggle to generalize to unseen objects and open-vocabulary instructions. Recent advances in Large Language Models (LLMs) have significantly enhanced joint vision-language understanding. Building on this progress, concurrent work like HOIGPT [ 13 ] proposes an LLM-based architecture for aligning textual instructions with HOI sequences. However, these approaches face fundamental generalization challenges in unseen shapes and complex 3D interaction scenarios due to the lack of 3D knowledge in LLM. Fig. 1 illustrates the comparison between conventional Closed-Set HOI and Open-World HOI. The growth of 3D datasets has advanced 3D multimodal large language models (MLLMs) with strong capabilities in understanding and grounding 3D geometric, semantic, and functional relationships. Critically, fine-grained part-level grounding and affordance grounding [ 34 ] [ 4 ] can identify actionable object regions and their interaction possibilities, providing strong priors for synthesizing physically consistent hand-object interactions (HOI). Building on this insight, we propose OpenHOI, the first open-world HOI synthesis framework capable of generating long-horizon manipulation sequences for unseen objects guided by open-vocabulary instructions. Our approach fine-tunes a 3D MLLM endowed with comprehensive affordance reasoning priors, enabling two core capabilities: 1) Generalizable Affordance Prediction: Precise localization of interaction regions (e.g., handles, buttons) for both known and novel objects. 2) Instruction Decomposition: Breaking down open-vocabulary commands (e.g., “I’m feeling thirsty, could you find a water bottle and take a sip”) into executable sub-tasks grounded in object affordances. By integrating these priors with a diffusion-based interaction generation and physics-aware refinement, OpenHOI achieves unprecedented generalization across object categories, task horizons, and linguistic complexity. In summary, our contributions are as follows: • We introduce OpenHOI, the first open-world hand-object interaction synthesis framework capable of generating long-horizon manipulation sequences for unseen objects guided by open-vocabulary instructions. Unlike prior closed-set HOI methods, OpenHOI generalizes to novel objects and complex, linguistically diverse commands. • OpenHOI involves fine-tuning a 3D MLLM that jointly learns geometric affordance priors and semantic task decomposition. Subsequently, affordance-driven HOI Diffusion with Physical Refinement is developed to generate realistic HOI sequences for each task. • Our experiments demonstrate state-of-the-art performance, outperforming existing methods with a large margin. Notably, OpenHOI generalizes robustly to unseen objects and open-vocabulary instructions, achieving strong compositional generalization across diverse scenarios. Table 1: Comparative analysis of our method versus existing HOI synthesis approaches. Methods Open-world 3D MLLM Open-vocabulary Long-horizon Planning Motion In-between HandDiffuse [ 23 ] ✓ Text2hoi [ 2 ] HOIGPT [ 13 ] ✓ ✓ Ours ✓ ✓ ✓ ✓ ✓ ✓ 2 Related Work Hand-Object Interaction Synthesis. Synthesizing realistic and diverse hand-object interactions (HOI) have gained significant attention, with numerous approaches exploring this problem under various settings. These include the creation of large-scale datasets like GigaHands [ 7 ] and OAKINK2 [ 53 ] , methods for affordance learning [ 21 , 56 , 32 , 46 ] . Others have focused on synthesizing two-hand interactions like InterHandGen [ 19 ] and HandDiffuse [ 23 ] , or on physics-aware synthesis and dexterous grasping [ 44 , 45 , 57 ] . Several key works have specifically tackled text-guided or semantically rich HOI synthesis. Text2HOI [ 2 ] introduced a pioneering approach to generate 3D Hand-Object Interaction Sequences from text by decomposing the task into contact prediction and motion generation using a diffusion model, but it can struggle with fine-grained control from low-level text and tends to produce short HOI interaction sequences. HOIGPT [ 13 ] adopts an LLM-based sequential model to predict hand-object trajectories from text for long-horizon sequences, yet it lacks explicit affordance-guided mechanisms and cannot ensure smooth motion in-between multiple sub-sequences in complex interactions. SemGrasp [ 20 ] focuses on semantic-aware grasp generation, but it discretizes motion space and is limited to static grasps. Grasp as You Say [ 45 ] addresses category-level language-guided grasping, but it requires hand-crafted hand-object contact templates and lacks dynamics. While these methods have advanced the field, generating fine-grained, diverse, and long-horizon hand-object interaction sequences with high-level text on unseen objects cohesively remains a significant challenge. Multimodal Large Language Model. The advent of Large Language Models (LLMs) has revolutionized natural language understanding, and their influence is gradually expanding into 3D perception and interaction tasks. Some recent works leverage pure LLM architectures to handle motion generation from language. For instance, HOIGPT [ 13 ] and MotionGPT [ 16 ] adopt VQ-VAE tokenization and GPT-style architectures to achieve bidirectional generation between natural language and HOI systhesis. However, these models operate primarily on discrete motion tokens, and lack grounding in actual 3D perception or reasoning, they cannot perceive objects or environments in 3D nor reason about spatial affordances or constraints, thus limiting their ability to generate context-aware interactions. Conversely, Multimodal Large Language Models (MLLMs) have emerged as powerful tools that extend the success of LLMs into various domains, aiming to bridge the gap between language and other modalities such as images, videos, and 3D representations, enabling tasks like visual question answering like ShapeLLM [ 34 ] , 3D object generation like Point-E [ 29 ] and CLIP-Forge [ 37 ] ,

Read the complete text