InteractAnything: Zero-shot Human Object Interaction Synthesis via LLM Feedback and Object Affordance Parsing
Jinlu Zhang, Yixin Chen, Zan Wang, Jie Yang, Yizhou Wang, Siyuan Huang · 2025
Read InteractAnything: Zero-shot Human Object Interaction Synthesis via LLM Feedback and Object Affordance Parsing by Jinlu Zhang, Yixin Chen, Zan Wang, Jie Yang, Yizhou Wang, Siyuan Huang (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2505-24315.txt.
Introduction
Jinlu Zhang and five co-authors build a zero shot system that poses 3D people with objects from a text brief, in a 2025 arXiv paper. Making a person grasp, sit on, or lift a new object is hard for generators, above all for objects never seen in training. Poses must match both the words and the object's shape.
The system draws on large pre-trained models at each step. A language model first works out the person object relation and sets the object's place. An image model then reads the new object to find its grip and contact spots. Many pose guesses are drawn and scored in 3D. A final polish step uses language model feedback to force true contact at the right body parts, hands above all. Tests show natural, exact poses.
Catalog record
| CREATOR | Jinlu Zhang, Yixin Chen, Zan Wang, Jie Yang, Yizhou Wang, Siyuan Huang |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 35 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2505-24315
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
InteractAnything: Zero-shot Human Object Interaction Synthesis via LLM Feedback and Object Affordance Parsing Jinlu Zhang, Yixin Chen, Zan Wang, Jie Yang, Yizhou Wang, Siyuan Huang Abstract Recent advances in 3D human-aware generation have made significant progress. However, existing methods still struggle with generating novel Human Object Interaction (HOI) from text, particularly for open-set objects. We identify three main challenges of this task: precise human-object relation reasoning, affordance parsing for any object, and detailed human interaction pose synthesis aligning description and object geometry. In this work, we propose a novel zero-shot 3D HOI generation framework without training on specific datasets, leveraging the knowledge from large-scale pre-trained models. Specifically, the human-object relations are inferred from large language models (LLMs) to initialize object properties and guide the optimization process. Then we utilize a pre-trained 2D image diffusion model to parse unseen objects and extract contact points, avoiding the limitations imposed by existing 3D asset knowledge. The initial human pose is generated by sampling multiple hypotheses through multi-view SDS based on the input text and object geometry. Finally, we introduce a detailed optimization to generate fine-grained, precise, and natural interaction, enforcing realistic 3D contact between the 3D object and the involved body parts, including hands in grasping. This is achieved by distilling human-level feedback from LLMs to capture detailed human-object relations from the text instruction. Extensive experiments validate the effectiveness of our approach compared to prior works, particularly in terms of the fine-grained nature of interactions and the ability to handle open-set 3D objects. Project page: jinluzhang.site/projects/interactanything . Figure 1 : 3D human object interaction synthesis by InteractAnything. Given a simple text description with goal interaction and any object mesh as input, our method enables the generation of diverse, natural, detailed, and novel interactions for open-set 3D objects in a zero-shot manner. The orange and green boxes of (b) indicate detailed contact poses from different views. 1 Introduction Synthesizing the interaction of humans and any given object holds great significance across diverse applications, including AR/VR, computer simulation, and animation. Traditionally, creating high-quality 3D human object interaction (HOI) assets requires expensive, lab-controlled data capture systems and significant human effort. Although some datasets [ 2 , 21 , 26 , 20 , 53 , 16 ] have collected these data, the models [ 41 , 58 , 26 , 27 , 54 , 60 ] trained on them can only generate interactions for specific object categories. These methods struggle to handle the diversity of object geometries, scales, and interactions at the same time, making text-driven zero-shot HOI synthesis a challenging and highly demanding task. Given recent advances in text-to-3D techniques [ 29 , 6 , 56 ] , e.g . , leveraging Score Distillation Sampling (SDS) [ 43 ] to generate 3D content from text, a natural question is whether this paradigm can be applied to zero-shot HOI synthesis. However, directly integrating 3D humans and objects into a scene without understanding the complex relationships between them remains challenging. Recent works [ 67 , 9 , 7 ] are proposed to generate realistic 3D HOIs. Among them, InterFusion [ 9 ] uses Neural Radiance Fields (NeRF) [ 37 ] for individual human and object representations but struggles with precise interaction area, leading to suboptimal and unrealistic HOIs. DreamHOI [ 67 ] optimizes on both mesh and NeRF representations through bidirectional rendering but faces challenges in ensuring accurate contact points and relative scales due to solely optimizing human states while keeping the given object static, leading to limited interactions. In this paper, we first identify the core challenge of zero-shot 3D HOI generation as accurately capturing human-object relations, contact areas, and global consistency, which are not typically presented in single-object or non-contact generation scenarios. These challenges prevent the generation of high-quality, geometrically precise, and textually consistent results. Furthermore, while large-scale pre-trained models, e.g . , 2D diffusion models [ 42 , 47 ] and large language models (LLMs) [ 10 , 1 , 39 ] , cannot fully capture the complexity of human object interactions (including body dynamics, expressive details, and object geometry), they can offer valuable prior knowledge of common interaction patterns based on textual descriptions. Consequently, we argue that the solution for effective generation is to leverage this diverse prior knowledge to guide the optimization of interactions. Following the above insights, we decompose the complex zero-shot 3D HOI task into these stages: 1) human-object relation reasoning and initialization, 2) open-set object contact affordance parsing, and 3) human pose synthesis aligning text and object geometry. We first leverage the LLM as a human-level feedback provider, which reasons and unravels complex semantic interaction relationships, e.g . , relative position and orientation, from simple text input. It initializes human and object states and designs a series of optimization functions to generate detailed interactions. The initial human pose is optimized by multi-view SDS loss for interaction in line with the given textual description and object geometry. To parse the 3D object affordance, we utilize initialized human-object relation and sample adaptive inpainting masks during the diffusion process to obtain the contact information. This enables the extraction of contact probabilities on the object, which are then used to segment and define the weighted dynamic contact regions. Finally, we design a coarse-to-fine optimization based on force closure [ 8 , 32 ] to generate expressive, diverse, and physically plausible interactions, allowing for more accurate and detailed contact. To summarize, our contributions are: • We propose InteractAnything for zero-shot 3D HOI generation, which effectively distills human and object interactions from LLM and pre-trained diffusion models to synthesize diverse, detailed, and novel interactions. • We design a novel open-set object affordance parsing, which generates contact maps on 3D geometry by utilizing adaptive inpainting and potential contact distribution. • Our model leverages both text and object geometry to drive human pose synthesis, enabling realistic and contextually appropriate interactions for open-set objects. Results show that our synthesized pose accurately reflects the semantic meaning of the input while respecting the spatial constraints of the object, leading to improved performance, as shown in Figure 1 . 2 Related Work 2.1 Text-driven 3D Content Generation The generation of 3D content from text has seen rapid advancements, primarily driven by diffusion-based models that build on text-to-2D image generation approaches [ 46 , 49 , 50 , 45 ] . One pioneering approach, DreamFusion [ 43 ] , introduced Score Distillation Sampling (SDS), which leverages the pre-trained 2D diffusion model to optimize 3D content through techniques like NeRF (Neural Radiance Fields) [ 37 ] and 3D Gaussian Splatting [ 24 ] . A significant area of research aims to address the complexity of 3D human generation. For example, recent methods have been proposed for generating controllable 3D human avatar [ 17 , 4 , 33 , 18 , 65 , 57 ] . However, these works focus on diverse and high-quality appearance but animate human avatars with given human poses or motion, which do not satisfy 3D HOI requirements in human pose synthesis. The proliferation of large 3D datasets has also catalyzed progress, leading to more sophisticated models like Zero-1-to-3 [ 31 ] , Zero123++ [ 51 ] , and MVDream [ 52 ] , which employ 2D diffusion models to generate