Reconstructing In-the-Wild Open-Vocabulary Human-Object Interactions

Boran Wen Dingbang Huang Zichen Zhang Jiahong Zhou Jianbin Deng, Jingyu Gong Yulong Chen Lizhuang Ma Yong-Lu Li · 2025

Read Reconstructing In-the-Wild Open-Vocabulary Human-Object Interactions by Boran Wen Dingbang Huang Zichen Zhang Jiahong Zhou Jianbin Deng, Jingyu Gong Yulong Chen Lizhuang Ma Yong-Lu Li (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2503-15898.txt.

Introduction

Boran Wen and co-authors build tools to rebuild three-dimensional scenes of people handling objects from single photos. Their paper appeared in 2025. Most past work was trained on indoor scenes, since three-dimensional data were scarce and objects were few. New methods that rebuild objects from one image now make wider data possible. The team labeled more than 2,500 such scenes from existing photo sets.

The result is Open3DHOI, the first open set of this kind covering real-world scenes and many object types. The team also built a program that fits people and objects together and learns where hands touch. They propose new test tasks for this kind of scene reading. Data and code are shared openly, so other groups can build on the set and test their own methods.

Catalog record

CREATORBoran Wen Dingbang Huang Zichen Zhang Jiahong Zhou Jianbin Deng, Jingyu Gong Yulong Chen Lizhuang Ma Yong-Lu Li
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE49 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2503-15898

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Reconstructing In-the-Wild Open-Vocabulary Human-Object Interactions Boran Wen Dingbang Huang Zichen Zhang Jiahong Zhou Jianbin Deng, Jingyu Gong Yulong Chen Lizhuang Ma Yong-Lu Li Abstract Reconstructing human-object interactions (HOI) from single images is fundamental in computer vision. Existing methods are primarily trained and tested on indoor scenes due to the lack of 3D data, particularly constrained by the object variety, making it challenging to generalize to real-world scenes with a wide range of objects. The limitations of previous 3D HOI datasets were primarily due to the difficulty in acquiring 3D object assets. However, with the development of 3D reconstruction from single images, recently it has become possible to reconstruct various objects from 2D HOI images. We therefore propose a pipeline for annotating fine-grained 3D humans, objects, and their interactions from single images. We annotated 2.5k+ 3D HOI assets from existing 2D HOI datasets and built the first open-vocabulary in-the-wild 3D HOI dataset Open3DHOI , to serve as a future test set. Moreover, we design a novel Gaussian-HOI optimizer, which efficiently reconstructs the spatial interactions between humans and objects while learning the contact regions. Besides the 3D HOI reconstruction, we also propose several new tasks for 3D HOI understanding to pave the way for future work. Data and code will be publicly available at https://wenboran2002.github.io/3dhoi/ . 1 Introduction Human-Object Interaction (HOI) is an important area in action understanding, with numerous datasets and methods proposed. In the 2D HOI domain, large-scale image datasets such as HICO-DET [ 52 ] and HAKE [ 21 ] have been introduced. For 3D HOI, datasets like BEHAVE [ 2 ] and InterCap [ 12 ] have also been proposed to study human interactions with objects in 3D. Despite achieving promising results in open-vocabulary and in-the-wild scenarios within 2D HOI, the 3D HOI field faces challenges in generalizing existing methods to real-world images due to dataset limitations and the lack of 3D open-world HOI data. In 3D HOI, many datasets have been introduced. Datasets such as BEHAVE [ 2 ] , InterCap [ 12 ] , ImHOI [ 55 ] , and PROX-S [ 56 ] provide multi-view RGBD sequences and 3D annotations in indoor scenes. Though datasets like WildHOI and 3DIR [ 13 , 50 ] are constructed from in-the-wild images, they contain limited object categories and unreal CAD objects. To better understand 3D HOIs and apply them to the real world, we need to collect more realistic and diverse data on interactions with objects. Thus, in this work, we propose a novel 3D HOI annotation method for real-world images of any objects and interactions. In detail, we built our annotation pipeline on two bases: 1) Existing 2D HOI datasets provide rich 2D annotations, including bounding boxes, and a wide variety of objects and actions. This diversity creates the potential for reconstructing 3D assets from 2D HOI images. 2) The development of existing image-based 3D object/human reconstruction techniques. We selected images with contact interactions from existing 2D HOI datasets, e.g . , HAKE [ 21 ] and SWIG-HOI [ 42 ] . Next, we used InstantMesh [ 46 ] and OSX [ 24 ] to reconstruct the objects and the human body respectively, and designed an algorithm for automatically reconstructing rough 3D interactions. Furthermore, we developed two annotation tools: one for filtering the reconstruction quality and the other for annotating 3D spatial positions. We manually annotated over 2.5k+ images to create an open-vocabulary, in-the-wild HOI dataset, as the test set for future 3D HOI studies and designed tasks and metrics to evaluate their performance. It consists of 370 3D human-object pairs, 2,561 objects in 133 categories, and 3,671 interactions in 120 categories. Figure 1 : We aim to reconstruct 3D HOIs from arbitrary open-world images. We propose a pipeline for annotating fine-grained reconstructions to build a dataset. Additionally, we introduce a new optimizer suitable for reconstructing arbitrary objects. Given the new dataset, we also proposed a training-free algorithm for reconstructing 3D HOIs from monocular images. On one hand, previous training-free algorithms do not require specific object categories or templates, but their optimization performance is generally limited, and they rely on manual annotations. On the other hand, previous training-based methods perform well for specific object categories and scenes but struggle to generalize to open-world environments. To this end, we leveraged the 3D Gaussian Splatting model to propose a novel Gaussian-HOI optimizer to improve the reconstruction quality. It utilizes Gaussian rendering capabilities to ensure that the reconstructed 3D assets are aligned with the image from the target view and takes advantage of the opacity attribute of Gaussians to identify contact regions, which makes the optimization of 3D interaction relationships more effective. Overall, our contributions are: 1) We utilized SOTA 3D reconstruction tools to develop a 3D HOI annotation method. 2) We built a new and extensive 3D HOI dataset Open3DHOI consisting of 2.5k+ images with rich 2D and 3D annotations. 3) We designed a 3D HOI optimizer based on 3D Gaussian Splatting to reconstruct the spatial interactions between humans and objects from single images. 2 Related Works 2.1 HOI Benchmarks The development of 2D HOI benchmarks [ 21 , 6 , 8 , 42 , 57 , 30 , 19 , 22 , 20 ] has made our 3D HOI reconstruction approach possible. Datasets like HICO-DET [ 6 ] and HAKE [ 21 ] provide annotations for 80 object categories and 117 action categories. Additionally, open-vocabulary datasets such as SWIG-HOI [ 42 ] include annotations for over 1,000 object categories. In contrast, 3D HOI datasets contain significantly fewer action and object categories, and most are recorded in fixed indoor environments. BEHAVE [ 2 ] , as the earliest one, introduced a method for obtaining accurate 3D HOI annotations from multi-view videos, providing interaction data for 20 common objects. InterCap [ 12 ] further built upon it by offering more detailed hand interaction information. Recently, several benchmarks [ 50 , 13 ] for reconstructing 3D HOIs from real-world images have been proposed. However, they have notable limitations. Their object categories are focused on a few common types—such as balls, skateboards, and bicycles—and the number of instances is limited, with all objects derived from fixed 3D CAD models. 2.2 3D Reconstruction 3D reconstruction has seen rapid advancements recently, both in humans and objects. After SMPL [ 32 ] , parametric human body modeling has rapidly evolved. Currently, the SMPL-X model [ 35 ] , which includes detailed hand and facial expression modeling, is widely used in the field [ 24 , 5 , 53 , 23 ] . We utilized a state-of-the-art one-stage model [ 24 ] for our human body reconstruction. What’s more, image-to-3D has emerged as a rapidly advancing area in 3D vision recently. From SDS loss optimization methods [ 36 , 38 ] that leverage 2D diffusion priors to Multi-view Diffusion Models [ 41 , 27 , 31 , 34 ] and Large Reconstruction Models [ 9 , 18 , 46 ] based on large-scale data, the quality and efficiency of 3D generation from a single image have seen significant improvements. To generate our 3D HOI dataset, we need to reconstruct a large volume of image data, requiring a balance between the quality and efficiency of existing 3D

Read the complete text