SHOW3D: Capturing Scenes of 3D Hands and Objects in the Wild

Patrick Rim 1,2 Kevin Harris 1 Braden Copple 1 Shangchen Han 1 Xu Xie 1 Ivan Shugurov 1 Sizhe An 1 He Wen 1 Alex Wong 2 Tomas Hodan 1 Kun He 1 1 Meta Reality Labs 2 Yale University · 2026

Read SHOW3D: Capturing Scenes of 3D Hands and Objects in the Wild by Patrick Rim 1,2 Kevin Harris 1 Braden Copple 1 Shangchen Han 1 Xu Xie 1 Ivan Shugurov 1 Sizhe An 1 He Wen 1 Alex Wong 2 Tomas Hodan 1 Kun He 1 1 Meta Reality Labs 2 Yale University (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2603-28760.txt.

Introduction

Rim and co-authors present SHOW3D, a dataset of 3D hands and objects captured in the wild. Hand-object datasets usually come from controlled studios, which limits diversity and generalization. Their marker-less multi-camera system allows nearly free movement in real settings. A lightweight back-mounted rig works with a user-worn VR headset for capture. An ego-exo tracking pipeline creates precise 3D annotations for hands and objects. The system reduces the trade-off between realism and annotation accuracy.

Catalog record

CREATORPatrick Rim 1,2 Kevin Harris 1 Braden Copple 1 Shangchen Han 1 Xu Xie 1 Ivan Shugurov 1 Sizhe An 1 He Wen 1 Alex Wong 2 Tomas Hodan 1 Kun He 1 1 Meta Reality Labs 2 Yale University
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE59024
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2603.28760

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

SHOW3D: Capturing Scenes of 3D Hands and Objects in the Wild Patrick Rim 1,2 Kevin Harris 1 Braden Copple 1 Shangchen Han 1 Xu Xie 1 Ivan Shugurov 1 Sizhe An 1 He Wen 1 Alex Wong 2 Tomas Hodan 1 Kun He 1 1 Meta Reality Labs 2 Yale University Abstract Figure 1 : SHOW3D is the first dataset of in-the-wild hand–object interactions with accurate 3D annotations as well as text descriptions. The dataset was captured with our novel mobile multi-camera rig in diverse indoor and outdoor scenes, and annotated with 3D shapes and poses with our multi-view pipeline. Overlays show 3D annotations projected to egocentric images (hands in red and blue, object in green). 1 1 footnotetext: Work done during an internship at Meta. Abstract Accurate 3D understanding of human hands and objects during manipulation remains a significant challenge for egocentric computer vision. Existing hand–object interaction datasets are predominantly captured in controlled studio settings, which limits both environmental diversity and the ability of models trained on such data to generalize to real-world scenarios. To address this challenge, we introduce a novel marker-less multi-camera system that allows for nearly unconstrained mobility in genuinely in-the-wild conditions, while still having the ability to generate precise 3D annotations of hands and objects. The capture system consists of a lightweight, back-mounted, multi-camera rig that is synchronized and calibrated with a user-worn VR headset. For 3D ground-truth annotation of hands and objects, we develop an ego-exo tracking pipeline and rigorously evaluate its quality. Finally, we present SHOW3D, the first large-scale dataset with 3D annotations that show hands interacting with objects in diverse real-world environments, including outdoor settings. Our approach significantly reduces the fundamental trade-off between environmental realism and accuracy of 3D annotations, which we validate with experiments on several downstream tasks. show3d-dataset.github.io 1 Introduction Figure 2 : Our mobile multi-camera capture rig. Left: Annotated hardware layout showing five MoCap cameras (red), and eight exocentric monochrome cameras mounted in a half-dome configuration (green), and two egocentric monochrome cameras on the Meta Quest 3 headset (blue). The MoCap cameras are only used for headset pose tracking, by tracking optical markers rigidly attached to the headset. The ten total exocentric and egocentric monochrome cameras are used for marker-free annotation of 3D hand and object poses. Right: The rig in use during in-the-wild capture sessions, demonstrating its lightweight (about eight kilograms) and wearable design that allows natural interaction while maintaining stable, synchronized multi-view coverage under mostly unconstrained motion. Accurate 3D tracking of human hands and their interactions with objects in everyday environments is essential for egocentric perception systems. In augmented and virtual reality (AR/VR), understanding of hands and manipulated objects enables natural and expressive user interactions without reliance on external input devices or constrained input spaces. In robotics and teleoperation, this ability is critical for advising and transferring dexterous manipulation skills. Despite recent advances, existing hand–object perception models 10 ; 11 ; 29 ; 40 ; 38 ; 30 often struggle when deployed outside controlled and indoor environments. There is therefore a pressing need for data collection systems that can produce realistic, unconstrained, and annotated captures of hand–object interactions in the wild . However, building such systems is highly challenging due to a fundamental trade-off. On one hand, the need for realistic hand/object appearance, environmental diversity, and user mobility limits the application of external sensors, e.g . , cameras, IMUs, ToF, and motion capture systems 15 ; 34 ; 37 ; 1 ; 32 . On the other hand, the availability and quality of 3D ground truth are bounded by sensor fidelity and coverage. Existing hand–object interaction datasets often fall at one of two extremes in this trade-off. Recent datasets, including GigaHands 8 , HOT3D 3 , and ARCTIC 7 , are captured under controlled conditions, typically within indoor studio spaces equipped with motion capture systems or stationary multi-camera arrays. While these setups can offer high 3D accuracy, they inherently limit environmental diversity and, in the case of marker-based motion capture, hinder visual realism by altering natural hand and object appearance with reflective markers. At the other extreme, datasets with significant environmental variation, such as Ego-Exo4D 9 and Nymeria 20 , lack dense or accurate 3D ground truth for hands and objects due to their limited sensor setup. In this work, we break the trade-off between realism and ground-truth quality by designing a novel multi-camera capture system inspired by prior work 10 but optimized for mobility. Our lightweight backpack-style rig acquires accurate 3D ground truth while enabling users to move freely, allowing for capture of realistic interactions in diverse environments, including the outdoors (Figure 1 ). The system provides ten fisheye monochrome cameras synchronized at 60Hz, including eight exocentric ones attached on the rig and two egocentric ones from a user-worn Meta Quest 3 headset (Figure 2 ). To generate accurate ground-truth 3D poses of hands and objects, we develop a novel ego-exo annotation pipeline that leverages all ten camera views. For hand pose annotation, we use state-of-the-art models for keypoint detection, and fuse them via robust triangulation with RANSAC. For object pose annotation, we develop a multi-view extension of recent DINOv2-based methods for 2D detection 25 , pose estimation 28 , and refinement 24 . Both components produce their own confidence estimates, which allow for automated filtering when building training and evaluation datasets. We assess the quality of both our hand and object pose annotations via evaluation against independent references, demonstrating sub-centimeter accuracy. We use our novel capture system and annotation pipeline to collect SHOW3D , a large-scale dataset of hand–object interactions with dense and accurate 3D annotations, and the first such dataset that is captured across diverse real-world environments. Besides images from ten egocentric and exocentric cameras and 3D pose annotations of hands and objects, the dataset includes fitted 3D meshes, 2D segmentation masks, contact regions, and text descriptions. Cross-dataset experiments show that models for 3D hand pose estimation and hand–object interaction field estimation trained on existing studio datasets generalize poorly to in-the-wild data, while SHOW3D-trained models maintain strong performance on studio benchmarks, confirming that our dataset captures broader environmental diversity. Furthermore, we demonstrate the utility of our text annotations through object trajectory forecasting experiments, where text-conditioned models significantly outperform baselines relying only on past trajectory, highlighting the value of semantic context for understanding hand–object interactions. In summary, we make the following contributions: 1. A mobile data capture system: The first capture setup for recording and automatically generating 3D annotations of in-the-wild hand–object interactions. Our setup consists of a mobile multi-camera capture rig that enables the acquisition of accurate ground-truth hand and object pose annotations from multi-view imagery. 2. Ego-exo 3D ground truth: We build a pipeline to annotate 3D hand and object poses using ego-exo fusion, and automatically filter the resulting annotations using reliable confidence estimates. We also rigorously evaluate the quality of our 3D ground truth for hands and objects. 3. Large and diverse dataset: Our dataset SHOW3D comprises 4.3M frames of synchronized multi-view images with comprehensive annotations including 3D hand poses and MANO meshes, 6DoF object poses with 3D models, and descriptive action-level text captions. 4. Demonstrated dataset utility: Empirical validation via cross-dataset generalization experiments shows that models trained on SHOW3D transfer better

Read the complete text