Learning human-to-robot handovers through 3D scene reconstruction

Yuekun Wu, Yik Lung Pang, Andrea Cavallaro, Changjae Oh · 2025

Read Learning human-to-robot handovers through 3D scene reconstruction by Yuekun Wu, Yik Lung Pang, Andrea Cavallaro, Changjae Oh (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2507-08726.txt.

Introduction

Yuekun Wu and co-authors teach a robot to take objects from a human hand without ever practicing on a real robot. Their 2025 paper deals with human-to-robot handovers, where real trial data are costly to gather and simulators look fake. Their learner rebuilds handover scenes from sparse photos with Gaussian splatting, a sharp 3D rendering trick. Inside those rebuilt scenes, a gripper camera's moves become robot demonstrations with image-action pairs.

Training used a set of household objects only. The taught policy then ran live in the real world. Tests in rebuilt and real scenes back the method, the team reports.

Catalog record

CREATORYuekun Wu, Yik Lung Pang, Andrea Cavallaro, Changjae Oh
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE35 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2507-08726

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Learning human-to-robot handovers through 3D scene reconstruction Yuekun Wu, Yik Lung Pang, Andrea Cavallaro, Changjae Oh Abstract Learning robot manipulation policies from raw, real-world image data requires a large number of robot-action trials in the physical environment. Although training using simulations offers a cost-effective alternative, the visual domain gap between simulation and robot workspace remains a major limitation. Gaussian Splatting visual reconstruction methods have recently provided new directions for robot manipulation by generating realistic environments. In this paper, we propose the first method for learning supervised-based robot handovers solely from RGB images without the need of real-robot training or real-robot data collection. The proposed policy learner, Human-to-Robot Handover using Sparse-View Gaussian Splatting (H2RH-SGS), leverages sparse-view Gaussian Splatting reconstruction of human-to-robot handover scenes to generate robot demonstrations containing image-action pairs captured with a camera mounted on the robot gripper. As a result, the simulated camera pose changes in the reconstructed scene can be directly translated into gripper pose changes. We train a robot policy on demonstrations collected with 16 household objects and directly deploy this policy in the real environment. Experiments in both Gaussian Splatting reconstructed scene and real-world human-to-robot handover experiments demonstrate that H2RH-SGS serves as a new and effective representation for the human-to-robot handover task. Fig. 1: Rendered images from a Gaussian Splatting (GS) scene (top) and hand-eye images from a camera in a human-to-robot environment (bottom). In the 3DGS scene, the images are sequentially rendered according to the 6D pose of a virtual hand-eye camera view. In the real robot environment where we perform human-to-robot handovers, our method takes a single image captured by the hand-eye camera on the gripper. Since the movement of the virtual camera in 3DGS scenes can mimic the movement of a hand-eye robot arm in the real robot environment, we train a human-to-robot handover policy from 3DGS scenes and directly deploy this model to the physical human-to-robot environment. t_{0} ( t^{\prime}_{0} ) to t_{5} ( t^{\prime}_{5} ) indicate the sampled time steps when the camera pose is moving from the initial pose to the pre-grasp pose. I Introduction Training robots to interact with humans in real-world environments presents significant time and safety challenges [ 1 ] . While simulations provide a cost-effective alternative to collect training data, they suffer from the domain gap, particularly due to discrepancies in visual appearances and dynamics between simulated and real-world environments [ 2 ] . To mitigate the domain gap, several approaches focus on improving the rendering of simulated environments to make them more realistic. For example, domain randomization [ 3 ] alters environmental variables, helping the model adapt to diverse real-world conditions. Domain adaptation [ 4 , 5 ] refines the learned policies to better align with real-world data, often using techniques like adversarial learning [ 6 ] or feature alignment [ 7 ] . Generative Adversarial Networks are used in domain adaptation to improve the realism of simulated images to facilitate the transfer of skills from simulation to real-world tasks [ 8 , 9 , 10 ] . Compared to GAN-based methods, diffusion-based rendering [ 11 , 12 ] generates high-quality images with better generalization, lower data requirements, and more advanced conditioning techniques, such as layout-to-image, text-to-image, and image-to-image generation. However, these methods often require large amounts of data, precise camera calibration, or computationally intensive training [ 13 ] . Additionally, these methods may be restricted to translational motion without rotational adjustments during reaching tasks. In more complex scenarios, these limitations impose additional burdens on learning, which could lead to failure [ 14 ] . Beyond rendering-based solutions, other methods have used images from the real environment to bridge the visual domain gap. Radiance field-based reconstruction methods [ 15 ] capture high-fidelity 3D scene representations, enabling realistic simulation environments for robot training. A sim-to-real method with Gaussian Splatting (GS) [ 14 ] leverages 3D Gaussian Splatting (3DGS) to construct objects within a simulator, enabling a sim-to-real reinforcement learning framework to address the domain gap. SLAM-based methods [ 16 ] construct 3D environment representations through motion and multi-view observations, enhancing spatial understanding and sim-to-real transfer. Additionally, OHPL [ 17 ] learns a policy for robotic reaching movements using a single image captured from the hand-eye camera in the real environment without requiring calibration information or sim-to-real adaptation. However, these methods are designed for robot manipulation without human interaction. Hence, they do guarantee human safety during robotic grasping. While several works [ 18 , 19 , 20 ] proposed simulation environments for human-to-robot handovers using 3D object models and recorded or generated hand and object motion, simulators do not render realistic images of the environment, thus limiting the input modality of the trained models to point cloud only. Reconstructing 3D environments for robot training often relies on extensive multi-view datasets, which are costly, time-intensive, and potentially unsafe to acquire. This challenge has spurred interest in efficient learning from sparse camera views, with recent advances in sparse-view GS. DNGaussian [ 21 ] , CoherentGS [ 22 ] , and FSGS [ 23 ] address the challenges of sparse-view reconstruction by leveraging monocular depth estimators to infer depth maps from limited viewpoints. Other methods (e.g. [ 24 ] ) employ multi-view stereo techniques to generate novel-view images as ground truth. These approaches offer a potential solution to the challenge of training robot control models without relying on large-scale multi-camera datasets. However, point clouds derived from RGB images without depth ground-truth often suffer from inaccurate scale estimation. To address the above-mentioned challenges and facilitate the robot’s ability to reach and grasp objects without real training data, we present a novel framework based on hand-eye images changed by a series of discrete robot actions with GS simulations (see Fig. 1 ). We hypothesize that, when appropriately formulated, a model trained with GS can effectively learn the necessary skills to perform reaching, grasping and handover tasks within a robot’s workspace. To this end, we propose Human-to-Robot Handover using Sparse-View Gaussian Splatting (H2RH-SGS) for human-to-robot handover tasks with reaching and grasping. Unlike existing approaches, H2RH-SGS eliminates the need for real-robot training, human demonstrations, and sim-to-real adaptation. H2RH-SGS derives the robot’s actions directly from a single image captured by the robot’s hand-eye camera along with the corresponding object and hand masks, enabling handover control through supervised learning. Therefore, the proposed framework effectively bridges the domain gap and addresses the challenges of costly and unsafe data collection with high-fidelity 3DGS for robot action training tasks 1 1 1 Code and videos: https://qm-ipalab.github.io/H2RH-SGS/ . (a) Gaussian Splatting reconstruction (b) Generating demonstrations (c) Handover demonstrations (d) Handover policy training Fig. 2: Overview of our method. (a) Given sparse-view RGB-D handover images, we reconstruct a 3D scene using Gaussian Splatting (GS) and then estimate grasp poses from the object and hand point clouds extracted from the GS scene. (b) We then use the GS scene and grasp pose to generate the gripper’s trajectory toward the pre-grasp pose and the hand-eye image at each sampled pose. (c) Each trajectory becomes a handover demonstration dataset that includes hand-eye images, object and hand masks, transformations of the gripper pose, and pre-grasp pose labels. (d) The dataset is used to train a handover policy. For inference, only the hand-eye RGB image and

Read the complete text