Pro-HOI : P erceptive Ro ot-guided H umanoid- O bject I nteraction
Yuhang Lin 1,2 Jiyuan Shi 1 Dewei Wang 1,3 Jipeng Kong 1,4 Yong Liu 2 Chenjia Bai 1 2 Xuelong Li 1 2 · 2026
Read Pro-HOI : P erceptive Ro ot-guided H umanoid- O bject I nteraction by Yuhang Lin 1,2 Jiyuan Shi 1 Dewei Wang 1,3 Jipeng Kong 1,4 Yong Liu 2 Chenjia Bai 1 2 Xuelong Li 1 2 (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2603-01126.txt.
Introduction
Lin and six colleagues give humanoid robots a general control interface for carrying boxes and similar loco-manipulation tasks. Pro-HOI conditions its policy on a desired root trajectory while using reference motion only as a reward. This removes fiddly reward tuning and lets planners drive navigation and manipulation together. A persistent object estimator fusing detection with a digital twin spots slippage and triggers re-grasping. Tests on a Unitree G1 show reliable long-horizon execution beating baselines.
Catalog record
| CREATOR | Yuhang Lin 1,2 Jiyuan Shi 1 Dewei Wang 1,3 Jipeng Kong 1,4 Yong Liu 2 Chenjia Bai 1 2 Xuelong Li 1 2 |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 52791 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2603.01126
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Pro-HOI : P erceptive Ro ot-guided H umanoid- O bject I nteraction Yuhang Lin 1,2 Jiyuan Shi 1 Dewei Wang 1,3 Jipeng Kong 1,4 Yong Liu 2 Chenjia Bai 1 2 Xuelong Li 1 2 Abstract Executing reliable Humanoid-Object Interaction (HOI) tasks for humanoid robots is hindered by the lack of generalized control interfaces and robust closed-loop perception mechanisms. In this work, we introduce P erceptive Ro ot-guided H umanoid- O bject I nteraction, Pro-HOI, a generalizable framework for robust humanoid loco-manipulation. First, we collect box-carrying motions that are suitable for real-world deployment and optimize penetration artifacts through a Signed Distance Field loss. Second, we propose a novel training framework that conditions the policy on a desired root-trajectory while utilizing reference motion exclusively as a reward. This design not only eliminates the need for intricate reward tuning but also establishes root trajectory as a universal interface for high-level planners, enabling simultaneous navigation and loco-manipulation. Furthermore, to ensure operational reliability, we incorporate a persistent object estimation module. By fusing real-time detection with Digital Twin, this module allows the robot to autonomously detect slippage and trigger re-grasping maneuvers. Empirical validation on a Unitree G1 robot demonstrates that Pro-HOI significantly outperforms baselines in generalization and robustness, achieving reliable long-horizon execution in complex real-world scenarios. The project page is available on project page . Fig. 1: Pro-HOI enables humanoid robots to perform generalizable and robust whole-body object interaction using exclusively onboard sensing and computing resources on a Unitree G1. a) Stylized Behaviors : Guided by motion tracking rewards, the robot exhibits stylized skills, such as running while carrying. b) Navigation & Obstacle Avoidance : The root-guided interface facilitates seamless integration with high-level planners for simultaneous carrying and obstacle avoidance. c) Autonomous Re-grasping : The robot autonomously detects an accidental drop, utilizing a digital twin module to enable state estimation and retrieval even when the object falls out of the camera’s field of view. d) Continuous Operation : The robot successfully completes over 15 consecutive carrying cycles, demonstrating long-horizon robustness. TABLE I: Capability comparison across different methods Method No External Devices Whole-Body Control Generalizable Interaction Controlable Failure Recovery OmniRetarget [ 43 ] \checkmark \checkmark \times \times \times HDMI [ 37 ] \times \checkmark \times \times \times DemoHLM \checkmark \checkmark \checkmark \times \times PhysHSI [ 34 ] \checkmark \checkmark \checkmark \times \times Falcon [ 47 ] \checkmark \checkmark \checkmark \checkmark \times OURS \checkmark \checkmark \checkmark \checkmark \checkmark I Introduction Humanoid robots, characterized by their high degrees of freedom (DoFs) and anthropomorphic configurations, are widely viewed as the ideal platform for general-purpose robots. While state-of-the-art methods have demonstrated remarkable capabilities for performing highly dynamic motions [ 16 , 39 , 8 , 48 ] and traversing complex terrain [ 49 , 18 , 33 ] , executing whole-body manipulation tasks, such as box carrying, remains a significant challenge. This difficulty arises from the rigorous demands of maintaining balance under external loads, closed-loop perception for robust interaction with objects [ 7 ] , and the extensibility to integrate with high-level planners [ 5 ] . Bridging the gap from isolated locomotion primitives to integrated perception-control frameworks is essential for achieving reliable carrying capabilities and represents a critical milestone in transitioning humanoid robots from laboratory demonstrations to practical applications in human-centric environments. Existing works on humanoid robot carrying tasks face various limitations. Traditional model-based methods often require task-specific design of optimization objectives and are limited to precise dynamics modeling [ 6 , 1 , 17 ] , making it difficult to generalize to unknown scenarios. Reinforcement Learning (RL) has emerged as a scalable alternative for discovering complex contact-rich behaviors [ 9 , 41 , 12 ] . However, end-to-end RL approaches often necessitate laborious reward engineering, requiring distinct parameter tuning for different task phases to ensure convergence [ 46 ] . Recent advancements have attempted to address naturalness and generalization by incorporating Adversarial Motion Priors (AMP) [ 25 ] and a coarse-to-fine perception scheme [ 34 ] . HDMI [ 37 ] proposes to learn interaction skills from videos and presents a unified interaction task design, yielding a unified HOI training framework. However, this method not only relies on optical motion capture systems that are difficult to migrate, but also limits the location of object interactions to the training motion. What’s more, existing approaches often lack a generalized command interface for high-level planners [ 40 ] and fail to incorporate closed-loop perception during task execution [ 43 ] . For instance, if a robot drops a box while carrying, a practical system should enable the robot to autonomously detect the failure and pick it up. These limitations render current methods impractical for real-world carrying tasks. In this work, we present Pro-HOI ( P erceptive Ro ot-guided H umanoid- O bject I nteraction), a generalizable and robust RL controller for HOI task. First, we decouple the motion tracking objective from the observation space, conditioning the policy solely on the desired root trajectory while utilizing whole-body tracking exclusively as a reward. This enables the policy to learn the dynamic correlation between root trajectory and task states, allowing for generalizable object interaction without requiring full-body reference motions. Furthermore, this design establishes the root trajectory as a universal interface, facilitating seamless integration with diverse high-level planners for executing carrying, navigation, and obstacle avoidance tasks simultaneously. Second, to address real-world uncertainties such as accidental drops, we introduce a persistent object estimation module. Combining real-time object detection with Digital Twin, this module enables the robot to detect drops and re-acquire objects autonomously, thereby substantially enhancing system reliability. Table I presents a qualitative comparison between our framework and state-of-the-art HOI methods. While early approaches like OmniRetarget [ 43 ] and HDMI [ 37 ] established the foundation for whole-body control, they are limited by either poor generalization or reliance on external devices. Recent imitation-based works, such as PhysHSI [ 34 ] and Falcon [ 47 ] , have significantly improved interaction generalizability and controllability. However, a critical gap remains: none of these baselines account for execution robustness. In contrast, our method stands out as the only holistic solution that integrates controllable, generalizable interaction with a distinct failure recovery mechanism, ensuring reliable deployment without the need for external instrumentation. To evaluate our approach, we conducted extensive experiments on the box carrying task with various practical challenges, such as continuous multi-cycle transport of boxes with diverse locations, carrying with obstacle avoidance, and autonomous re-grasping under disturbances. The results confirm that Pro-HOI not only supports seamless high-level integration but also achieves superior robustness, enabling reliable long-horizon task execution. In summary, the main contributions of this work are as follows: • We propose a root trajectory guided RL framework for HOI task. This framework achieves generalizable humanoid HOI capabilities without requiring extensive reference motions or intricate reward engineering. Furthermore, the root trajectory serves as a flexible interface that seamlessly integrates with high-level planners to enable simultaneous navigation and obstacle avoidance. • We introduce a persistent object estimation module that combines real-time object detection with Digital Twin. This module allows autonomous re-grasping of dropped objects, which significantly enhances system robustness and practicality in real-world scenarios. • We deploy the complete system on a Unitree G1 robot, with all modules running on