SceneBot: Contact-Prompted General Humanoid Whole Body Tracking with Scene-Interaction
Sirui ChenStanford, Amazon FARUnited States [email protected] Shibo ZhaoAmazon FARUnited States [email protected] Zhen WuAmazon FARUnited States [email protected] Jiaman LiAmazon FARUnited States [email protected], Guanya ShiCMU, Amazon FARUnited States [email protected], C. Karen LiuStanford, Amazon FARUnited States [email protected] · 2026
Read SceneBot: Contact-Prompted General Humanoid Whole Body Tracking with Scene-Interaction by Sirui ChenStanford, Amazon FARUnited States [email protected] Shibo ZhaoAmazon FARUnited States [email protected] Zhen WuAmazon FARUnited States [email protected] Jiaman LiAmazon FARUnited States [email protected], Guanya ShiCMU, Amazon FARUnited States [email protected], C. Karen LiuStanford, Amazon FARUnited States [email protected] (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2606-27581.txt.
Introduction
Humanoid robots learn control through reinforcement learning. That means the controller improves by trial and error. Such robots move well in open space. They falter when tasks involve touching objects or crossing rough ground. Sirui Chen and five colleagues present SceneBot in 2026. SceneBot is a single policy for free movement, uneven terrain, and whole-body manipulation.
The policy reads reference motions plus per-link contact labels. These labels state which body parts should touch the world at each moment. Labelled interaction data barely exists, though. So Chen's team rebuilt scenes after the fact from retargeted human motion. They inferred which objects and surfaces each motion involved. Training used seven and a half hours of this rebuilt data. The policy then generalised to motions and settings it had never seen. In one test it carried a box upstairs. That task mixes climbing with holding.
Catalog record
| CREATOR | Sirui ChenStanford, Amazon FARUnited States [email protected] Shibo ZhaoAmazon FARUnited States [email protected] Zhen WuAmazon FARUnited States [email protected] Jiaman LiAmazon FARUnited States [email protected], Guanya ShiCMU, Amazon FARUnited States [email protected], C. Karen LiuStanford, Amazon FARUnited States [email protected] |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 32 KB |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2606.27581
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
SceneBot: Contact-Prompted General Humanoid Whole Body Tracking with Scene-Interaction Sirui ChenStanford, Amazon FARUnited States [email protected] Shibo ZhaoAmazon FARUnited States [email protected] Zhen WuAmazon FARUnited States [email protected] Jiaman LiAmazon FARUnited States [email protected], Guanya ShiCMU, Amazon FARUnited States [email protected], C. Karen LiuStanford, Amazon FARUnited States [email protected] Abstract Current humanoid reinforcement-learning policies excel at free-space motions but struggle with contact-rich tasks, as pure kinematic tracking cannot resolve the physical ambiguities of interacting with objects and uneven terrain. To address this, we introduce SceneBot, a unified motion-tracking framework capable of handling freespace locomotion, terrain traversal, and whole-body manipulation. SceneBot sconditions a single policy on both reference motions and per-link contact labels, explicitly defining expected environmental interactions. To overcome the lack of annotated interaction data, we propose a hindsight scene reconstruction approach that infers scene-interaction graphs from retargeted human motion. Trained on 7.5 hours of this reconstructed, contact-rich data, SceneBot successfully generalizes to unseen motions and environments. Our results demonstrate that SceneBot is the first general framework to seamlessly unify free-space and contact-rich behaviors—executing complex, long-horizon tasks like carrying a box upstairs and establishing contact conditioning as a powerful interface for humanoid control. All code and data will be open-sourced. More demos and informations are available at: https://ericcsr.github.io/scenebot/ Figure 1: SceneBot enables a single motion tracking policy to accurately achieve free-form locomotion, uneven terrain traversal, and object manipulation. Keywords: Humanoid, Loco-manipulation, Learning from Human Data 1 Introduction Physical intelligence requires robots to intentionally leverage contacts with the world for support, manipulation and locomotion. Consider a household robot picking up a basket, climbing stairs while maintaining balance, and placing the basket near a washing machine. Accomplishing such a contact-rich task requires coordinated interaction with both objects and terrain over long horizons. Recent progress in humanoid control has been driven by reinforcement-learning policies trained to track large collections of reference motions. These general motion trackers can execute diverse free-space behaviors such as walking, running, dancing, and kicking with a single policy. However, their success has largely been limited to environments where the reference motion can be followed without reasoning about external contact. Once the robot interacts with large objects or non-flat terrain, kinematics alone becomes ambiguous. Moving both wrists to the sides of a box may correspond to reaching toward the box, lightly touching it, or applying sufficient force to lift it. Similarly, placing a foot near the edge of a stair does not guarantee that the robot generates sufficient ground reaction force to step upward. Thus, successful task execution depends not only on reaching desired poses, but also on knowing which body parts should establish, maintain, and utilize contact forces with the environment. We introduce SceneBot, a contact-conditioning motion tracking framework that enables a single policy to track humanoid free-form locomotion, terrain traversal (e.g. stairs), and whole-body manipulation. SceneBot conditions a general, whole-body policy on both reference motion and per-link contact labels . The reference motion specifies desired kinematic behavior, while contact labels explicitly indicate which body links should expect and exploit contact forces from different parts of the scene. Compared to existing motion trackers, SceneBot asks the high-level decision maker to provide one additional command: which robot links should actively establish contact with the scene. However, these labels are defined only on the robot body, not on scene objects or terrain. This design keeps SceneBot as a low-level motion tracking policy, with a minimal extension for scene interaction without turning into a vision-based task policy. Training such a contact-aware controller presents a major data challenge: it requires motion trajectories paired with corresponding object geometry, terrain geometry, and contact annotations. Unfortunately, large-scale datasets containing both motion and scene interaction information are scarce, and scalable scene-aware retargeting remains difficult. SceneBot addresses this data bottleneck through hindsight scene reconstruction. Given a retargeted robot motion, we infer a scene-interaction graph that identifies which robot links should contact terrain or movable objects over time. We then reconstruct plausible terrain and object assets consistent with those inferred contacts. This turns ordinary human motion data into contact-rich robot-scene interaction data suitable for training. We show that Scenebot is, to the best of our knowledge, the first general motion tracking framework capable of handling free-space locomotion, uneven terrain traversal, and whole-body object manipulation within a single policy. Scenebot generalizes to unseen motions and unseen environments, while maintaining performance comparable to state-of-the-art motion trackers in free-space settings. We further demonstrate long-horizon tasks requiring simultaneous terrain and object interaction, such as picking up a large box, carrying it, and walking upstairs. Our results show that contact conditioning provides an effective interface for extending general humanoid motion tracking from free-space motion to contact-rich scene interaction. 2 Related Works 2.1 General motion tracking Recent RL-based motion tracking has evolved from task-specific policies [ 12 , 18 , 8 ] to unified controllers capable of tracking diverse trajectories [ 15 , 5 , 34 , 28 , 14 , 4 ] . Supported by large-scale datasets [ 15 , 16 ] , scalable retargeting [ 20 , 1 ] , and GPU simulators [ 17 , 27 ] , these generalized policies serve as robust low-level controllers for downstream tasks like teleoperation and Vision-Language-Action architectures [ 28 , 29 , 15 , 4 ] . However, a critical gap remains: current pipelines primarily target free-space movement. Existing datasets and retargeting methods fundamentally lack the scene-awareness required for complex human-environment interactions. 2.2 Terrain aware whole body control Mastering complex terrain traversal has shifted from model-based footstep planning and MPC [ 6 , 10 ] to RL policies utilizing root velocity commands and gait-based reward shaping [ 2 , 32 , 30 ] . Agile maneuvers, such as parkour, have been achieved through scene-aware retargeting [ 26 ] , policy distillation [ 23 ] , and AMP-style rewards [ 19 , 37 ] . Despite these advances, most terrain traversal policies rely exclusively on root velocity control, severely limiting their utility as general-purpose whole-body controllers for loco-manipulation. While recent works [ 25 , 33 ] attempt to bridge this gap by co-training controllers with terrain-aware kinematic generators, their reliance on kinematically generated data significantly constrains dexterous upper-body movement. 2.3 Humanoid object interaction Beyond locomotion, humanoid object manipulation has largely been confined to single-handed, lightweight pick-and-place tasks [ 3 , 15 , 21 ] . For bimanual grasping of heavy objects, existing impedance controllers [ 4 ] still depend on high-level policies or human teleoperators to manually regulate force. Although recent methods leverage human-object datasets [ 11 , 13 ] to achieve single-trajectory tracking for large objects [ 22 , 26 , 24 ] , the field lacks a general-purpose, whole-body controller capable of zero-shot deployment for complex manipulation. To address this, our work presents a generalized motion tracking controller designed for zero-shot deployment, seamlessly enabling both unseen terrain traversal and robust object interaction. Tracking Single Motion Tracking Diverse Motion Terrain Interaction Object Interaction BeyondMimic [ 12 ] ✓ ✗ ✗ ✗ OmniRetarget [ 26 ] ✓ ✗ ✓ ✓ HDMI [ 22 ] ✓ ✗ ✗ ✓ ResMimic [ 36 ] ✓ ✗ ✗ ✓ SONIC [ 15 ] ✓ ✓ ✗ ✗ CHIP [ 4 ] ✓ ✓ ✗ ✓ TWIST [ 28 ] ✓ ✓ ✗ ✗ Any2Track [