Imagine2Real: Towards Zero-shot Humanoid-Object Interaction via Video Generative Priors
2026
Read Imagine2Real: Towards Zero-shot Humanoid-Object Interaction via Video Generative Priors (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2605-22272.txt.
Introduction
Chen and colleagues propose Imagine2Real, a framework that lets a humanoid robot interact with objects without prior 3D training data for the task. Video generation priors supply the motion knowledge, but earlier methods stumbled on geometry mismatch and complex retargeting. The team instead tracks robot and object motion as unified four-dimensional point paths, following only sparse key points. A behaviour foundation model guides the search, and the system deploys with no task-specific training in motion capture tests.
Catalog record
| YEAR | 2026 |
|---|---|
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 40016 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2605.22272
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Imagine2Real: Towards Zero-shot Humanoid-Object Interaction via Video Generative Priors Abstract Whole-body Humanoid-Object Interaction (HOI) is bottlenecked by the scarcity of high-fidelity 3D data. While video generative priors offer a promising alternative, existing methods suffer from Representation Misalignment due to their reliance on geometric priors (e.g., explicit CAD models), and Retargeting Complexity arising from intensive morphing and morphological mismatch. We propose Imagine2Real, a zero-shot HOI framework for flexible, geometry-free interaction. To resolve misalignment, we formulate robot and object motions as unified 4D point trajectories. To overcome retargeting complexity, our Keypoints Tracker tracks only sparse critical points (base, hands, and object), entirely bypassing the error-amplifying retargeting process. To maintain natural gaits despite these sparse signals, we utilize the latent space of a Behavior Foundation Model (BFM) as the tracker’s search domain. Using a progressive training strategy, Imagine2Real learns robust behaviors with simple tracking rewards, enabling zero-shot physical deployment within a motion capture(mocap) system. I m a g i n e 2 R e a l : Towards Zero-shot Humanoid-Object Interaction via Video Generative Priors Jiahe Chen 1,2∗ Zirui Wang 1,2∗ Feiyu Jia 2 Xiao Chen 3,2 Xiaojie Niu 2 Weishuai Zeng 3,2 Tianfan Xue 3 Xiaowei Zhou 1 Jiangmiao Pang 2† Jingbo Wang 2† 1 Zhejiang University 2 Shanghai AI Laboratory 3 The Chinese University of Hong Kong \ast : co-first authors \dagger : co-corresponding authors Figure 1: The Imagine2Real zero-shot deployment loop. Given (1) an image and text instruction, we (2) synthesize an interaction video, (3) extract 4D keypoints trajectories, and (4) deploy the policy within a mocap system, turning imagined interactions into physical reality. 1 Introduction Whole-body Humanoid-Object Interaction (HOI) is critical for deploying humanoid robots in unstructured environments such as households and factories. While proprioceptive control has matured significantly 8 ; 27 ; 32 , extending these capabilities to complex interactions remains difficult. The primary bottleneck is the lack of high-fidelity 3D interaction data for humanoid robots, which prevents the learning of robust and versatile interaction policies. This data scarcity imposes a generalization ceiling on current HOI methodologies, where task versatility remains strictly limited by the available training samples. Scaling-based models such as VLA 22 ; 72 ; 19 pursue broad generalization through end-to-end training but are constrained by the lack of massive, humanoid-specific datasets. In contrast, approaches incorporating AMP 41 ; 51 utilize sparse, high-quality data to learn specialized behaviors, yet these methods often suffer from mode collapse 45 , which restricts them to specific task categories and necessitates manual, task-specific reward engineering. Motion tracking methods 70 ; 55 attempt to bypass data scarcity by mimicking single trajectories, but these are essentially limited to rigid motion replay and lack the autonomy needed for novel scenarios. To address the shortage of 3D data, recent research employs internet-scale video generation models to provide visual and physical priors for downstream motion trackers. While this pipeline has shown promise in motion tracking 36 , applying it to HOI introduces two major challenges. 1) Representation Misalignment. Extracting 3D motion from 2D videos typically requires strong geometric priors (e.g., explicit object CAD models), which are seldom available in real-world settings. Furthermore, existing methods lack a unified way to jointly estimate robot-object motions. Estimating them independently through mature tools(e.g., GVHMR 47 for robot and FoundationPose 54 for object) introduces spatial and depth discrepancies, leading to low-quality reference motions and tracking failures. Consequently, many works resort to manually designing 55 or heuristically generating 52 methods to rigidly attach objects to robot motions, which still requires CAD models and distorts actual object motion. 2) Retargeting Complexity . Most motion tracking models rely on dense, low-level signals like joint positions. To obtain these from human videos, the extracted motion must be explicitly retargeted to a specific humanoid embodiment. In HOI scenarios, this retargeting becomes exceedingly complex: the introduction of object geometry requires intensive, time-consuming morphing 60 ; 17 to jointly align the robot and object meshes, which inherently amplifies estimation errors from the video source. Avoiding this complex retargeting by tracking sparse keypoints is a promising alternative, but existing methods often struggle with quality. VisualMimic 63 adopts 5-keypoints tracking but fails to achieve natural gaits due to the absence of joint-level information. HEAD 7 employs 3-keypoints tracking and relies on AMP to ensure natural locomotion; however, this reliance restricts its scalability from navigation to interaction tasks. In this paper, we propose a zero-shot HOI framework that uses unified 3D point trajectories and a Keypoints Tracker based on the Behavior Foundation Model (BFM) 25 ; 66 ; 32 to achieve flexible, geometry-free interaction. To resolve representation misalignment , we extract robot and object motions using a unified 3D point tracker, representing both as 4D point trajectories. This shared representation avoids the need of geometric priors and ensures motions are aligned within a common frame. To address retargeting complexity , our tracker follows only three interaction-critical points: the robot base and hands. By focusing on this minimal set, the framework entirely bypasses the complex, error-amplifying retargeting process. To achieve these capabilities despite scarce HOI data, we employ a three-stage progressive training strategy: (1) a whole-body tracking BFM on large-scale proprioceptive motions, (2) a keypoints tracker on loco-manipulation data, and (3) an interaction adaptor on specific HOI data. This approach ensures each stage benefits from the most relevant supervision at scale. By leveraging the BFM, sparse keypoints can still produce stable gaits, allowing the final stages to use simple tracking rewards instead of the complex machinery 55 ; 70 ; 57 typically required for learning from scratch. This work makes three primary contributions. First, we introduce a unified 4D point trajectory representation for HOI that removes the dependency on geometric priors and resolves motion misalignment. Second, we develop a Keypoints Tracker based on the BFM that focuses on interaction-critical points, enabling flexible interaction alongside stable, natural gaits. Third, we design a progressive training framework that effectively utilizes multi-scale datasets to achieve robust zero-shot HOI without complex, hand-engineered mechanisms. 2 Related Work 2.1 Humanoid Behavior Foundation Models Humanoid motion tracking 40 ; 13 serves as the technical cornerstone for robust whole-body control, aiming to produce human-like movements by imitating reference trajectories. To scale these capabilities, recent research 31 ; 8 ; 14 ; 15 ; 20 ; 65 ; 71 ; 38 ; 53 focuses on training general-purpose motion trackers using large-scale datasets to reproduce diverse human behaviors. Building upon these tracking techniques, Behavior Foundation Models (BFMs) 25 ; 32 ; 27 ; 62 ; 66 have emerged to construct universal and reusable motion representation spaces. These models distill diverse motor skills into a compact latent space, providing strong physical priors for downstream tasks. Following this paradigm, our proposed framework trains a BFM backbone to function as a robust motion prior. By utilizing the learned latent representation of the BFM as the policy search space, the framework effectively constrains the movements of the humanoid, ensuring natural gaits and whole-body stability even in highly sparse keypoints tracking settings. 2.2 Humanoid-Object Interaction Humanoid-Object Interaction (HOI) has recently achieved significant progress in complex tasks, such as ball sports 69 ; 42 ; 43 and pick-and-place operations 51 ; 48 ; 9 ; 29 ; 67 . However, many of these methods rely heavily on task-specific