EgoMI: Learning Active Vision and Whole-Body Manipulation from Egocentric Human Demonstrations
Justin Yu, Yide Shentu, Di Wu, Pieter Abbeel, Ken Goldberg, Philipp Wu · 2025
Read EgoMI: Learning Active Vision and Whole-Body Manipulation from Egocentric Human Demonstrations by Justin Yu, Yide Shentu, Di Wu, Pieter Abbeel, Ken Goldberg, Philipp Wu (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2511-00153.txt.
Introduction
Justin Yu, Yide Shentu, and Di Wu present EgoMI, a framework for teaching robots active vision and whole-body manipulation from egocentric human demonstrations. Humans coordinate head and hand movement while manipulating, fixating visually before acting, which creates dynamic head motion that static robot sensors cannot copy, causing distribution shift. Their interface captures synchronized end-effector and active head trajectories from human demonstrators.
Robots can then learn policies that include where to look, not just how to move. Robotics readers get a way to transfer a subtle human skill: viewpoint control as part of manipulation. Closing the embodiment gap on vision may matter as much as matching arm motions.
Catalog record
| CREATOR | Justin Yu, Yide Shentu, Di Wu, Pieter Abbeel, Ken Goldberg, Philipp Wu |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 35 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2511-00153
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
EgoMI: Learning Active Vision and Whole-Body Manipulation from Egocentric Human Demonstrations Justin Yu, Yide Shentu, Di Wu, Pieter Abbeel, Ken Goldberg, Philipp Wu Abstract Imitation learning from human demonstrations offers a promising approach for robot skill acquisition, but egocentric human data introduces fundamental challenges due to the embodiment gap. During manipulation, humans actively coordinate head and hand movements, continuously reposition their viewpoint and use pre-action visual fixation search strategies to locate relevant objects. These behaviors create dynamic, task-driven head motions that static robot sensing systems cannot replicate, leading to a significant distribution shift that degrades policy performance. We present EgoMI (Egocentric Manipulation Interface), a framework that captures synchronized end-effector and active head trajectories during manipulation tasks, resulting in data that can be retargeted to compatible semi-humanoid robot embodiments. To handle rapid and wide-spanning head viewpoint changes, we introduce a memory-augmented policy that selectively incorporates historical observations. We evaluate our approach on a bimanual robot equipped with an actuated camera head and find that policies with explicit head-motion modeling consistently outperform baseline methods. Results suggest that coordinated hand–eye learning with EgoMI effectively bridges the human-robot embodiment gap for robust imitation learning on semi-humanoid embodiments. Project page: https://egocentric-manipulation-interface.github.io I INTRODUCTION Learning from large-scale human demonstrations represents a powerful pathway toward scalable robot skill acquisition. Recent advances in imitation learning have shown promising results in training robots directly from human collected data. However, many critical challenges emerge from the embodiment gap between humans and robots. During manipulation tasks, humans naturally integrate head movements with hand actions, dynamically adjusting their viewpoint to maintain visual contact with objects, anticipate occlusions, and retain spatial information. This active perception strategy is fundamental to how we understand and interact with our environment. In contrast, many robotic systems rely on fixed external or wrist-camera views that cannot replicate this coordinated visual behavior. When learning from egocentric human demonstrations, this embodiment gap creates severe distribution shifts, as the dynamic viewpoint changes present in human data cannot be reproduced by static robot perception systems. Various methods attempt to minimize the embodiment gap through strategies such as restricting to wrist mounted cameras [ 1 , 2 , 3 , 4 ] or projecting top camera views into a coordinate-invariant representation [ 5 , 6 ] . However, for more complex tasks that require searching or looking with the head, the learned policy is unable to reproduce the demonstrated behavior. The inability for robot policies to remember what it has seen in the past further adds to this gap. Together, these mismatches create substantial embodiment gaps that hinder policy learning and degrade imitation performance. To address this challenge, we introduce Egocentric Manipulation Interface (EgoMI), as illustrated in , a framework that captures the full degrees of freedom of human perceptual movements with minimal embodiment gap. EgoMI simultaneously records both end-effector and head movements during human demonstrations. Proprioception and camera data streams for both the wrist and head result in more complete conditioning information for the downstream policy. We can then retarget a policy’s prediction of the whole-body motion data onto a robot. We use a modified Rainbow-RBY1 (wheeled, semi humanoid robot), equipped with a 6-DoF arm that functions as a neck to enable faithful retargeting of human motions to the robot. Additionally, we present Spatial-Aware Robust Keyframe Selection (SPARKS), a lightweight algorithm that selects a compact set of past head keyframe images to mitigate the loss of context caused by rapid head viewpoint shifts, while avoiding out-of-distribution failures. SPARKS is a simple yet effective mechanism that emphasizes keyframes rich in spatial information, and scores past frames using viewpoint novelty, temporal recency, and motion smoothness as a training-free proxy for visual informativeness. By embedding spatial memory into robot policies, SPARKS enables more stable long-horizon reasoning and resilience to viewpoint shifts. Through real world robotics experiments, we show that incorporating head observations and movement from the egocentric demonstration is crucial during both the data collection stage and policy deployment stage. Policies trained without the head trajectory data or memory consistently fail due to a lack of critical context. Remarkably, our framework achieves zero-robot-data transfer without relying on any augmentations or viewpoint re-rendering, demonstrating that capturing egocentric head and hand motions and observations is sufficient to bridge the embodiment gap. In summary, we: • Demonstrate the importance of an actuated head for imitation learning in everyday robotic manipulation tasks. • Introduce a simple yet effective approach for training robot policies with spatial memory, addressing the challenges posed by rapid perspective changes from the egocentric head camera. • Develop a data collection device that records key egocentric data. • Evaluate the approach in real-world experiments, demonstrating capabilities enabled by head retargeting and memory-aware policies. • Release code, hardware designs, and experiments to facilitate reproducibility and further research. II Related Work II-A Data collection devices for imitation learning Progress in imitation learning is closely intertwined with advancements in the devices used to capture demonstrations, as the choice of interface heavily influences both the quality of collected data and scalability of the process. Devices used for teleoperation vary widely, ranging from joysticks and 3D spacemice [ 7 , 8 ] , VR controllers [ 9 , 10 , 11 ] , and camera based sensors [ 1 , 12 , 13 , 14 ] . While effective for many tasks, these devices typically require control abstractions to be simplified into end-effector space due to mismatches between human morphology and robot kinematics. Leader–follower systems such as ALOHA [ 15 ] , GELLO [ 16 ] and AirExo-2 [ 17 ] mitigate this issue by matching the teleoperation device more directly to the robot morphology. UMI [ 2 ] represents a middle ground, using only the robot gripper as the input device. Large-scale datasets such as EgoDex [ 18 ] exemplify this approach. However, these methods still exhibit a large embodiment gap, since human motions, perspectives, and physical morphology differ substantially from a robot’s own embodiment. II-B Imitation learning approaches from human data Recent advances in large-scale imitation learning have resulted in robot foundation model approaches that integrate pre-trained vision and language representations with behavior cloning to scale imitation learning across numerous tasks [ 19 , 20 , 21 , 22 , 23 ] . Follow-on systems such as RoboFlamingo [ 24 ] , Octo [ 25 ] , OpenVLA [ 26 ] \pi 0 [ 27 ] [ 28 ] , \pi 0.5 [ 29 ] , and Seed GR-3 [ 30 ] confirm that combining demonstration data with foundation models yields strong zero-shot generalization and versatile open-world behavior. While the above largely use teleoperated data, others focus on data from human egocentric data. One notable example is EgoMimic [ 31 ] , which captures egocentric video and detailed 3D hand tracking using augmented reality (AR) glasses and trains policies on both human and robot data to improve generalization. DexCap [ 6 ] introduces scalable finger-level demonstrations collected via motion-capture gloves alongside a specialized imitation algorithm to precisely capture fine manipulation details. Building upon their works, ARCap [ 5 ] harnesses AR feedback to guide novice operators in recording robot-executable trajectories, improving demonstration quality and usability. These efforts collectively push forward scalable and accessible high-fidelity data collection for imitation learning.