ImMimic: Cross-Domain Imitation from Human Videos via Mapping and Interpolation

Yangcen Liu, Woo Chul Shin, Yunhai Han, Zhenyang Chen, Danfei Xu · 2025

Read ImMimic: Cross-Domain Imitation from Human Videos via Mapping and Interpolation by Yangcen Liu, Woo Chul Shin, Yunhai Han, Zhenyang Chen, Danfei Xu (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2509-10952.txt.

Introduction

Robots that learn housework from watching people are Yangcen Liu and five co-authors' goal in this 2025 robotics paper. Human videos are plentiful, robot demonstrations expensive, but bodies differ: hands are not grippers, and viewpoints and physics shift too. Their framework, ImMimic, trains across both domains at once and needs only a small set of teleoperated robot demonstrations.

The method maps retargeted human hand poses onto robot joints using dynamic time warping, guided by actions or by vision. Then MixUp interpolation blends paired human and robot trajectories, manufacturing intermediate domains that smooth the adaptation. On four real tasks, picking and placing, pushing, hammering, and flipping, across four different robot hands, ImMimic improved success rates and made execution smoother. The deeper lesson: retargeted hand motion is itself a useful action label.

Catalog record

CREATORYangcen Liu, Woo Chul Shin, Yunhai Han, Zhenyang Chen, Danfei Xu
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE63 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2509-10952

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

ImMimic: Cross-Domain Imitation from Human Videos via Mapping and Interpolation Yangcen Liu, Woo Chul Shin, Yunhai Han, Zhenyang Chen, Danfei Xu Abstract Learning robot manipulation from abundant human videos offers a scalable alternative to costly robot-specific data collection. However, domain gaps across visual, morphological, and physical aspects hinder direct imitation. To effectively bridge the domain gap, we propose ImMimic , an embodiment-agnostic co-training framework that leverages both human videos and a small amount of teleoperated robot demonstrations. ImMimic uses Dynamic Time Warping (DTW) with either action- or visual-based mapping to map retargeted human hand poses to robot joints, followed by MixUp interpolation between paired human and robot trajectories. Our key insights are (1) retargeted human hand trajectories provide informative action labels, and (2) interpolation over the mapped data creates intermediate domains that facilitate smooth domain adaptation during co-training. Evaluations on four real-world manipulation tasks (Pick and Place, Push, Hammer, Flip) across four robotic embodiments (Robotiq, Fin Ray, Allegro, Ability) show that ImMimic improves task success rates and execution smoothness, highlighting its efficacy to bridge the domain gap for robust robot manipulation. The project website can be found at https://sites.google.com/view/immimic . Keywords: Learning from Human, Imitation learning, Dexterous Manipulation Figure 1: ImMimic enables embodiment-agnostic co-training between human and robot demonstrations. It leverages large-scale human videos and a small amount of teleoperated robot data, using a MixUp interpolation to enable smooth domain transfer. We validate ImMimic on four diverse manipulation tasks across four robotic embodiments. 1 Introduction Teaching robots to perform diverse manipulation tasks in real-world environments remains a significant challenge because collecting robot-specific demonstration data is expensive. As an alternative, human videos have emerged as a promising resource, offering abundant examples of people engaging in everyday manipulation activities [ 1 , 2 ] . Leveraging these human videos for robot training provides a scalable and cost-effective approach to enhance robotic skills without extensive robot demonstration collection or simulation [ 3 , 4 , 5 ] . However, learning robot skills from human demonstration videos still faces an inherent limitation: a substantial domain gap arising from stark differences in visual appearance, embodiment structure, physical constraints, and other factors. In general, the challenge of enabling robots to learn human demonstrations can be formulated as a domain adaptation problem: the robot (representing the target domain) aims to emulate the behaviour of the human demonstrator (representing the source domain). A relevant application of this concept can be seen in several recent, well-established vision-based teleoperation systems [ 6 , 7 , 8 ] where the human demonstrator often first practices with the teleoperation setup before being able to collect high-quality robot data. This process indeed reflects an instance of inverse adaptation, where the human demonstrator adapts to the robot system rather than the way around. However, such inverse adaptation is absent in human demonstration videos, as human demonstrators do not consider the robot’s subsequent operation. Therefore, to enable effective adaptation when learning robot skills from such human data, recent works often preprocess the input data of both domains—for example, masking out embodiments in images [ 9 , 4 ] to mitigate visual differences, or restricting the action space to only 3D translations to address the embodiment-specific action gap [ 3 , 10 ] . Additionally, another line of works encourages the adaptation of the latent spaces of visual inputs from both domains using unsupervised learning objectives [ 11 , 12 , 13 , 14 ] , but often overlooks human actions, i.e., the hand trajectories, instead learning the action decoder solely from robot demonstrations. To develop a more generalizable adaptation method across diverse robot embodiments and manipulation tasks, we introduce ImMimic (Interpolation-via-Mapping Mimic) , an embodiment-agnostic co-training framework that learns jointly from human demonstration videos and robot teleoperations. Our key insights are: (1) beyond the visual contexts, the retargeted human hand trajectories can serve as action labels for human demonstrations, (2) creating intermediate domains via interpolation leads to robust adaptation, and (3) establishing an effective mapping between human and robot data for interpolation is essential for co-training. Specifically, we begin by retargeting human hand poses into the robot’s action space. We then perform sequence‑level mapping via Dynamic Time Warping (DTW), using either visual features or action distance to pair each human timestep with its best‑matching robot timestep. Finally, inspired by MixUp‑based adaptation [ 15 , 16 ] , we interpolate both condition and predicted actions along these DTW mapping, enabling adaptation through intermediate (human–robot) domains. To demonstrate the benefits of the ImMimic, we conduct comprehensive experiments across four different types of embodiments: Robotiq Gripper, FR Gripper [ 17 , 18 , 19 ] , Allegro Hand, and Ability Hand. These embodiments are evaluated on four manipulation tasks: Pick and Place, Push, Hammer, and Flip. We show that ImMimic achieves higher success rates and smoother motions across all embodiments and tasks compared to baseline methods. We observe that action-based mapping provides greater improvements than visual-based mapping, suggesting that the rich action information of the human hand trajectories is equally, if not more, beneficial for co-training. Furthermore, we find that performance improves when the average action distance between human and robot is smaller, and interestingly, due to the factors such as hand-arm mounting conditions and arm kinematics, solely using a human-mimetic end-effector does not necessarily result in smaller action distances. Finally, we analyze failure cases in terms of hardware structure and algorithmic factors. 2 Related Work Learning from Human Videos. Human videos offer an efficient and scalable source of supervision for robot learning. Retrieval‑based approaches [ 20 , 14 , 21 , 11 , 12 ] search large video corpora for sequences that resemble the desired behavior for augmented learning, while typically relying on the robot’s own data for action decoding. To better utilize human videos, two-stage methods [ 5 , 22 , 23 , 3 , 24 , 25 ] first learn high‑level policies on human data and then adapt to robot demonstrations, but limit low-level action learning. To address this, co‑training [ 4 ] jointly optimizes on human and robot data, yet typically relies on heavy visual preprocessing or simplified action spaces, leaving the core domain shift largely unaddressed. Our work adopts a co-training pipeline, and treating human videos as domain-adaptive supervision to smoothly address the domain shifts. Dexterous Retargeting. Dexterous retargeting translates human hand poses into robot joint poses, ensuring that human trajectories are mapped into the same action space used for robot execution. Most recent works adopt this technique to generate robot action commands for teleoperation by observing a human demonstrator moving their own hand [ 6 , 7 , 26 , 8 , 27 ] . However, continuous guidance from the human demonstrator is crucial for the robot’s success during teleoperation [ 28 ] . Other works [ 29 , 30 , 5 , 31 , 32 , 33 , 34 , 35 , 36 ] leverage retargeted robot trajectories to learn robot policies through bespoke refinements tailored to dexterous grasping [ 29 , 30 ] , reinforcement learning [ 34 , 32 , 33 , 35 ] , fine-tuning with additional robot data [ 5 , 31 ] , or human-in-the-loop corrections [ 36 ] . This additional process underscores the domain

Read the complete text