X-Diffusion : Training Diffusion Policies on Cross-Embodiment Human Demonstrations

Maximus A. Pace, Prithwish DanChuanruo NingAtiksh BhardwajAudrey DuEdward W. Duan, Wei-Chiu Ma, Kushal KediaCornell University https://portal-cornell.github.io/X-Diffusion/ · 2025

Read X-Diffusion : Training Diffusion Policies on Cross-Embodiment Human Demonstrations by Maximus A. Pace, Prithwish DanChuanruo NingAtiksh BhardwajAudrey DuEdward W. Duan, Wei-Chiu Ma, Kushal KediaCornell University https://portal-cornell.github.io/X-Diffusion/ (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2511-04671.txt.

Introduction

Maximus A. Pace leads eight authors with X-Diffusion, training robot policies from human videos. Human demonstrations are plentiful, but human and robot bodies differ sharply. Many human actions cannot be copied directly by a robot arm.

Still, the videos carry rich cues about objects and task intent. The key move treats human actions as noisy versions of robot actions. Ambient Diffusion then uses low-quality data only at high-noise steps of training. That way, intent transfers while unworkable human motions fade out. The method aims at robot skills learned from coarse guidance alone.

Catalog record

CREATORMaximus A. Pace, Prithwish DanChuanruo NingAtiksh BhardwajAudrey DuEdward W. Duan, Wei-Chiu Ma, Kushal KediaCornell University https://portal-cornell.github.io/X-Diffusion/
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE30 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2511-04671

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

X-Diffusion : Training Diffusion Policies on Cross-Embodiment Human Demonstrations Maximus A. Pace, Prithwish DanChuanruo NingAtiksh BhardwajAudrey DuEdward W. Duan, Wei-Chiu Ma, Kushal KediaCornell University https://portal-cornell.github.io/X-Diffusion/ Abstract Human videos can be recorded quickly and at scale, making them an appealing source of training data for robot learning. However, humans and robots differ fundamentally in embodiment, resulting in mismatched action execution. Direct kinematic retargeting of human hand motion can therefore produce actions that are physically infeasible for robots. Despite these low-level differences, human demonstrations provide valuable motion cues about how to manipulate and interact with objects. Our key idea is to exploit the forward diffusion process: as noise is added to actions, low-level execution differences fade while high-level task guidance is preserved. We present X-Diffusion , a principled framework for training diffusion policies that maximally leverages human data without learning dynamically infeasible motions. X-Diffusion first trains a classifier to predict whether a noisy action is executed by a human or robot. Then, a human action is incorporated into policy training only after adding sufficient noise such that the classifier cannot discern its embodiment. Actions consistent with robot execution supervise fine-grained denoising at low noise levels, while mismatched human actions provide only coarse guidance at higher noise levels. Our experiments show that naive co-training under execution mismatches degrades policy performance, while X-Diffusion consistently improves it. Across five manipulation tasks, X-Diffusion achieves a 16% higher average success rate than the best baseline. Fig. 1 : Overview of X-Diffusion : We introduce X-Diffusion , a framework to train diffusion policies on cross-embodiment human data containing a variety of execution styles. Naively co-training diffusion policies on human and robot datasets with mismatched dynamics can lead the denoising process to output dynamically infeasible actions for the robot, degrading performance below standard robot-only diffusion policy training. Instead, X-Diffusion trains a classifier to distinguish between noised human and robot actions, and integrates noised human actions into policy training only when the classifier is unsure of which embodiment produced the actions, thus, effectively learning from large and diverse human demonstrations. † † footnotetext: ∗ Equal contribution. † Equal advising. I Introduction Imitation learning (IL) is an effective and flexible method for teaching robot skills, but collecting large amounts of robot data is costly and slow. Human video demonstrations offer a scalable alternative, since they are easier and faster to collect. However, such data cannot be directly used to train state-of-the-art IL methods [ 1 , 2 ] , since humans and robots significantly differ in embodiment. To leverage human data, recent works aim to unify human and robot action spaces [ 3 , 4 , 5 ] . Utilizing advances in 3D hand-pose estimation [ 6 ] , hand motions recorded from human videos can be converted into robot actions via kinematic retargeting. Consequently, these methods have been applied to crowd-sourced human video collection [ 7 , 8 ] and datasets of egocentric human videos [ 9 , 10 ] . Nevertheless, these approaches typically rely on fine-tuning with robot teleoperation data to produce actions compatible with the robot’s kinematics and dynamics. If final robot performance is the goal, is it optimal to train on all human demonstrations indiscriminately — or can some demonstrations, mismatched in execution with the robot, hurt policy learning? We study this problem setting where we have access to a large dataset of human demonstrations, and a small dataset of high-quality robot teleoperation data. Even for the same manipulation task, humans and robots often differ in execution style. For example, in Fig. 1 , when tasked with moving a plate on the table, a human can dexterously slide their fingers underneath to pick and place it. However, a robot with a parallel-jaw gripper may more reliably push or slide the plate across the table. Even with such differences in action execution, human videos still provide rich motion cues about how objects should be manipulated and interacted with. In diffusion policy learning, as noise is added to both human and robot actions, these low-level discrepancies start to fade away, preserving high-level guidance on how to complete the task. Our key insight is that training diffusion policies on noised human actions can improve task performance without sacrificing robot feasibility. We propose X-Diffusion , a framework for co-training diffusion policies with large-scale human demonstrations and a smaller set of robot teleoperation data (Fig. 1 ). Before policy training, we train a classifier to distinguish between noised human and robot actions in the forward diffusion process. To maximally leverage human data without introducing dynamically infeasible behaviors, we define the minimum indistinguishability step : the earliest diffusion step at which the classifier can no longer discern whether an action comes from a human or a robot. Actions that are compatible with robot kinematics and dynamics are integrated at lower noise levels, while actions that diverge from the robot’s execution style are only included at higher noise levels. As a result, feasible human and robot demonstrations provide precise, low-level supervision throughout the diffusion process, whereas mismatched human actions contribute only coarse, high-level guidance. This enables our method to extract useful signal from all human data while avoiding degradation from execution mismatches. Our contributions: 1. We propose X-Diffusion , a framework for training diffusion policies on cross-embodiment human data while preserving dynamically feasible robot motions. 2. We demonstrate that prior methods that train on all human demonstrations often generate infeasible robot actions. Through ablations and analyses, we demonstrate that our selective training strategy outperforms both naive co-training and manual human annotation. 3. Across 5 manipulation tasks, we show that X-Diffusion outperforms a range of cross-embodiment learning baselines by 16% on average. II Related Work Our work is related to the following topics: Learning from Human Hand Motion. Human videos typically lack action annotations that can be directly executed by robots. Recent progress in hand-pose estimation has enabled retargeting human hand motion into robot actions. A common approach is to track 6DoF hand trajectories and map them to the robot end-effector [ 11 , 12 , 13 ] , which is particularly effective for dexterous robotic hands [ 14 , 15 , 16 , 17 ] . Other works define corresponding keypoints between human and robot hands [ 3 , 4 ] to unify their state and action spaces. Retargeting has also been leveraged to synthesize robot data by overlaying rendered robot arms on human videos [ 18 , 5 , 19 ] . When combined with open-world vision models, these methods can further enable object-aware retargeting of human motion [ 20 , 21 , 22 ] . While promising, these approaches often assume that every human hand motion can be feasibly executed by a robot. This assumption breaks down in practice, as many human actions involve kinematics or dynamics outside a robot’s capabilities, limiting the reliability of direct retargeting. Extracting Rewards from Human Data Beyond imitation, reinforcement learning (RL) can leverage human data by defining rewards from tracking reference motion [ 23 , 24 ] , video similarity [ 25 , 26 , 27 ] , language alignment [ 28 , 29 ] , or object-centric signals in real-to-sim-to-real pipelines [ 30 , 31 , 32 ] . Preference learning methods further derive

Read the complete text