Bridging the Embodiment Gap: Disentangled Cross-Embodiment Video Editing

Zhiyuan Li, Wenyan Yang, Wenshuai Zhao, Yue Ma, Yuanpeng Tu, Pekka Marttinen, Joni Pajarinen · 2026

Read Bridging the Embodiment Gap: Disentangled Cross-Embodiment Video Editing by Zhiyuan Li, Wenyan Yang, Wenshuai Zhao, Yue Ma, Yuanpeng Tu, Pekka Marttinen, Joni Pajarinen (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2605-03637.txt.

Introduction

Zhiyuan Li and six co-authors address the data bottleneck in robot learning: human videos are plentiful, but robots move differently from people. Their framework edits a human demonstration video into a robot execution video. The key is separating what the task is from how a particular body performs it.

The method factorizes a video into two independent latent spaces, one for task content and one for embodiment, enforced by a dual contrastive objective. A small adapter feeds those codes into a frozen video diffusion model, producing a coherent robot version from a single human demonstration, with no paired human-robot data needed. Experiments show temporally consistent, morphologically accurate robot demonstrations.

Catalog record

CREATORZhiyuan Li, Wenyan Yang, Wenshuai Zhao, Yue Ma, Yuanpeng Tu, Pekka Marttinen, Joni Pajarinen
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE37 KB
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2605.03637

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Bridging the Embodiment Gap: Disentangled Cross-Embodiment Video Editing Zhiyuan Li, Wenyan Yang, Wenshuai Zhao, Yue Ma, Yuanpeng Tu, Pekka Marttinen, Joni Pajarinen Abstract Learning robotic manipulation from human videos is a promising solution to the data bottleneck in robotics, but the distribution shift between humans and robots remains a critical challenge. Existing approaches often produce entangled representations, where task-relevant information is coupled with human-specific kinematics, limiting their adaptability. We propose a generative framework for cross-embodiment video editing that directly addresses this by learning explicitly disentangled task and embodiment representations. Our method factorizes a demonstration video into two orthogonal latent spaces by enforcing a dual contrastive objective: it minimizes mutual information between the spaces to ensure independence while maximizing intra-space consistency to create stable representations. A parameter-efficient adapter injects these latent codes into a frozen video diffusion model, enabling the synthesis of a coherent robot execution video from a single human demonstration, without requiring paired cross-embodiment data. Experiments show our approach generates temporally consistent and morphologically accurate robot demonstrations, offering a scalable solution to leverage internet-scale human video for robot learning. 1 Introduction The development of generalist embodied agents for unstructured real-world environments has been a longstanding goal in robotics ( 33 ) . While foundation models in natural language processing (NLP) ( 1 ; 9 ) and computer vision (CV) ( 4 ; 23 ; 53 ; 55 ) have succeeded by training on large, diverse internet datasets, robotics faces a persistent data bottleneck. Collecting large-scale robotic demonstration data requires substantial resources and complex procedures, which limits the scope and generalization of learned policies. This constraint has led researchers to explore human videos from the internet as a scalable data source ( 15 ; 3 ) . These videos contain diverse tasks, physical interactions, and goal-oriented behaviors that could provide supervisory signals for training robotic skills. However, learning from human video (LfV) presents a fundamental challenge that prevents direct application of standard imitation learning methods ( 8 ; 29 ) : a substantial distribution shift ( 33 ) exists between human demonstrators and robotic learners. This ”embodiment gap” encompasses visual differences in appearance ( 24 ) and viewpoint, as well as critical morphological and dynamic differences arising from distinct kinematics, degrees of freedom, and physical affordances between human hands and robotic end-effectors. This gap makes the direct transfer of skills from human video a formidable problem. Existing approaches have attempted to bridge this gap with several strategies, yet they face key limitations. One line of research attempts to create intermediate representations to abstract away embodiment differences. This includes methods that define a unified state-action space to align human and robot data ( 20 ; 39 ) or those that learn a high-level plan from human video to guide a low-level robot controller ( 65 ; 46 ) . However, they often result in entangled representations where task-relevant information remains coupled with human-specific kinematic biases. A second line of work employs visual adaptation to modify the demonstration videos directly. These methods typically involve a procedural pipeline: inpainting removes the human agent, and a rendered robot model from a simulator is overlaid onto the scene ( 25 ; 24 ) . This approach is constrained by the photorealism of the simulator and, more critically, the simple overlay process can produce unrealistic video plans which are unsuitable for downstream policy learning. Faced with these challenges, we aim to design a generative framework for cross-embodiment video editing that explicitly learns disentangled representations of task and embodiment. We hypothesize that a demonstration video can be decomposed into two orthogonal components: a latent representation of the task capturing the goal, object dynamics, and interaction sequence, and a separate representation of the embodiment capturing the agent’s morphology and kinematics. Specifically, we introduce a framework built upon a large-scale, pre-trained generative model (VACE ( 19 ) ), which serves as a frozen generative backbone. Our approach augments this backbone with trainable encoders and a parameter-efficient adapter to factorize a source video into the distinct task and embodiment representations. Crucially, this disentanglement is enforced during training through a dual contrastive objective. This objective concurrently minimizes mutual information between the task and embodiment spaces using CLUB estimator ( 6 ) to ensure their independence, while simultaneously maximizing intra-space consistency within each space via an InfoNCE ( 35 ) loss to create stable representations. At inference, the framework extracts the task representation from a human video, computes a representation for a target robot’s embodiment, and composes them to synthesize a novel, robot-centric execution of the task. The resulting video provides a coherent, robot-specific visual demonstration, translating the human’s actions into a dense visual plan. Our contributions are summarized as follows: • We propose a generative, disentangled representation learning framework that addresses the key challenge of distribution shift in learning from human video. • We augment a frozen video diffusion model with trainable encoders and a parameter-efficient adapter, trained via a dual contrastive objective that explicitly disentangles task and embodiment representations by minimizing mutual information between them while maximizing consistency within each representation space. • Through extensive experiments, we demonstrate that our model synthesizes temporally coherent and morphologically consistent robot demonstration videos from a single human video, providing a scalable data source to help mitigate the data bottleneck in robotic imitation learning. Figure 1 : Framework Overview. Our framework for cross-embodiment video editing learns to disentangle a demonstration into two orthogonal latent representations. First, a trainable Task Encoder creates an embodiment-invariant Task Embedding from the text description, hand motion, and object trajectory. Simultaneously, an Embodiment Encoder generates an Embodiment Embedding from a static image of the agent’s end-effector. This disentanglement is enforced by a dual contrastive objective that minimizes mutual information between the two spaces while maximizing consistency within each. The embeddings are injected into a frozen Diffusion Transformer via a parameter-efficient Adapter. The generative model is also conditioned on the static background to focus synthesis on the agent. In inference, the task embedding from a source human video can be composed with a target robot’s embodiment embedding to synthesize a novel, robot-centric execution of the task. 2 Related Work 2.1 Learning from Videos Learning policies from human videos must reconcile the embodiment gap between humans and robots. Recent work establishes unified representation spaces for cross-embodiment alignment through shared policy backbones ( 20 ; 39 ) , joint motion manifolds ( 38 ) , or relative spatial relationships between agents and objects ( 51 ) . Visual domain adaptation offers an alternative by editing demonstrations through human inpainting and robot overlay ( 25 ; 26 ) . The absence of explicit action labels motivates learning intermediate representations that decouple observation from control. These include latent action plans ( 46 ; 56 ; 27 ; 21 ) , end-effector pose sequences ( 34 ) , task-specific parameterizations like screw motions for bimanual manipulation ( 2 ) , and object-centric state graphs ( 65 ) . Geometric reasoning provides another pathway: dense frame correspondences ( 22 ) or task-invariant feature matching ( 60 ) enable direct action inference. Generative models trained on internet-scale video learn physical dynamics as implicit world models for inverse dynamics control ( 16 ; 52 ) and synthesize trajectory datasets

Read the complete text