Mitty: Diffusion-based Human-to-Robot Video Generation
Yiren SongCheng LiuWeijia Mao, Mike Zheng Shou · 2025
Read Mitty: Diffusion-based Human-to-Robot Video Generation by Yiren SongCheng LiuWeijia Mao, Mike Zheng Shou (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2512-17253.txt.
Introduction
Yiren Song and three co-authors present Mitty, a system that turns a video of a person doing a task into a video of a robot doing the same task. Older methods first reduce the human video to key points or paths, and that shortcut loses detail and piles up errors. Mitty skips the shortcut and works end to end.
Built on a pretrained video diffusion model, Mitty squeezes the demo into tokens and blends them with the robot video as it forms. The team also built a pipeline that makes matched human and robot clips from large first-person video sets, easing a shortage of training pairs. On two test sets, Mitty beat prior methods and handled new rooms it had never seen.
Catalog record
| CREATOR | Yiren SongCheng LiuWeijia Mao, Mike Zheng Shou |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 40 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2512-17253
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Mitty: Diffusion-based Human-to-Robot Video Generation Yiren SongCheng LiuWeijia Mao, Mike Zheng Shou Abstract Learning directly from human demonstration videos is a key milestone toward scalable and generalizable robot learning. Yet existing methods rely on intermediate representations such as keypoints or trajectories, introducing information loss and cumulative errors that harm temporal and visual consistency. We present Mitty, a Diffusion Transformer that enables video In-Context Learning for end-to-end Human2Robot video generation. Built on a pretrained video diffusion model, Mitty leverages strong visual-temporal priors to translate human demonstrations into robot-execution videos without action labels or intermediate abstractions. Demonstration videos are compressed into condition tokens and fused with robot denoising tokens through bidirectional attention during diffusion. To mitigate paired-data scarcity, we also develop an automatic synthesis pipeline that produces high-quality human-robot pairs from large egocentric datasets. Experiments on Human2Robot and EPIC-Kitchens show that Mitty delivers state-of-the-art results, strong generalization to unseen environments, and new insights for scalable robot learning from human observations. Our project homepage is available at https://github.com/showlab/Mitty Figure 1 : We propose Mitty, a paradoxical in-context learning–based video generation method built on Diffusion Transformers. It can convert human demonstration videos into robotic manipulation videos and achieves high task success rates. † † footnotetext: \dagger Corresponding author. 1 Introduction Humans excel at rapidly acquiring new skills by observing others. If robots could directly learn manipulation policies from a single human demonstration video and generate corresponding robot-execution videos, this would provide a critical path toward cross-task and cross-environment generalization. Yet achieving this has long been a highly challenging goal in robotics. Existing approaches typically rely on intermediate representations such as keypoints, trajectories, or depth maps to bridge human and robot videos. They first extract keypoints or trajectories from the human demonstration and then condition a rendering module to synthesize robot execution videos. While intuitive, this approach fails to fully exploit the rich information embedded in demonstration videos and struggles to capture the fine-grained spatio-temporal dynamics essential for robust generalization. Moreover, errors accumulated in the intermediate estimation stage can further degrade performance. This raises a natural question: can we bypass intermediate representations and directly achieve end-to-end Human2Robot video generation? This task presents several key challenges: (1) Appearance and scene consistency—the generated robot video must match the scene of the human demonstration while preserving a stable, plausible robot embodiment; (2) Action and strategy alignment—the robot’s actions must follow the human demonstration yet adapt to structural differences between human hands and robot arms; (3) Data scarcity—despite abundant human and robot videos separately, finely aligned human–robot video pairs are extremely rare. The only public H2R dataset currently contains just 2,600 video pairs across nine tasks, making it difficult to learn generalizable skills from limited data. To alleviate this scarcity, we employ an automatic paired-data synthesis pipeline 23 using egocentric human videos. Starting from large-scale human activity datasets such as EPIC-Kitchens, we estimate 3D hand keypoints, remove human hands, and inpaint clean backgrounds. We then map keypoint sequences to robot end-effector poses and render robot arms into the video, producing high-quality human–robot paired videos. This approach bypasses traditional intermediate representations and significantly improves both scale and fine-grained temporal consistency, providing stronger training and generalization capacity for our model. Building on this foundation, we introduce Mitty, a Diffusion Transformer framework for video In-Context Learning. In-context learning (ICL) has shown promise for fewshot learning, offering data-efficient and rapid adaptation at test time. By simply conditioning on one human demonstrations, ICL can predict robot actions to achieve novel tasks at test time without expensive retraining. Our method conditions directly on human demonstration videos to generate corresponding robot-execution videos in an end-to-end manner, requiring no explicit action labels. Mitty leverages Wan 2.2, a powerful video generation model pretrained on massive natural video corpora, to inherit strong visual and temporal priors. Concretely, we compress the human demonstration into condition tokens via a VAE (kept noise-free) and concatenate them with the robot denoise tokens through a bidirectional attention mechanism during diffusion, enabling cross-domain action translation. Mitty supports two inference modes—first-frame-controlled and zero-frame generation—offering greater flexibility in deployment. We further conduct a systematic evaluation across models and settings to provide actionable insights for the community. Across both the Human2Robot dataset and EPIC-Kitchens, Mitty significantly outperforms existing baselines and demonstrates strong generalization. Although our work does not yet realize a full Human2Robot control loop, generating high-fidelity, temporally aligned, and semantically accurate Human2Robot videos is itself a crucial and challenging task. Such videos offer richer supervision than keypoints or trajectories and form the foundation for future video-to-policy inversion. Once the generated videos are reliable enough, they hold the potential to be translated into executable control signals, making high-quality Human2Robot video generation a key first step toward end-to-end Human2Robot mapping. We summarize our contributions as follows: 1. We propose Mitty, the first end-to-end Human2Robot video generation framework built upon a Video Diffusion Transformer. 2. Technically, we leverage in-context learning to achieve both appearance and scene consistency as well as action consistency, significantly improving cross-task generalization. 3. We design an efficient data synthesis strategy and combine it with existing datasets for mixed training, which markedly enhances the model’s generalization ability on unseen tasks and environments. Extensive experiments demonstrate the effectiveness and superiority of our approach in terms of generation quality and cross-task consistency. 2 Related Works 2.1 Video Generation Models Video generation models have evolved rapidly from early GAN-based approaches 33 , UNet-based approaches 13 ; 53 ; 42 to today’s Diffusion Transformer architectures 34 ; 47 ; 59 ; 18 . Modern Diffusion Transformers can generate high-quality, temporally coherent videos conditioned on text, images, or multi-modal inputs, enabling applications such as controllable video generation 26 ; 18 ; 29 ; 31 ; 30 and world modeling 8 . Many recent studies also leverage large pretrained video generation models for tasks in robotics and mechanical manipulation 7 , highlighting their potential for cross-domain generalization and interactive learning. 2.2 Learning from Human Videos A growing body of work investigates how large human-centric video datasets can be used to improve robot policy learning 51 ; 38 ; 22 ; 39 ; 32 ; 2 ; 19 ; 27 ; 54 ; 24 ; 45 . Compared to costly and time-consuming teleoperation, large-scale human videos provide a scalable and diverse source of demonstrations. Earlier studies focused on extracting visual representations 4 , deriving reward functions 14 , or directly estimating motion priors from human videos 48 ; 35 . However, many approaches still rely on additional robot data or specialized hardware such as VR and hand-tracking devices, limiting scalability. Recent progress in 3D hand pose estimation 5 helps extract action information directly from RGB videos, but cross-embodiment transfer remains difficult. Humanoid robots can partially alleviate this gap due to their kinematic similarity to humans. Building on these trends, we propose Mitty, which achieves end-to-end generation of robot videos directly from human demonstrations without extracting intermediate representations such as pose, trajectories, or depth, and better leverages the fine-grained details contained in the original human demonstration videos. 2.3 In-Context Learning In-context learning (ICL) 3 ; 1 has demonstrated remarkable capability for adapting models to new tasks at inference time. In the visual generation domain, recent approaches have leveraged ICL to achieve high-quality image generation 58