ZeroMimic: Distilling Robotic Manipulation Skills from Web Videos
Junyao Shi, Zhuolun Zhao, Tianyou Wang, Ian Pedroza, Amy Luo, Jie Wang, Jason Ma, Dinesh JayaramanUniversity of Pennsylvania zeromimic.github.io, [-4.0ex] · 2025
Read ZeroMimic: Distilling Robotic Manipulation Skills from Web Videos by Junyao Shi, Zhuolun Zhao, Tianyou Wang, Ian Pedroza, Amy Luo, Jie Wang, Jason Ma, Dinesh JayaramanUniversity of Pennsylvania zeromimic.github.io, [-4.0ex] (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2503-23877.txt.
Introduction
Junyao Shi and seven co-authors teach robots household skills straight from human web videos. Their paper appeared in two thousand twenty-five. Imitation learning usually needs demonstrations on the very same robot. Collecting those is slow and costly. The web already holds vast human manipulation footage. ZeroMimic mines that footage for deployable robot policies. No robot demonstrations or exploration are required.
The system outputs image-goal-conditioned skills directly. Opening, closing, pouring, picking, cutting, and stirring are covered. Each skill acts across varied objects and rooms. Diverse scenes in the wild are the training strength. Robots gain a skill library distilled from people. Manipulation research escapes the lab-bound demo bottleneck.
Catalog record
| CREATOR | Junyao Shi, Zhuolun Zhao, Tianyou Wang, Ian Pedroza, Amy Luo, Jie Wang, Jason Ma, Dinesh JayaramanUniversity of Pennsylvania zeromimic.github.io, [-4.0ex] |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 49 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2503-23877
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
ZeroMimic: Distilling Robotic Manipulation Skills from Web Videos Junyao Shi, Zhuolun Zhao, Tianyou Wang, Ian Pedroza, Amy Luo, Jie Wang, Jason Ma, Dinesh JayaramanUniversity of Pennsylvania zeromimic.github.io, [-4.0ex] Abstract Many recent advances in robotic manipulation have come through imitation learning, yet these rely largely on mimicking a particularly hard-to-acquire form of demonstrations: those collected on the same robot in the same room with the same objects as the trained policy must handle at test time. In contrast, large pre-recorded human video datasets demonstrating manipulation skills in-the-wild already exist, which contain valuable information for robots. Is it possible to distill a repository of useful robotic skill policies out of such data without any additional requirements on robot-specific demonstrations or exploration? We present the first such system ZeroMimic, that generates immediately deployable image goal-conditioned skill policies for several common categories of manipulation tasks (opening, closing, pouring, pick&place, cutting, and stirring) each capable of acting upon diverse objects and across diverse unseen task setups. ZeroMimic is carefully designed to exploit recent advances in semantic and geometric visual understanding of human videos, together with modern grasp affordance detectors and imitation policy classes. After training ZeroMimic on the popular EpicKitchens dataset of ego-centric human videos, we evaluate its out-of-the-box performance in varied \chadded [id=ax]real-world and simulated kitchen settings \chadded [id=ax]with two different robot embodiments, demonstrating its impressive abilities to handle these varied tasks. To enable plug-and-play reuse of ZeroMimic policies on other task setups and robots, \chadded [id=ax]we release software and policy checkpoints of our skill policies. Fig. 1 : ZeroMimic distills robotic manipulation skills from egocentric web videos for zero-shot deployment \chadded [id=js]across diverse real-world and simulated environments, a variety of objects, and different robot embodiments. I Introduction It is clear that animals and humans are able to observe third-person experiences to acquire functional sensorimotor skills, often “zero-shot” with limited or no need for additional practice. For example, one can learn to cook pasta, use a wood lathe, plant a garden, or tie a necktie, with reasonable proficiency by watching how-to video demonstrations on the web. While “imitation learning” has also been instrumental in many recent successes for robotic manipulation [ 1 , 2 , 3 , 4 ] , these robots largely rely on a much stronger kind of demonstration — gathered by manually operating the very same robot in the same small set of scenarios (scenes, viewpoints, objects, lighting, background textures, and distractors) to perform the task of interest. This is an immediate stumbling block on the road to developing general-purpose robots: gathering robot- and scenario-specific demonstrations scales poorly. Learning robot skills from in-the-wild human videos offers the enticing prospect that data would no longer be a bottleneck: videos of humans demonstrating varied manipulation tasks in diverse scenarios are already available on the web, it is easy to gather many more if needed, and further, the same videos could be re-used for many robots. However, there are serious challenges. Robots differ from humans in embodiments, action spaces, and hardware capabilities. Individual web videos often do not conveniently present all the details of how to perform a task (e.g. occlusions, out-of-frame objects and actions, or shaky moving cameras). Finally, the distribution of in-the-wild videos spans very large variations that may be hard to handle. We present an approach, ZeroMimic, that systematically overcomes these challenges and distills in-the-wild egocentric videos from EpicKitchens [ 5 ] into a repository of off-the-shelf deployable image goal-conditioned robotic manipulation skill policies that transfer across scenarios. Briefly, we abstract the action spaces of humans and standard robot arms with two-fingered grippers to permit coarse action transfer, we exploit video activity understanding and pre-existing visuomotor robot primitives such as grasping to transfer the finer details of control, we exploit modern structure-from-motion systems to maintain 3D maps of noisy and shaky in-the-wild egocentric human videos, and demonstrate that large policy classes can digest the diversity of web video to learn useful behaviors. The resulting system empirically demonstrates zero-shot robotic manipulation capabilities to perform a wide range of skills with diverse objects. In summary, our contributions are: 1. We develop ZeroMimic, a system that distills robotic manipulation skills from web videos that can be deployed zero-shot in diverse everyday environments. 2. We evaluate ZeroMimic on 9 different skills and show that ZeroMimic achieves 71.0% out-of-the-box success rate \chadded [id=ax]in the real world, 73.8% success rate in simulation, can generalize to new objects unseen in our curated web video \chadded [id=ax], and can be deployed across different robot embodiments. 3. Our ablation studies reveal important lessons of what is important in learning and executing robotic skills purely from in-the-wild human videos. II Related Work Fig. 2 : Representative related work organized by Generality of Source Human Videos and Level of Knowledge Transfer . ZeroMimic learns diverse zero-shot policies from in-the-wild web videos. Popular recent approaches [ 1 , 2 , 3 ] for enabling robot manipulation often rely on costly high-quality in-domain robot demonstrations. Therefore, recent works in robot learning have increasingly focused on leveraging unstructured or out-of-domain data. Some works have demonstrated the zero-shot capabilities of models trained on large robotics datasets [ 6 , 7 , 8 , 9 , 10 , 11 , 12 , 13 , 14 ] , but the curation of such datasets incurs a significant cost. Some have exploited recent advances in VLMs, trained on “web” data without any connection to robotics, and directly elicit zero-shot robotic actions [ 15 , 16 , 17 , 18 , 19 , 20 , 21 ] . \chadded [id=js]These policies are limited by the lack of physical understanding and slow inference speed of VLMs, \chadded [id=ax]as demonstrated by our experiments in Section IV-D . Human web videos [ 22 , 5 , 23 , 24 , 25 , 26 ] , due to their abundance, diversity, and rich information about interactions, emerge as a promising source of data for robotic skill acquisition. Since generating robot policies from out-of-domain human videos directly is difficult, many works instead train representations [ 27 , 28 , 29 ] (e.g. R3M [ 27 ] ), rewards [ 30 , 31 , 32 , 33 ] (e.g. VIP [ 31 ] ), or affordances [ 34 , 35 , 36 , 37 , 38 , 39 , 40 , 41 , 42 , 43 , 44 , 45 , 46 , 47 , 48 , 49 , 50 , 51 , 52 , 53 , 54 , 55 , 56 , 57 , 58 , 59 , 60 ] (See Fig 2 ). \chadded [id=js]Some works [ 34 , 35 , 36 , 37 , 38 , 39 , 40 , 41 , 42 , 43 , 44 , 45 ] (e.g. MimicPlay [ 34 ] , WHIRL [ 36 ] , and ATM [ 39 ] ) explored learning afoordances from in-domain human videos. Recent works [ 46 , 47 , 48 , 49 , 50 , 51 , 52 , 53 , 54 , 55 , 56 , 57 , 58 , 59 , 60 ] (e.g. VRB [ 53 ] , Track2Act [ 57 ] , LAPA [ 60 ] ) extended these approaches to learning from in-the-wild human videos. Since