VidBot : Learning Generalizable 3D Actions from In-the-Wild 2D Human Videos for Zero-Shot Robotic Manipulation
Hanzhi Chen, Boyang Sun, Anran Zhang, Marc Pollefeys, Stefan Leutenegger · 2025
Read VidBot : Learning Generalizable 3D Actions from In-the-Wild 2D Human Videos for Zero-Shot Robotic Manipulation by Hanzhi Chen, Boyang Sun, Anran Zhang, Marc Pollefeys, Stefan Leutenegger (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2503-07135.txt.
Introduction
Hanzhi Chen and four co-authors teach robots household manipulation by watching ordinary human videos. Their framework, VidBot, appeared in twenty twenty-five. Training robots directly is slow and physical, while the internet already holds endless footage of hands doing tasks.
The pipeline lifts three-dimensional hand paths out of flat monocular clips. Depth estimation and structure-from-motion rebuild the paths at true scale, consistent over time. A two-stage learner first picks the coarse action, then a diffusion model drafts the fine contact trajectory, steered by test-time constraints. Across thirteen manipulation tasks, VidBot beat rivals in zero-shot tests, never having practiced the tasks. It also transferred onto real robot hardware in everyday rooms.
Catalog record
| CREATOR | Hanzhi Chen, Boyang Sun, Anran Zhang, Marc Pollefeys, Stefan Leutenegger |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 42 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2503-07135
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
VidBot : Learning Generalizable 3D Actions from In-the-Wild 2D Human Videos for Zero-Shot Robotic Manipulation Hanzhi Chen, Boyang Sun, Anran Zhang, Marc Pollefeys, Stefan Leutenegger Abstract Future robots are envisioned as versatile systems capable of performing a variety of household tasks. The big question remains, how can we bridge the embodiment gap while minimizing physical robot learning, which fundamentally does not scale well. We argue that learning from in-the-wild human videos offers a promising solution for robotic manipulation tasks, as vast amounts of relevant data already exist on the internet. In this work, we present VidBot , a framework enabling zero-shot robotic manipulation using learned 3D affordance from in-the-wild monocular RGB-only human videos. VidBot leverages a pipeline to extract explicit representations from them, namely 3D hand trajectories from videos, combining a depth foundation model with structure-from-motion techniques to reconstruct temporally consistent, metric-scale 3D affordance representations agnostic to embodiments. We introduce a coarse-to-fine affordance learning model that first identifies coarse actions from the pixel space and then generates fine-grained interaction trajectories with a diffusion model, conditioned on coarse actions and guided by test-time constraints for context-aware interaction planning, enabling substantial generalization to novel scenes and embodiments. Extensive experiments demonstrate the efficacy of VidBot , which significantly outperforms counterparts across 13 manipulation tasks in zero-shot settings and can be seamlessly deployed across robot systems in real-world environments. VidBot paves the way for leveraging everyday human videos to make robot learning more scalable. Figure 1 : We present VidBot , a framework to learn interactions from in-the-wild RGB-only human videos. Our affordance model can be deployed across robots for daily manipulation tasks. Project website: https://hanzhic.github.io/vidbot-project/ 1 Introduction Advancements in AI are accelerating the development of personalized devices, such as smart glasses that offer virtual guidance to users [ 22 , 28 , 13 , 62 ] . In the near future, robots will also become personalized systems, akin to smartphones or smart glasses, designed to provide physical assistance to humans. However, the diverse and novel forms of robotic embodiment pose a significant challenge for deploying AI to perform open-ended tasks in open-ended environments. State-of-the-art approaches attempting to learn robot manipulation skills still rely heavily on human experts’ teleoperated demonstrations, which are used to train robot policies under the Imitation Learning (IL) paradigm [ 35 , 37 , 95 , 64 ] . However, this process remains costly, time-consuming, and labor-intensive. While recent efforts have gathered large-scale robotic demonstrations for everyday manipulation tasks–such as Open X-Embodiment [ 59 ] and DROID [ 39 ] –scaling data collection remains challenging due to the combinatorial explosion of embodiments, tasks, and environments. We argue that human videos offer a promising scalable solution: there are already massive amounts of web videos capturing humans performing diverse tasks across various environments. Several previous approaches have explored human-to-robot skill transfer [ 78 , 65 , 3 , 97 , 2 , 73 , 79 , 88 ] . Nevertheless, they face certain limitations, such as requiring static cameras or scenes, depth sensors, MoCap systems, etc. These constraints often result in in-lab settings lacking diversity in scenes, illuminations, or viewpoints. One line of research has explored leveraging internet human videos with rich scene contexts to boost robot learning tasks, focusing on learning visual representations for visuomotor policies [ 85 , 52 , 57 , 67 ] . However, one major limitation is the reliance on humans to collect task-specific teleoperated data in every new environment with every new embodiment to fine-tune the pre-trained model. More recently, works like [ 4 ] have progressed by explicitly extracting agent-agnostic interaction trajectories. Nevertheless, these extracted motions are simplified as 2D vectors in pixel space, limiting their direct deployment to robots. We argue that, beyond visual representations or pixel-level action cues limited to the 2D image plane, 3D affordance–specifically, the contact points and interaction trajectories with spatial awareness–is crucial for unifying different embodiments to interpret action from perception. However, extracting general 3D affordance data from everyday human videos remains a significant challenge, impeding robots from learning manipulation skills by watching humans. In this work, we aim to enable zero-shot robot learning from a large amount of unlabeled everyday human videos, tackling two key questions: (1) How can 3D actionable knowledge be extracted from raw RGB-only human videos? (2) How can this extracted knowledge be reliably transferred to novel environments and new robot embodiments in a zero-shot manner? To answer the first question, we adopt a principled approach by exploiting Structure-from-Motion (SfM) for robot learning, developing a gradient-based optimization pipeline that extracts 3D hand trajectories from in-the-wild videos. Our pipeline combines learned monocular depth with geometric constraints, ensuring temporally consistent, metric-scale reconstructions. This allows for the recovery of contact points and smooth hand trajectories in 3D, serving as agent-agnostic 3D affordance representations. We introduce a coarse-to-fine affordance learning framework for the second question to learn rich actions from the diverse extracted training data. At the coarse prediction stage, our affordance model identifies high-level actions from pixels, i.e., contact points and goal points, based on RGB-D observations and task instructions. In the fine prediction stage, we employ a diffusion model to generate fine-grained interaction trajectories conditioned on coarse-stage outputs and task observations. Rather than relying solely on learned action priors from humans, we incorporate several differentiable cost functions as test-time sampling guidance [ 32 ] . These cost functions guide the diffusion denoising process by perturbing outputs to satisfy test-time constraints during deployment. Differentiable objectives, such as multi-goal reaching and collision avoidance, capture the distribution of coarse outputs and leverage geometric cues from the scene context. The guidance terms improve the plausibility of interactions by accounting for the variance in scene contexts and robot morphology while providing intuitive heuristics for selecting the optimal plans. We conducted extensive experiments in both simulation and real-world settings to evaluate the effectiveness of our framework. Utilizing in-the-wild 2D human videos only, our 3D affordance model outperformed several baselines trained using simulator exploration or pre-trained with teleoperated demonstrations. Furthermore, we demonstrated our model’s versatility in various downstream robot learning tasks, such as visual goal-reaching and exploration, showcasing rapid convergence to superior performance. 2 Related Work 2.1 Visual Affordance Learning Affordance centers around determining where and how an agent should interact with a given scene. One line of work regress affordance using manually annotated datasets [ 20 , 16 , 55 , 19 ] . However, collecting affordance labels is highly costly. Hence, a more recent line of work addressed this challenge by deploying agents in simulated environments to explore effective interaction [ 54 , 58 , 83 , 24 , 12 ] . Despite offering a data collection alternative without human intervention, these methods often suffer from the cost of obtaining diverse virtual assets. In contrast, human videos have gained attention as a more general source of affordance priors. Several approaches [ 48 , 27 , 4 , 56 ] predict per-pixel affordance scores by leveraging hand-object contact labels from human videos. However, these pipelines usually only identify contact regions or model interaction actions within the image plane, lacking spatial awareness. More recent works [ 92 , 7 ] attempted to address this limitation by utilizing flows as spatially-aware affordance representations. However, these