Parse-Augment-Distill: Learning Generalizable Bimanual Visuomotor Policies from Single Human Video

Georgios Tziafas, Jiayun Zhang, Hamidreza Kasaei · 2025

Read Parse-Augment-Distill: Learning Generalizable Bimanual Visuomotor Policies from Single Human Video by Georgios Tziafas, Jiayun Zhang, Hamidreza Kasaei (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2509-20286.txt.

Introduction

Teaching a two-armed robot from one human video is the goal of a 2025 arXiv paper by Georgios Tziafas, Jiayun Zhang, and Hamidreza Kasaei. Collecting robot demonstrations by teleoperation is slow and costly. Human videos are plentiful but hard for robots to use directly. Their framework, PAD, bridges that gap in three steps.

First, it parses the video into a keypoint-action path a robot can execute. Next, task-and-motion planning multiplies that single demonstration into many variants, with no simulator needed. Finally, the augmented paths are distilled into a policy that acts on keypoints. PAD beat leading augmentation methods on success rate and cost in their tests. Deployed tasks included pouring drinks, clearing trash, and opening containers. The learned policies handled new layouts, objects, and background clutter.

Catalog record

CREATORGeorgios Tziafas, Jiayun Zhang, Hamidreza Kasaei
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE53 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2509-20286

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Parse-Augment-Distill: Learning Generalizable Bimanual Visuomotor Policies from Single Human Video Georgios Tziafas, Jiayun Zhang, Hamidreza Kasaei Abstract Learning visuomotor policies from expert demonstrations is an important frontier in modern robotics research, however, most popular methods require copious efforts for collecting teleoperation data and struggle to generalize out-of-distribution. Scaling data collection has been explored through leveraging human videos, as well as demonstration augmentation techniques. The latter approach typically requires expensive simulation rollouts and trains policies with synthetic image data, therefore introducing a sim-to-real gap. In parallel, alternative state representations such as keypoints have shown great promise for category-level generalization. In this work, we bring these avenues together in a unified framework: PAD (Parse-Augment-Distill), for learning generalizable bimanual policies from a single human video. Our method relies on three steps: (a) parsing a human video demo into a robot-executable keypoint-action trajectory, (b) employing bimanual task-and-motion-planning to augment the demonstration at scale without simulators, and (c) distilling the augmented trajectories into a keypoint-conditioned policy. Empirically, we showcase that PAD outperforms state-of-the-art bimanual demonstration augmentation works relying on image policies with simulation rollouts, both in terms of success rate and sample/cost efficiency. We deploy our framework in six diverse real-world bimanual tasks such as pouring drinks, cleaning trash and opening containers, producing one-shot policies that generalize in unseen spatial arrangements, object instances and background distractors. Supplementary material can be found in the project webpage https://gtziafas.github.io/PAD_project/ . Fig. 1: PAD framework overview: Given one human video demonstration, PAD executes three subsequent steps: (a) parsing the video into a robot-executable state-action trajectory, (b) spatially augmenting the demo at scale via bimanual TAMP, and (c) distilling the generated data into a closed-loop keypoint policy. The obtained policy can generalize to unseen spatial arrangements, object instances and background scene noise. I INTRODUCTION Visuomotor policy learning for robot manipulation has seen great success in recent years [ 1 , 2 , 3 , 4 , 5 ] , yet it typically demands costly and time-consuming data collection from expert demonstrators. This data-hungriness stems from the different required axes of generalization: a competent policy must generalize in unseen object arrangements ( spatial ) and object instances ( object ), as well as be robust to environmental conditions such as scene background, camera placement etc. ( background ). As a result, most common policies struggle to generalize in out-of-distribution scenarios where corresponding data has not been collected. A recent methodology to tackle this data scarcity is to tap into the vast repository of videos available in the web, showcasing humans interacting with objects in diverse scenarios [ 6 , 7 , 8 , 9 , 10 ] . Here the main challenge is bridging the embodiment gap between humans and robot morphologies [ 11 , 12 , 13 ] . Alternatively, a recent line of works aims at dealing with spatial generalization by augmenting a small number of source demos with structured, object-centric task-and-motion planning (TAMP) procedures [ 14 , 15 , 16 , 17 ] . However, most works train image policies that require calibrated digital twins and expensive on-robot rollouts to generate the augmentations, therefore introducing a visual sim-to-real gap, while still struggling with object and background generalization. When it comes to bimanual manipulation, additional considerations related to arm collaboration strategies for different task scenarios further complicate data collection / generation. In this work we wish to tackle these challenges by proposing PAD ( P arse- A ugment- D istill), a unified framework for learning bimanual visuomotor policies from a single human video demonstration. Our framework works in three steps (see Fig. 1 ): (a) parsing the video into robot-executable data, (b) augmenting the data in a simulation-free fashion and, (c) distilling the augmented data into a closed-loop policy. In our work, we explicitly seek spatial, object and background generalization. To accommodate this, we utilize 3D keypoint coordinates as state representations for our trained policy, which offers three important advantages: First, keypoints abstract the visual scene into a low-dimensional geometric representation, which is task-specific and decoupled from object semantics, and therefore has empirically shown to aid in sample-efficiency and robustness to background noise [ 18 , 13 , 19 ] . Second, keypoints facilitate category-level object generalization, inherited by the open-world capabilities of pretrained vision models for identifying semantic correspondences [ 20 , 21 , 22 ] . Finally, 3D point states enable efficient spatial augmentations, as keypoint coordinates can be computed on-the-fly through 3D rigid geometry assumptions [ 23 ] . This alleviates the need for a digital twin and expensive simulation rollouts, which would be required by a typical image policy to obtain image observations [ 14 , 15 , 16 , 17 ] . In turn, this significantly improves data collection time and bridges the sim-to-real gap introduced by simulators. Concretely, in PAD we introduce a general TAMP framework for spatial demo augmentations, specialized for bimanual manipulation. To that end, we introduce bimanual task templates, symbolic representations that declare information about each arm’s object assignments, involved contacts and requirements for arm synchronization, while abstracting away the specific semantics of the task. We particularly focus on handling issues related to bimanual manipulation, such as out-of-range arm-object assignments and re-synchronization between the arms during motion planning, which are missing from previous works in bimanual demo augmentation [ 17 ] . Our augmentation framework is general, cost-efficient and embodiment-agnostic, as it uses human video as the source demo that can be mapped to any given morphology. Finally, we use prescribed 3D keypoints as our state representation instead of RGB images or point-clouds, and accompany them with augmentations that aid the policy in object generalization. To distill the augmented data, we introduce Kp-RDT, an adapted version of RDT [ 4 ] for learning bimanual diffusion policies with keypoint conditioning. Empirically, we show that our framework outperforms state-of-the-art bimanual demo augmentation methods [ 17 ] in four simulation tasks from the DexMimicGen robosuite benchmark [ 24 ] , both in terms of success rates, as well as sample-efficiency and data generation time. We further apply our framework in six diverse real-world tasks and show that PAD obtains policies that generalize to unseen spatial arrangements, object instances and background scene noise, while doing so from a single human demonstration. In summary, our contributions with this work are threefold: • We introduce PAD, a unified framework for generalizable bimanual policy learning from a human video. • We propose a general bimanual TAMP framework for spatial demo augmentations, applicable to a wide variety of manipulation skills and arm-coordination strategies, as well as open-ended object categories. • We perform extensive robot experiments in 10 tasks, 4 in simulation and 6 with hardware, demonstrating significant gains compared to previous works in terms of success rates and sample/cost efficiency, as well as strong generalization in real-world tasks. II RELATED WORK Learning from egocentric video. Many recent datasets [ 6 , 7 , 8 , 9 , 10 ] represent large-scale efforts to collect egocentric hand-object interaction videos in diverse real-world scenes. Such datasets have been used for learning robotics-tailored visual representations as a pretraining step [ 25 , 26 , 27 , 28 , 29 , 30 ] . Other works learn coarse policies from human videos,

Read the complete text