One-Shot Dual-Arm Imitation Learning
Yilong Wang, Edward Johns · 2025
Read One-Shot Dual-Arm Imitation Learning by Yilong Wang, Edward Johns (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2503-06831.txt.
Introduction
Yilong Wang and Edward Johns present a 2025 paper on teaching two-armed robots a task from a single demonstration. Precise, coordinated two-arm tasks, such as inserting or folding, usually need many examples or special setup. Their system learns from just one demo, with no prior knowledge of the task or objects.
A three-stage visual alignment method first lines up the gripper and target with care. After that, replaying the demo path is enough to do the task. A new coordination scheme keeps the two arms in step. On a real robot, the system handled six precise tasks well and coped with clutter and partial blocking of the view.
Catalog record
| CREATOR | Yilong Wang, Edward Johns |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 34 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2503-06831
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
One-Shot Dual-Arm Imitation Learning Yilong Wang, Edward Johns Abstract We introduce One-Shot Dual-Arm Imitation Learning (ODIL), which enables dual-arm robots to learn precise and coordinated everyday tasks from just a single demonstration of the task. ODIL uses a new three-stage visual servoing (3-VS) method for precise alignment between the end-effector and target object, after which replay of the demonstration trajectory is sufficient to perform the task. This is achieved without requiring prior task or object knowledge, or additional data collection and training following the single demonstration. Furthermore, we propose a new dual-arm coordination paradigm for learning dual-arm tasks from a single demonstration. ODIL was tested on a real-world dual-arm robot, demonstrating state-of-the-art performance across six precise and coordinated tasks in both 4-DoF and 6-DoF settings, and showing robustness in the presence of distractor objects and partial occlusions. Videos are available at: https://www.robot-learning.uk/one-shot-dual-arm . I Introduction Efficient dual-arm robot manipulation remains very challenging in robotics, particularly for tasks that demand precise and coordinated actions. Recent advancements in imitation learning [ 1 , 2 ] have significantly improved the acquisition of visuomotor skills through human demonstrations. However, these methods are often data-intensive, due to the high dimensionality of the state-action space. For dual-arm manipulation, tasks typically require fifty [ 1 ] to thousands [ 3 ] of demonstrations per task. Furthermore, learning dual-arm coordination from human demonstrations remains an open problem due to the complex spatial and temporal correlations [ 4 ] . This leads to our first research question: (1) Can a dual-arm robot learn precise and coordinated tasks from a single demonstration? To achieve one-shot efficiency, imitation learning can be modeled as an alignment between the end-effector (EE) and target object, followed by trajectory transfer or replay of the demonstration trajectory [ 5 , 6 , 7 ] . Within this paradigm, the main challenge now is to extract a coordinated dual-arm trajectory from a single demonstration, and to perform precise alignment without the need for extensive real-world data collection following the demonstration. Alignment can be achieved using visual servoing to align the EE and the target object in the same way as observed during the demonstration. To avoid the need for prior object knowledge and CAD models for pose estimation, [ 5 , 7 ] performed visual servoing using self-supervised learning. However, such methods require significant additional data collection beyond the demonstration. This leads to two further research questions: (2) How can a coordinated trajectory be derived from a single demonstration? (3) How can precise and robust visual servoing be achieved without relying on object models or additional data collection? Fig. 1: ODIL learns precise, coordinated tasks from a single demonstration (first column) and adapts to novel robot, object, and scene configurations. Starting from an arbitrary position (second column), the robot first aligns with the object (third column), and then replays the demonstration (last column). Fig. 2: Overall Framework. During demonstration, the end-effector E is first moved to a bottleneck {B} , with the task object O visible to both the global and wrist cameras, {C}_{g} and {C}_{w} , respectively. Then, from each camera, a bottleneck RGB-D image is captured and segmented, and a demonstration trajectory is recorded and parameterized into a coordinated trajectory \boldsymbol{\tau} . During testing, starting from an arbitrary initial robot pose where O is visible to {C}_{g} , we use our 3-VS controller to align E with the new {B} , then execute \boldsymbol{\tau} to complete the task. Through answering questions (1) to (3), we introduce O ne-shot D ual-arm I mitation L earning (ODIL), a method that enables a dual-arm robot with a basic position controller to learn precise, coordinated tasks from a single demonstration (Figure 1 ). At its core, ODIL features a novel Dual-Arm Coordination Paradigm , which parametrizes a coordinated object interaction trajectory from the demonstration, and a Three-Stage Visual Servoing (3-VS) controller for highly precise and robust visual alignment. Once servoing is complete, the parameterized dual-arm trajectory is adapted and executed. 3-VS relies solely on one eye-to-hand global camera and one eye-in-hand wrist camera, both with only approximate hand-eye calibration. It is inspired by recent advances in deep feature matching [ 8 , 9 , 10 ] . Unlike modern VS methods requiring object-specific pre-training [ 5 ] or simulation-based training [ 11 ] , and those that do not model outliers [ 12 , 13 , 14 ] , we revisit the fundamentals of analytic VS [ 15 , 16 , 17 , 18 ] and show that combining this with task-agnostic deep matchers leads to the best performance. In our experiments, we rigorously benchmarked recent deep feature matching methods from the computer vision community for visual servoing. We then compared our imitation learning method to state-of-the-art methods [ 13 , 14 ] , achieving significantly better performance in 4-DoF and 6-DoF real-world tasks across diverse scenes. II Related Work Dual-arm Imitation Learning involves a high-dimensional action space, which is often managed by parameterized movement primitives [ 19 , 20 ] . However, these primitives rely on labor-intensive, hand-crafted motions, making them difficult to scale for everyday tasks [ 21 ] . In contrast, recent advances in dual-arm teleoperation [ 1 , 22 , 23 , 24 ] have enabled the collection of high-quality demonstrations suitable for end-to-end training. This progress has allowed researchers to extend dual-arm imitation learning to impressive fine-manipulation tasks such as slotting a battery [ 1 ] , cooking shrimp [ 25 ] , and preparing tea [ 26 ] . Nevertheless, these methods require numerous human demonstrations, along with specialized hardware for collecting them, whereas our method needs only a single demonstration with minimal resources. One-Shot Imitation Learning addresses the data scarcity problem by attempting to learn tasks from a single demonstration. [ 27 , 28 ] propose meta-learning for one-shot imitation but need pre-training on similar tasks. Instead, more relevant to our work, [ 5 , 7 ] employ self-supervised learning to train a VS to reach a bottleneck state, followed by executing the demonstration trajectory. Such a VS can be trained using CNNs [ 5 ] or Siamese CNNs [ 29 ] . However, these methods require extensive real-world data collection per object instance. [ 30 , 11 ] address this issue by training the VS entirely in simulation, but suffer from the sim-to-real gap. [ 12 ] uses learning-based optical flow for imitation learning. [ 6 ] trains an unseen object pose estimator, [ 31 ] trains a shape warper, and [ 32 , 33 ] train a graph neural network for in-context learning, but these operate only on noisy point clouds and so cannot perform precise tasks. More recently, keypoint representations from pre-trained visual foundation models, such as DINO-ViT [ 34 ] and TAPIR [ 35 ] , have been applied to VS, eliminating the need for task-specific training and achieving state-of-the-art performance in one-shot and few-shot manipulation across various everyday tasks [ 13 , 36 , 14 ] . However, variability and outliers among keypoints can overwhelm the robot controllers, causing oscillations and inaccuracies. In contrast, our method integrates pre-trained deep matchers [ 8 ] , designed to reject outliers, with robust VS techniques [ 16 , 17 ]