Annotation-Free One-Shot Imitation Learning for Multi-Step Manipulation Tasks

Vijja Wichitwechkarn, Emlyn Williams, Charles Fox, Ruchi Choudhary · 2025

Read Annotation-Free One-Shot Imitation Learning for Multi-Step Manipulation Tasks by Vijja Wichitwechkarn, Emlyn Williams, Charles Fox, Ruchi Choudhary (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2509-24972.txt.

Introduction

Vijja Wichitwechkarn, Emlyn Williams, Charles Fox, and Ruchi Choudhary taught a robot in 2025 to copy multi-step tasks from one demo, with no labels and no retraining. One-shot learning lets a robot pick up a skill from a lone show. Past tools did well on one-step tasks but stalled on long chains. Their method takes the one demo as it stands and breaks the job down on the fly.

On multi-step tasks the method won about eighty-three percent of runs. On one-step tasks it won ninety percent. Both marks met or beat the base lines it was set against. The team also weighed which pre-trained vision parts gave the best trade of skill and speed inside the frame.

Catalog record

CREATORVijja Wichitwechkarn, Emlyn Williams, Charles Fox, Ruchi Choudhary
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE36 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2509-24972

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Annotation-Free One-Shot Imitation Learning for Multi-Step Manipulation Tasks Vijja Wichitwechkarn, Emlyn Williams, Charles Fox, Ruchi Choudhary Abstract Recent advances in one-shot imitation learning have enabled robots to acquire new manipulation skills from a single human demonstration. While existing methods achieve strong performance on single-step tasks, they remain limited in their ability to handle long-horizon, multi-step tasks without additional model training or manual annotation. We propose a method that can be applied to this setting provided a single demonstration without additional model training or manual annotation. We evaluated our method on multi-step and single-step manipulation tasks where our method achieves an average success rate of 82.5% and 90%, respectively. Our method matches and exceeds the performance of the baselines in both these cases. We also compare the performance and computational efficiency of alternative pre-trained feature extractors within our framework. Project repository: https://github.com/vijja-w/AF-OSIL-MS I INTRODUCTION Recent advances in imitation learning have enabled robots to perform increasingly complex tasks. However, these methods still require hundreds to thousands of demonstrations per task [ 1 , 2 , 3 , 4 ] , making them impractical for real-world deployment. In contrast, humans can quickly acquire and deploy new skills. To bring robots closer to this level of deployability, it is crucial to minimize the effort required to acquire new manipulation skills. Ideally, robots should be able to learn and deploy new skills from a single demonstration, without the need for additional data collection, model training, or manual annotation. One-shot imitation learning (OSIL) offers this possibility by teaching a robot a new behavior from a single demonstration, with a large group of methods based on pose estimation combined with visual servoing and trajectory replay. These include methods that train models for segmentation and visual servoing [ 5 ] , and methods that train pose estimation models [ 6 ] . Canonical models of objects can also be trained to allow generalization to different objects within the same category [ 7 ] . OSIL can also be achieved without training models, using classical visual features with pre-trained matchers [ 8 ] , or correspondence algorithms with pre-trained features [ 9 ] . However, these methods typically assume the availability of object masks [ 6 , 8 , 7 ] and therefore require manual keypoint annotation for pre-trained segmentation models such as SAM2 [ 10 ] . Additionally, to extend replay to multistep tasks, demonstrations must be manually decomposed into multiple single-step demonstrations, which are then further annotated by selecting the bottleneck poses and segmenting the trajectories for replay. There are also other approaches to OSIL beyond trajectory replay. For example, trajectory optimization based on keypoints obtained through pre-trained models and heuristics has been used for OSIL [ 11 ] . World models can also be trained to predict latent trajectories that are decoded into physical waypoints for OSIL [ 12 ] . These methods however both require manual decomposition to be applied to multi-step tasks similar to trajectory replay based methods [ 11 , 12 ] . A human-to-robot alignment method has also been demonstrated for OSIL, requiring a vision encoder and action policy model to be trained [ 13 ] . Dynamical systems can also be applied to OSIL [ 14 ] , but this imposes strong priors such as attractors and smoothness constraints that limit their ability to capture topologically complex behaviors. An invariance-based method has also been shown to natively support arbitrary trajectories and multi-step tasks without additional annotation. This method however requires training a model that is used in an optimization-based framework to generate end-effector poses [ 15 ] . There are also few shot methods ( \sim\!10 demonstrations) that use large pre-trained models and do not require model training or manual annotations [ 16 ] . Variations improve upon this by relying on manual language annotations to provide additional context [ 17 ] . There are also methods that achieve few shot learning without manual annotation, but require models to be trained [ 18 , 19 ] . We propose a trajectory-replay-based imitation learning approach that requires only a single demonstration and does not depend on additional data collection, model training, or manual annotation. We demonstrate the effectiveness of our method on the manipulation of everyday objects, comparing against baselines on both multi-step and single-step tasks. In addition, we evaluate the performance of different pre-trained vision encoders within our framework and also evaluate their computational requirements in terms of runtime and memory usage. The source code can be found here: https://github.com/vijja-w/AF-OSIL-MS . II METHODS II-A Problem Formulation We consider the following formulation for one-shot imitation learning. We focus on a single robot arm with a wrist camera. A demonstration trajectory is defined as \tau\;=\;\{(x_{t},T_{t},g_{t})\}_{t=1}^{N}, (1) where T_{t}\in SE(3), (2) with x_{t} denoting the camera image observation, T_{t} the end-effector pose, g_{t} the gripper state, and N the total number of timesteps in the demonstration trajectory. We decompose the demonstration trajectory \tau into K ordered subtasks, each of which is a contiguous subsequence of the trajectory, using a decomposition function f_{\text{decompose}} : (\tau^{(1)},\tau^{(2)},\dots,\tau^{(K)})\;=\;f_{\text{decompose}}(\tau). (3) The function f_{\text{decompose}} selects K ordered, non-overlapping contiguous segments of the trajectory, possibly omitting intervals where no relevant action is taken, such as pauses. Each subtask \tau^{(k)} is defined as a contiguous subsequence \tau^{(k)}\;=\;\{(x_{t},T_{t},g_{t})\}_{t=s_{k}}^{e_{k}}, (4) where the start and end indices satisfy 1\leq s_{1}<e_{1}\leq s_{2}<e_{2}\leq\cdots\leq s_{K}<e_{K}\leq N. (5) Thus, the subtasks preserve the temporal order of the original trajectory, though they need not cover every timestep. For each subtask k , we select a keyframe index t_{k}^{\star}\in\{s_{k},\dots,e_{k}\} using a keyframe selector function f_{\text{keyframe}} : t_{k}^{\star}\;=\;f_{\text{keyframe}}\!\left(\tau^{(k)}\right), (6) and denote the corresponding state as x_{k}^{\star}:=x_{t_{k}^{\star}},\qquad T_{k}^{\star}:=T_{t_{k}^{\star}},\qquad g_{k}^{\star}:=g_{t_{k}^{\star}}. (7) Each subtask k is defined by two consecutive phases: the alignment phase and the execution phase. In the alignment phase , the robot is aligned to the selected keyframe by minimizing the discrepancy between the current camera image x and the keyframe image x_{k}^{\star} . This involves estimating a relative camera transform \Delta T^{\text{cam}}=f_{\text{align}}(x,x_{k}^{\star}), (8) where f_{\text{align}} is a visual alignment procedure. The relative motion in camera coordinates is converted into the end-effector frame as \Delta T^{\text{ee}}=T_{\text{cam},\text{ee}}\cdot\Delta T^{\text{cam}}\cdot T_{\text{cam},\text{ee}}^{-1}, (9) where the hand-eye calibration is given by a fixed transform T_{\text{cam},\text{ee}}\in SE(3) , denoting the camera pose expressed in the end-effector frame. The end-effector pose is updated as T\;\leftarrow\;T\cdot\Delta T^{\text{ee}}, (10) which is iterated until the alignment error between x and x_{k}^{\star} falls below a set threshold. This is followed by the execution phase where the robot replays the remaining demonstration actions of subtask k . Each action consists of a gripper command g_{j} and a relative motion \Delta T_{j} derived from the demonstration trajectory: \Delta T_{j}=T_{j}^{-1}T_{j+1},\qquad j=t_{k}^{\star},\dots,e_{k}-1. (11) This yields the control sequence, \mathbf{u}^{(k)}=\big(a_{t_{k}^{\star}},a_{t_{k}^{\star}+1},\dots,a_{e_{k}-1}\big),\qquad a_{j}=(g_{j},\Delta T_{j}). (12) These relative actions are applied sequentially starting from the pose obtained after the alignment phase: T\;\leftarrow\;T\cdot\Delta T_{j},\qquad g\;\leftarrow\;g_{j}. (13) Thus, each subtask is an atomic unit consisting of a closed-loop alignment to its keyframe followed by a replay of the corresponding action sequence. This is illustrated in Figure 1 . Although our formulation specifies control in terms of position waypoints, the same approach could also be realized with velocity commands. We choose waypoints because they are easy to interpret and compatible with standard controllers that

Read the complete text