HAND Me the Data: Fast Robot Adaptation via Hand Path Retrieval
Matthew Hong, Anthony Liang, Kevin Kim, Harshitha Rajaprakash, Jesse Thomason, Erdem Bıyık, Jesse Zhang Thomas Lord Department of Computer Science,University of Southern California · 2025
Read HAND Me the Data: Fast Robot Adaptation via Hand Path Retrieval by Matthew Hong, Anthony Liang, Kevin Kim, Harshitha Rajaprakash, Jesse Thomason, Erdem Bıyık, Jesse Zhang Thomas Lord Department of Computer Science,University of Southern California (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2505-20455.txt.
Introduction
Matthew Hong and six co-authors present HAND, a fast way to teach robots new tasks from human hand demos. Their twenty twenty-five paper avoids the slow norm: collecting robot demos by remote control for each new task. A person just shows the task with their own hand, on camera.
HAND tracks the hand's path and searches a pool of unlabeled robot play data in two steps, first by look, then by behavior. It fine-tunes a policy on the matched clips. New tasks were learned in under four minutes, with no calibrated cameras or fine hand pose data. On real robots it more than doubled the success rate of past retrieval methods.
Catalog record
| CREATOR | Matthew Hong, Anthony Liang, Kevin Kim, Harshitha Rajaprakash, Jesse Thomason, Erdem Bıyık, Jesse Zhang Thomas Lord Department of Computer Science,University of Southern California |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 52 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2505-20455
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
HAND Me the Data: Fast Robot Adaptation via Hand Path Retrieval Matthew Hong, Anthony Liang, Kevin Kim, Harshitha Rajaprakash, Jesse Thomason, Erdem Bıyık, Jesse Zhang Thomas Lord Department of Computer Science,University of Southern California Abstract We hand the community HAND, a simple and time-efficient method for teaching robots new manipulation tasks through human hand demonstrations. Instead of relying on task-specific robot demonstrations collected via teleoperation, HAND uses easy-to-provide hand demonstrations to retrieve relevant behaviors from task-agnostic robot play data. Using a visual tracking pipeline, HAND extracts the motion of the human hand from the hand demonstration and retrieves robot sub-trajectories in two stages: first filtering by visual similarity, then retrieving trajectories with similar behaviors to the hand. Fine-tuning a policy on the retrieved data enables real-time learning of tasks in under four minutes, without requiring calibrated cameras or detailed hand pose estimation. Experiments also show that HAND outperforms retrieval baselines by over 2 × 2\times in average task success rates on real robots. Videos can be found at our project website: https://liralab.usc.edu/handretrieval/ . † † *Equal Contribution, †Equal Advising Keywords: Robot Adaptation, Retrieval, Play Data, Hand Demonstration 1 Introduction Figure 1: HAND learns a policy from as little as one (1) human hand demonstration. Robots deployed in homes, warehouses, and other dynamic, human-centric settings will need to quickly learn many tasks specified by end-users. To support this goal, robot learning algorithms for these settings must (1) scale easily across many tasks and (2) enable fast adaptation for each new task. While imitation learning has shown promise in producing capable multi-task robot policies [ 1 , 2 , 3 , 4 , 5 ] , it remains difficult to scale due to its reliance on large amounts of expert-collected, task-specific teleoperation data. In contrast, task-agnostic play data —collected through free-form robot teleoperation [ 6 , 7 , 8 ] —is much easier to gather, as it does not require constant environment resets or task-specific labeling. The key challenge is making this data usable for teaching robots new tasks quickly. In this work, we tackle this problem by retrieving relevant robot behaviors from the play dataset using only a single human hand demonstration. We propose HAND, a simple and time-efficient approach that adapts pre-trained play policies to specific tasks using just one human hand demonstration (see Figure 1 ). Prior work on robot behavior retrieval relies on additional teleoperated target-task robot demonstrations [ 9 , 10 , 11 , 12 , 13 ] , which are difficult for non-expert users to provide. Instead, our key insight is to extract coarse guidance from the hand demonstration—specifically, 2D relative hand motion paths—to retrieve diverse yet relevant behaviors from the play dataset. Calling back to the motivation, we aim for HAND to be scalable and fast . Towards scalability, HAND avoids the need for calibrated depth cameras [ 14 , 15 ] , specialized eye-in-hand setups [ 16 ] , or detailed hand-pose estimation [ 16 , 17 ] . Instead, it first labels a robot play dataset with 2D gripper positions relative to the RGB camera frame by tracking the gripper using a visual point-tracking model [ 18 , 19 ] . When a human hand demonstration is provided, HAND tracks the hand trajectory with the same simple pipeline. The hand positions are then converted into 2D relative sub-trajectories, capturing motion independent of the starting point [ 20 ] . After an initial filtering step that removes unrelated behaviors using a visual foundation model [ 21 ] , HAND retrieves matching sub-trajectories from the play dataset based on the 2D relative hand path. Finally, for fast learning, a policy pre-trained on the play dataset is LoRA-fine-tuned on the retrieved sub-trajectories, encouraging the policy to specialize in the demonstrated task. Because HAND retrieves primarily based on hand motion, it is more robust to irrelevant visual features such as background clutter and lighting changes compared to purely visual retrieval methods. Our experiments, both in simulation in CALVIN [ 8 ] and in the real world on a WidowX robot, demonstrate that HAND enables quick adaptation to 8 diverse downstream tasks. Notably, HAND outperforms the best baseline by 2 \times on the real robot. We also demonstrate that HAND works with hand demos collected from completely different scenes from the robot’s. Finally, we perform a real-time learning experiment, where HAND learns a challenging long-horizon task in under 4 minutes of experiment time, from providing the hand demonstration to the trained policy, while being on average 5 \times faster to collect data for than robot teleoperation demos on our WidowX arm. 2 Related Works Robot Data Retrieval. Prior work has demonstrated retrieval as an effective mechanism for extracting relevant on-robot data for training robots [ 9 , 10 , 11 , 12 , 13 , 22 ] . For example, SAILOR [ 9 ] and Behavior Retrieval [ 10 ] pre-train variational auto-encoders (VAEs) on prior robot images and actions to learn a latent embedding. This latent embedding is used to retrieve states and actions from an offline dataset similar to ones provided in expert demonstration trajectories. However, retrieving based on learned full image encodings or even raw pixel values [ 13 ] can be noisy; Flow-Retrieval [ 11 ] instead trains a VAE to encode optical flows indicating movement of objects and the robot arm in the scene. Similar to Flow-Retrieval, our method HAND also retrieves based on robot arm movement. However, rather than training a dataset-specific VAE model that may not be robust to large visual differences, we retrieve from our offline robot data by primarily matching motions of a human hand demonstration using relative 2D paths of the robot end-effector in the prior data. This hand path retrieval helps us robustly retrieve relevant robot arm behaviors . STRAP [ 12 ] addresses visual retrieval robustness issues of prior work by using features from DINO-v2 [ 21 ] , a large pre-trained image-input foundation model for retrieval. However, STRAP, along with all aforementioned retrieval work, assumes access to expert robot demonstrations for the target tasks. HAND on the other hand, only requires a single , easier-to-collect human hand demonstration that results in more time-efficient learning of demonstrated tasks compared to methods requiring robot teleoperation data for retrieval. Moreover, experiments demonstrate HAND actually retrieves more task-relevant trajectories and therefore attains higher success rates compared to these methods. Learning From Human Hands. Similar to HAND, a separate line of work has proposed methods to use human hands to learn robot policies. One approach is to train models on human video datasets to predict future object flows [ 23 , 24 ] or human affordances [ 25 , 26 ] . These intermediate affordance and flow representations are then used to either train a policy conditioned on this representation [ 23 ] on robot data or control a heuristic policy [ 24 , 25 , 26 ] . Other works focus on learning directly from human hands [ 14 , 15 , 16 , 27 , 17 ] . These works generally use hand-pose detection models aided by multiple cameras or calibrated depth cameras to convert hand poses directly to robot gripper keypoints [ 14 , 15 , 17