AR-VRM: Imitating Human Motions for Visual Robot Manipulation with Analogical Reasoning

Dejie Yang, Zijing Zhao, Yang Liu · 2025

Read AR-VRM: Imitating Human Motions for Visual Robot Manipulation with Analogical Reasoning by Dejie Yang, Zijing Zhao, Yang Liu (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2508-07626.txt.

Introduction

Dejie Yang, Zijing Zhao, and Yang Liu teach robots to copy human hand movements, in a 2025 paper. Robot systems learn from language, camera views, and robot state, but robot training data are costly. Earlier work borrowed web data, which differs from robot tasks, or trained models to predict future video frames, which teaches the action only indirectly.

Their approach learns directly from large sets of human action videos. A vision-language model is first trained to predict human hand keypoints from video, so the action knowledge is direct. When fine-tuning on robot data, the system retrieves human clips with similar tasks and past views, then learns a map between human hand keypoints and robot parts. Focusing on keypoints over visual clutter gave leading scores on a standard benchmark and in real-world tests. In few-shot settings with little robot data, the method beat earlier approaches by wide margins.

Catalog record

CREATORDejie Yang, Zijing Zhao, Yang Liu
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE33 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2508-07626

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

AR-VRM: Imitating Human Motions for Visual Robot Manipulation with Analogical Reasoning Dejie Yang, Zijing Zhao, Yang Liu Abstract Visual Robot Manipulation (VRM) aims to enable a robot to follow natural language instructions based on robot states and visual observations, and therefore requires costly multi-modal data. To compensate for the deficiency of robot data, existing approaches have employed vision-language pretraining with large-scale data. However, they either utilize web data that differs from robotic tasks, or train the model in an implicit way (e.g., predicting future frames at the pixel level), thus showing limited generalization ability under insufficient robot data. In this paper, we propose to learn from large-scale human action video datasets in an explicit way (i.e., imitating human actions from hand keypoints), introducing Visual Robot Manipulation with Analogical Reasoning (AR-VRM). To acquire action knowledge explicitly from human action videos, we propose a keypoint Vision-Language Model (VLM) pretraining scheme, enabling the VLM to learn human action knowledge and directly predict human hand keypoints. During fine-tuning on robot data, to facilitate the robotic arm in imitating the action patterns of human motions, we first retrieve human action videos that perform similar manipulation tasks and have similar historical observations , and then learn the Analogical Reasoning (AR) map between human hand keypoints and robot components. Taking advantage of focusing on action keypoints instead of irrelevant visual cues, our method achieves leading performance on the CALVIN benchmark and real-world experiments. In few-shot scenarios, our AR-VRM outperforms previous methods by large margins , underscoring the effectiveness of explicitly imitating human actions under data scarcity. Code available at https://github.com/idejie/ar . 1 Introduction Visual robot manipulation (VRM) is an essential task in the robotics field [ 11 , 5 , 3 ] . Based on current state and visual inputs, the robot is required to execute some actions according to human natural language instructions, including tasks such as object grasping, placement, and assembly [ 23 , 2 , 1 , 31 ] . Training such task requires multi-modal data, including paired images, natural language instructions, and robot action states under specific scenarios where robots work, which is costly and requires time-consuming human demonstrations to manipulate robots. Therefore, the performance of VRM is often limited by the scarcity of robot trajectories and annotations for training [ 1 , 31 ] . (a) Previous methods: learning from large-scale data implicitly. (b) Our AR-VRM: explicitly learn from human action by hand keypoints. Figure 1 : Demonstration of the differences between our framework and previous methods: we propose to learn from human actions explicitly by hand keypoints with analogical reasoning. To compensate for the deficiency of robot data, existing approaches utilize large-scale vision-language data pretraining and fine-tune the model under robot scenarios. But, these pretraining data do not directly reflect object manipulation tasks, such as visual-question-and-answer datasets about animals and diets, and it is hard to provide relevant knowledge guidance for robot manipulation tasks. Some studies [ 20 , 31 ] have utilized human action video datasets that are more similar to robotic manipulation tasks for pretraining, but they train the model in implicit ways, either by contrastive learning in feature space or by predicting future frames with pixel-level generative models. Though acquiring human action knowledge, these approaches inevitably introduce irrelevant background information or pixel-level noise, which limits their performance on data-scarce VRM tasks. In this paper, we propose to imitate human motions explicitly from a large-scale human action video dataset, training the model with human hand keypoints to learn from the motion itself, ignoring the irrelevant visual information. To achieve this goal, two main challenges exist: (1) How to extract human action knowledge in terms of hand keypoints from large-scale human video datasets? (2) How to train the robot, which differs from the human arm, to imitate human actions, i.e., building the correlation between the robotic components and human keypoints for the manipulation task? To address these challenges in learning from human actions, we propose Visual Robot Manipulation with Analogical Reasoning (AR-VRM). Human hand keypoints and robotic actions share underlying similarities, especially when it comes to manipulating objects. Specifically, to extract action knowledge explicitly from a large-scale human video dataset, we propose a keypoint Visual-Language Model (VLM) pretraining scheme. We use a large-scale egocentric human action video dataset Ego4D [ 7 ] with common operation videos by human hands, which have similar manipulation tasks and environment views to robotic applications. We detect human hand keypoints in instruction videos, and pretrain a VLM to predict the keypoints of future actions based on the current visual inputs and language instructions. During the fine-tuning stage, to provide demonstrations under manipulation tasks, we first retrieve relevant human action videos that have similar history observations to the robot situation. Moreover, to bridge the gap between robotic arm and human, we propose an Analogical Reasoning (AR) map between robotic arm components and human hand keypoints to learn geometric and functional correlation . Learning this correspondence helps to guide the robot to imitate human actions explicitly, e.g., the process of approaching the object and the grasping operation, to achieve manipulation tasks. We perform extensive experiments on the challenging CALVIN benchmark. Our AR-VRM achieves state-of-the-art performance under every experimental setting. Specifically, under 10% robot training data as well as unseen scenarios, our method outperforms previous approaches by large margins, improving success rate from 40.0% to 45.6% under 10% training data, and from 61.2% to 65.9% under unseen scenes. These results demonstrate that our idea of explicitly imitating human actions by keypoints has stronger generalization ability under limited robot data and under new scenarios. In addition, our model also outperforms previous SOTA methods on real robot experiments. We conclude the contributions of our paper as follows: (1) We propose AR-VRM, the first visual robot manipulation approach that explicitly imitates human actions in terms of hand keypoints from large-scale human action video datasets. (2) We propose the keypoint vision language model pretraining scheme to extract action knowledge from human videos, and the analogical reasoning module to align robotic arms with human hand keypoints. (3) Our method achieves state-of-the-art performance on CALVIN benchmark and real-world robot experiments, especially under limited robot training data, demonstrating the effectiveness and generalization ability. 2 Related Work 2.1 Visual Robot Manipulation The visual robot manipulation task aims to train the robot to follow natural language instructions. It is a flexible and intuitive way for non-expert humans to instruct the robot to execute tasks such as object grasping, placement, and assembly [ 11 , 5 , 3 ] . Some pioneer approaches delved into diverse visual encoder pre-training methods, aiming to learn useful visual representations by masked image modeling [ 23 , 24 ] and contrastive learning [ 20 , 10 ] , and reward signals for reinforcement learning [ 4 ] . In recent years, some studies introduced large language models as the action planner [ 28 , 6 ] . ATM [ 30 ] proposes to imitate human actions for robot manipulation tasks, but requires paired human-robot action data under specific operation scenarios, limiting its generalization potential. These methods all have limited performance due to the insufficient multimodal data, which is difficult to obtain and costly to collect. In this paper, we employ large-scale human

Read the complete text