Point Policy: Unifying Observations and Actions with Key Points for Robot Manipulation

Siddhant Haldar, Lerrel Pinto · 2025

Read Point Policy: Unifying Observations and Actions with Key Points for Robot Manipulation by Siddhant Haldar, Lerrel Pinto (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2502-20391.txt.

Introduction

Siddhant Haldar and Lerrel Pinto published a paper in 2025 on teaching robots to handle objects by watching people, with no robot practice data at all. Gathering demonstrations through remote control is slow and costly, which starves robot learning. Their method, Point Policy, learns only from ordinary videos of human hands at work.

The system converts hand poses into robot poses and tracks objects through a set of meaningful key points. Because hands and grippers are both reduced to points, the same representation serves either body. On eight real tasks the approach beat earlier methods by a wide margin in matched settings. It also carried over to new objects and messy backgrounds, two classic failure points for video-trained robots.

Catalog record

CREATORSiddhant Haldar, Lerrel Pinto
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE36 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2502-20391

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Point Policy: Unifying Observations and Actions with Key Points for Robot Manipulation Siddhant Haldar, Lerrel Pinto Abstract Building robotic agents capable of operating across diverse environments and object types remains a significant challenge, often requiring extensive data collection. This is particularly restrictive in robotics, where each data point must be physically executed in the real world. Consequently, there is a critical need for alternative data sources for robotics and frameworks that enable learning from such data. In this work, we present Point Policy, a new method for learning robot policies exclusively from offline human demonstration videos and without any teleoperation data. Point Policy leverages state-of-the-art vision models and policy architectures to translate human hand poses into robot poses while capturing object states through semantically meaningful key points. This approach yields a morphology-agnostic representation that facilitates effective policy learning. Our experiments on 8 real-world tasks demonstrate an overall 75% absolute improvement over prior works when evaluated in identical settings as training. Further, Point Policy exhibits a 74% gain across tasks for novel object instances and is robust to significant background clutter. Videos of the robot are best viewed at point-policy.github.io . I Introduction Recent years have witnessed remarkable advancements in computer vision (CV) and natural language processing (NLP), resulting in models capable of complex reasoning [ 2 , 66 , 76 ] , generating photorealistic images [ 7 , 69 ] and videos [ 48 ] , and even writing code [ 15 ] . A driving force behind these breakthroughs has been the abundance of data scraped from the internet. In contrast, robotics has yet to experience a similar revolution, with most robots still confined to controlled or structured environments. While CV and NLP can readily take advantage of large-scale datasets from the internet, robotics is inherently interactive and requires physical engagement with the world for data acquisition. This makes collecting robot data significantly more challenging, both in terms of time and financial resources. A prominent approach for training robot policies has been the collection of extensive datasets, often through contracted teleoperators [ 53 , 12 , 71 ] , followed by training deep networks on these datasets [ 71 , 19 , 60 , 41 ] . While effective, these methods tend to require months or even years of human effort [ 12 , 41 ] and still result in datasets orders of magnitude smaller than those used in CV and NLP [ 60 , 41 ] . A potential solution to this data scarcity in robotics is to tap into the vast repository of human videos available online, showcasing individuals performing a wide range of tasks in diverse scenarios. The primary challenge in learning robot policies from human videos lies in addressing the morphology gap between robots and the human body [ 4 , 25 , 10 , 9 , 67 ] . Two notable trends have emerged in efforts to utilize human data for learning robot policies: (1) first learning visual representations or coarse policies from human datasets and then finetuning them for downstream learning on robot datasets [ 10 , 9 , 67 , 57 , 11 , 79 , 51 , 52 , 38 ] , and (2) using human videos to compute rewards for autonomous policy learning through reinforcement learning [ 81 , 4 , 25 , 43 ] . While the former requires a substantial amount of robot demonstrations to learn policies for downstream tasks, the latter often requires large amounts of online robot interactions in the real world, which can be time-consuming and potentially unsafe. In this work, we introduce Point Policy, a new technique to learn robot policies solely from offline human data without requiring robot interactions during training. Our key observation in building Point Policy is that both humans and robots occupy the same 3D space in the world, which can be tied together using key points derived from state-of-the-art vision models. Concretely, Point Policy works in three steps. First, given a dataset of human videos, a motion track of key points on the human hand and the object is computed using hand pose detectors [ 50 , 63 ] and minimal human annotation of one frame per task. These key points are computed from two camera views, which allows for projection in 3D using point triangulation. Second, a transformer-based policy [ 28 ] is trained to predict future robot points given the set of key points derived in the previous stage. Third, during inference, the predicted future robot points in 3D space are used to backtrack the 6 DOF pose of the robot’s end-effector using constraints from rigid-body geometry. The gripper state of the robot end effector is predicted as an additional token. The predicted end-effector pose and gripper state are then executed on the robot at 6 Hz. We demonstrate the effectiveness of Point Policy through experiments on 8 real-world tasks on a Franka robot. Our main findings are summarized below: 1. Point Policy exhibits an absolute improvement of 75% over prior state-of-the-art policy learning algorithms across 8 real world tasks when evaluated in identical settings as training. (Section V-E ). 2. Point Policy generalizes to novel object instances, exhibited a 74% absolute improvement over prior work on a held-out set of objects unseen in the training data. (Section V-F ). 3. Policies trained with Point Policy are robust to the presence of background distractors, performing at par with scenes without clutter (Section V-G ). 4. We provide an analysis of co-training Point Policy with teleoperated robot data (Section V-H ) and study the importance of several design choices in Point Policy (Section V-I ). All of our datasets, and training and evaluation code have been made publicly available. Videos of our trained policies can be seen here: point-policy.github.io . II Related Works II-A Imitation Learning Imitation Learning (IL) [ 33 ] refers to training policies with expert demonstrations, without requiring a predefined reward function. In the context of reinforcement learning (RL), this is often referred to as inverse RL [ 58 , 1 ] , where the reward function is derived from the demonstrations and used to train a policy [ 46 , 26 , 27 , 30 , 56 ] . While these methods reduce the need for extensive human demonstrations, they still suffer from significant sample inefficiency. As a result of this inefficiency in deploying RL policies in the real world, behavior cloning (BC) [ 65 , 75 , 70 , 68 ] has become increasingly popular in robotics. Recent advances in BC have demonstrated success in learning policies for both long-horizon tasks [ 13 , 54 , 73 ] and multi-task scenarios [ 28 , 8 , 61 , 10 , 9 ] . However, most of these approaches rely on image-based representations [ 82 , 28 , 14 , 8 , 61 , 35 ] , which limits their ability to generalize to new objects and function effectively outside of controlled lab environments. In this work, we propose Point Policy, which attempts to address this reliance on image representations by directly using key points as an input to the policy instead of raw images. Through extensive experiments, we observe that such an abstraction

Read the complete text