RoboPCA: Pose-centered Affordance Learning from Human Demonstrations for Robot Manipulation

Zhanqi Xiao, Ruiping Wang, Xilin Chen · 2026

Read RoboPCA: Pose-centered Affordance Learning from Human Demonstrations for Robot Manipulation by Zhanqi Xiao, Ruiping Wang, Xilin Chen (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2603-07691.txt.

Introduction

Xiao, Wang and Chen propose RoboPCA, pose-first affordance learning for robot handling from human demonstrations. Existing methods locate contact regions but leave poses to separate guessing, causing failures when regions and poses disagree. RoboPCA predicts contact regions and poses together, matched to the task, from instructions. A Human2Afford pipeline rebuilds 3D scene information and infers skill labels from demos, and the system beats baselines across images, simulation and real robots.

Catalog record

CREATORZhanqi Xiao, Ruiping Wang, Xilin Chen
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE35681
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2603.07691

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

RoboPCA: Pose-centered Affordance Learning from Human Demonstrations for Robot Manipulation Zhanqi Xiao, Ruiping Wang, Xilin Chen Abstract Understanding spatial affordances—comprising the contact regions of object interaction and the corresponding contact poses—is essential for robots to effectively manipulate objects and accomplish diverse tasks. However, existing spatial affordance prediction methods mainly focus on locating the contact regions while delegating the pose to independent pose estimation approaches, which can lead to task failures due to inconsistencies between predicted contact regions and candidate poses. In this work, we propose RoboPCA, a pose-centered affordance prediction framework that jointly predicts task-appropriate contact regions and poses conditioned on instructions. To enable scalable data collection for pose-centered affordance learning, we devise Human2Afford, a data curation pipeline that automatically recovers scene-level 3D information and infers pose-centered affordance annotations from human demonstrations. With Human2Afford, scene depth and the interaction object’s mask are extracted to provide 3D context and object localization, while pose-centered affordance annotations are obtained by tracking object points within the contact region and analyzing hand–object interaction patterns to establish a mapping from the 3D hand mesh to the robot end-effector orientation. By integrating geometry–appearance cues through an RGB-D encoder and incorporating mask-enhanced features to emphasize task-relevant object regions into the diffusion-based framework, RoboPCA outperforms baseline methods on image datasets, simulation, and real robots, and exhibits strong generalization across tasks and categories. I INTRODUCTION Fig. 1: Overview of our pipeline. (a) Pose-centered affordance annotations and complementary scene information are extracted from human demonstrations with Human2Afford for pose-centered affordance learning. (b) RoboPCA builds upon a diffusion framework to predict pose-centered affordances, an RGB-D encoder is used to effectively capture both geometry and appearance cues, and mask-enhanced features are incorporated to emphasize task-relevant object regions. (c) The predicted pose-centered affordances are transformed into 6-DoF poses using camera parameters, guiding the robot to complete the task. Humans excel at manipulating various objects in unstructured environments, including locating task-related interaction regions and selecting appropriate contact poses. This ability is largely ascribed to the profound understanding of spatial affordances [ 13 , 14 ] . As robots and humans operate in the same workspace, a comprehensive understanding of spatial affordances is crucial for enhancing robotic manipulation capabilities across diverse object categories and tasks. Affordance had already been extensively studied before its application to robotic manipulation, which is often represented as masks or heatmaps [ 35 , 41 , 30 ] . While such representation can indicate potential interaction regions on objects, it lacks the precise spatial localization required for accurate robot manipulation. To address this, recent works propose representing affordances in terms of contact points and post-contact trajectories [ 1 ] . In this representation, the interaction region is denoted by a 2D pixel to specify where the robot should make contact with the object, while the post-contact trajectories are represented by a 2D vector on the image to indicate the direction and movement the robot should execute after making contact. Building on this representation, subsequent works have made many improvements, such as introducing object retrieval or leveraging the reasoning capability of foundation models for generalization across categories [ 20 , 24 , 26 , 48 , 31 ] . However, contact points and post-contact trajectories alone do not specify the manipulator pose for task execution. Although the final pose could be obtained by filtering the grasp candidates from independent pose estimation approaches [ 11 , 40 ] based on the predicted contact point, the inconsistency between predicted contact points and grasp candidates may lead to suboptimal or even failed executions. We argue pose-centered affordances , represented jointly by the contact points and the corresponding contact poses , provide a more coherent and expressive formulation for robotic manipulation. By unifying contact localization and pose estimation, this formulation reduces the inconsistency between predicted contact points and grasp candidates and provides a principled basis for generating reliable manipulation strategies. Similar to other tasks in robot learning, learning pose-centered affordances that support the manipulation of arbitrary objects requires large-scale data. While recent works, such as DROID [ 22 ] , have gathered large-scale robotic demonstrations via human teleoperation, this approach remains difficult to scale to new environments or task demands. In contrast, human demonstrations provide a promising data source, not only due to the scale and diversity of scenes and tasks, but also because they naturally capture dynamics relevant to object interactions. Nevertheless, the absence of 3D information and low-level action labels limits their utility for pose-centered affordance learning, particularly for extracting contact poses. In this work, we aim to enable pose-centered affordance learning from a large amount of unlabeled human demonstrations, tackling two key questions: (1) How can pose-centered affordances be extracted from unlabeled human demonstrations? (2) How to effectively learn pose-centered affordances from collected datasets? To answer the first question, we devise Human2Afford , a data curation pipeline that automatically recovers scene-level 3D information and extracts pose-centered affordances from human demonstrations. With each identified pre-contact frame and contact frame pair, Human2Afford first recovers the depth information and extracts the interaction object’s mask in the pre-contact frame to provide 3D context and object localization to facilitate pose-centered affordance learning. To obtain pose-centered affordance annotations, we first recover the contact pose by analyzing the hand–object interaction patterns to establish a mapping between the estimated 3D hand mesh in the contact frame and the robot end-effector’s orientation, and then derive the contact point by tracking object pixels within the contact region in the contact frame back to the pre-contact frame. We collected 10K human–object interaction images with pose-centered affordance annotations and complementary scene information using Human2Afford. Based on the collected dataset, we propose RoboPCA for the second question, which is a P ose- C entered A ffordance prediction framework that jointly predicts task-appropriate contact point and corresponding contact pose given instructions. RoboPCA builds upon a diffusion framework to predict pose-centered affordances. To effectively capture both geometry and appearance cues, it leverages a state-of-the-art RGB-D encoder [ 47 ] that integrates color and depth information. Additionally, mask-enhanced features are incorporated to emphasize task-relevant object regions, improving the model’s ability to localize interaction points and accurately infer corresponding contact poses. The overview of our pipeline is shown in Fig. 1 . We evaluate RoboPCA through extensive experiments on image datasets, simulation, and real-world scenarios. It outperforms baseline methods in contact point prediction precision and manipulation task success rates, achieving improvements of 18.6% on AGD20K [ 29 ] evaluated on object categories feasible for robotic manipulation, 38.5% on RLBench [ 18 ] , and 24.9% in real-world experiments, which demonstrates that RoboPCA not only achieves higher precision in predicting contact points but also provides more reliable and effective guidance for manipulation tasks. We also demonstrate its compatibility with robotic data. II RELATED WORK Fig. 2: Overview of Human2Afford. (a) Given a human demonstration, we identify the demo description and extract key frames using a hand–object detector and VLMs. Depth and the interaction object mask are then obtained via metric depth estimation and segmentation. (b) Using the 3D hand mesh from a hand pose estimator, we extract the contact pose based on the inter-finger vector and palm normal. (c) Object points are tracked from the pre-contact to the contact frame, and points

Read the complete text