Out-of-Distribution Recovery with Object-Centric Keypoint Inverse Policy for Visuomotor Imitation Learning

George Jiayuan Gao, Tianyu Li, Nadia FigueroaUniversity of Pennsylvania · 2024

Read Out-of-Distribution Recovery with Object-Centric Keypoint Inverse Policy for Visuomotor Imitation Learning by George Jiayuan Gao, Tianyu Li, Nadia FigueroaUniversity of Pennsylvania (2024) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2411-03294.txt.

Description

Abstract We propose an object-centric recovery (OCR) framework to address the challenges of out-of-distribution (OOD) scenarios in visuomotor policy learning. Previous behavior cloning (BC) methods rely heavily on a large amount of labeled data coverage, failing in unfamiliar spatial states.

Catalog record

CREATORGeorge Jiayuan Gao, Tianyu Li, Nadia FigueroaUniversity of Pennsylvania
YEAR2024
COLLECTIONPAPERS
FORMATTXT
FILE SIZE35 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2411-03294

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Out-of-Distribution Recovery with Object-Centric Keypoint Inverse Policy for Visuomotor Imitation Learning George Jiayuan Gao, Tianyu Li, Nadia FigueroaUniversity of Pennsylvania Abstract We propose an object-centric recovery (OCR) framework to address the challenges of out-of-distribution (OOD) scenarios in visuomotor policy learning. Previous behavior cloning (BC) methods rely heavily on a large amount of labeled data coverage, failing in unfamiliar spatial states. Without relying on extra data collection, our approach learns a recovery policy constructed by an inverse policy inferred from the object keypoint manifold gradient in the original training data. The recovery policy serves as a simple add-on to any base visuomotor BC policy, agnostic to a specific method, guiding the system back towards the training distribution to ensure task success even in OOD situations. We demonstrate the effectiveness of our object-centric framework in both simulation and real robot experiments, achieving an improvement of 77.7% over the base policy in OOD. Furthermore, we show OCR’s capacity to autonomously collect demonstrations for continual learning. Overall, we believe this framework represents a step toward improving the robustness of visuomotor policies in real-world settings. Project Website: https://sites.google.com/view/ocr-penn Fig. 1 : Object-Centric Recovery (OCR) on Bottle Pick and Place Task. The base visuomotor policy ( Right ), trained on the bottle’s initial pose within the green-shaded region of the table, exhibits limited generalization when the bottle is initialized in the red-shaded region, which is considered out-of-distribution (OOD). ( Left ) showed the recovery policy using our OCR framework to recover from the red-shaded OOD region, returning the system to a region of high confidence for the base visuomotor policy, where it resumes control. 0 0 0 ∗ Corresponding author. (e-mail: [email protected]) 0 0 0 All authors are with School of Engineering and Applied Science, University of Pennsylvania, Pennsylvania, PA 19104 USA. I Introduction Robot learning has achieved significant success in deploying Imitation Learning (IL) methods on real-world robotic systems [ 1 ] . One widely studied approach within IL is Behavior Cloning (BC), which has been explored extensively in recent work [ 2 , 3 , 4 , 5 , 6 , 7 ] . BC methods enable learning control policies directly from demonstrations without the need for explicit environmental modeling, making the process relatively straightforward. However, despite producing promising results, BC is well-known for its susceptibility to the covariate shift problem [ 1 ] . This issue arises because traditional BC approaches depend heavily on large quantities of labeled data, which are often obtained through labor-intensive methods such as teleoperation or kinesthetic teaching. Consequently, BC may struggle to perform reliably in out-of-distribution (OOD) scenarios, where data is sparse or noisy—reflecting a broader challenge faced in supervised learning. Addressing this issue typically requires either returning to laborious data collection or utilizing corrective mechanisms, such as guidance from human operators or reinforcement learning (RL) agents [ 8 , 9 , 10 , 11 ] , both of which impose additional deployment efforts on robotic systems. To enjoy the benefits of strong performing BC policies in distribution (ID) settings while not requiring the human effort of collecting more data or the compute effort of running an RL step when OOD, in this work, we propose a recovery policy framework that brings the system back to the training distribution to ensure task success even when OOD. In particular, we focus on the key challenges of visuomotor policy learning by integrating a recovery policy constructed from the gradient of the training data manifold with a base visuomotor BC policy (e.g., a diffusion policy [ 2 ] ). Inspired by the “Back to the Manifold” approach [ 12 ] the recovery policy guides the system back towards the training manifold, at which point the base policy resumes control. However, unlike [ 12 ] , which focuses on recovering from OOD scenarios related to the robot’s state, our approach takes an object-centric perspective, specifically addressing OOD situations for task-relevant object states. We believe this object-centric approach significantly enhances the OOD recovery capabilities of visuomotor policies, leading to more robust learning for object manipulation tasks. Furthermore, our recovery framework is designed to be agnostic to the choice of base policy, allowing it to be seamlessly integrated with various BC implementations. This flexibility makes our method adaptable for future developments in imitation learning (IL). In this paper, we make the assumption that we have access to relevant object models. Also, we focus on OOD cases in which the relevant object enters unfamiliar spatial regions. Paper organization Section II presents an overview of the existing works. Section III describes the problem formulation. Section IV presents the object-centric recovery policy framework in detail, including its construction of the training data manifold and the keypoint inverse policy. In section V , we demonstrate the effectiveness of our approach on several benchmarks, including both simulation and real robot experiments, showing that our recovery policy improves performance when entering unfamiliar states. We also show that our method has the desired property for lifelong learning of visuomotor policies, improving the performance of OOD while not diminishing the in-distribution performance. Section VI discusses the limitations and future directions. II Related Work When deployed to the real world, vision-based IL could easily be initialized or moved to OOD situations, possibly due to bias in data collection and compounding errors. Deploying BC methods OOD could lead to unknown behavior in the low-data region. To address this, a well-known family of approaches is Data Aggregation, which gathers extra data from expert policies (usually provided by humans) through online interaction [ 13 , 8 , 10 , 11 ] . However, performing such an online data collection procedure is an additional burden to the human when building a system. Our method tries to avoid additional online interaction and cumbersome data collection by squeezing as much information from the existing training data. The OOD problem has received much attention from the offline RL community. Offline RL suffers less compounding errors than BC methods as it optimizes for long-term outcomes [ 14 ] . Yet, still struggles with distribution shifts like extrapolation errors due to limited data. To tackle the OOD problem, methods like [ 15 , 16 , 17 ] try to penalize actions that are far from the data. Moreover, several works [ 18 , 19 ] also propose to recover back to the training data region, which indirectly shares a similar idea as our work. The paradigm of BC+RL has also been a popular choice for addressing the OOD problem [ 20 , 9 , 21 ] . Our approach takes on a similar direction for training a recovery policy, but instead, we use object-centric BC as the add-on for the base BC policy. Closely related to our work is [ 12 ] , which introduces a vision-based OOD recovery policy by 1) learning an equivariant map that encodes visual observations into a latent space whose gradient corresponds to robot end-effector positions, and 2) following a gradient learned by a Mixture Density Network (MDN) [ 22 ] . Rather than recovering the robot action, our work focuses on recovering task-relevant objects and inducing the robot action. When dealing with objects, it could be more general to utilize object-centric representations. Many works [ 6

Read the complete text