ManipGPT: Is Affordance Segmentation by Large Vision Models Enough for Articulated Object Manipulation?

Taewhan Kim, Hojin Bae, Zeming Li, Xiaoqi Li, Iaroslav Ponomarenko, Ruihai Wu, Hao Dong · 2024

Read ManipGPT: Is Affordance Segmentation by Large Vision Models Enough for Articulated Object Manipulation? by Taewhan Kim, Hojin Bae, Zeming Li, Xiaoqi Li, Iaroslav Ponomarenko, Ruihai Wu, Hao Dong (2024) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2412-10050.txt.

Introduction

Taewhan Kim and six co-authors ask whether a large vision model alone can tell a robot where to grab a jointed object. Their paper is from two thousand twenty-four. Robots need to find the handle of a door or the lid of a box before they can move it. Older methods sample points or scan in three dimensions, which is slow and brittle.

Their framework fine-tunes a big pre-trained vision transformer on a small set of about ten thousand images. The images mix simulated and real scenes to close the gap between them. The model then paints a mask over the part to interact with. A control policy adjusts grip stiffness to suit the part. The pairing handled objects in both simulated and real tests without complex sensing rigs.

Catalog record

CREATORTaewhan Kim, Hojin Bae, Zeming Li, Xiaoqi Li, Iaroslav Ponomarenko, Ruihai Wu, Hao Dong
YEAR2024
COLLECTIONPAPERS
FORMATTXT
FILE SIZE31 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2412-10050

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

ManipGPT: Is Affordance Segmentation by Large Vision Models Enough for Articulated Object Manipulation? Taewhan Kim, Hojin Bae, Zeming Li, Xiaoqi Li, Iaroslav Ponomarenko, Ruihai Wu, Hao Dong Abstract Visual actionable affordance has emerged as a transformative approach in robotics, focusing on perceiving interaction areas prior to manipulation. Traditional methods rely on pixel sampling to identify successful interaction samples or processing pointclouds for affordance mapping. However, these approaches are computationally intensive and struggle to adapt to diverse and dynamic environments. This paper introduces ManipGPT, a framework designed to predict optimal interaction areas for articulated objects using a large pre-trained vision transformer (ViT). We create a dataset of 9.9k simulated and real images to bridge the visual sim-to-real gap and enhance real-world applicability. By fine-tuning the vision transformer on this small dataset, we significantly improve part-level affordance segmentation, adapting the model’s in-context segmentation capabilities to robot manipulation scenarios. This enables effective manipulation across simulated and real-world environments by generating part-level affordance masks, paired with an impedance adaptation policy, sufficiently eliminating the need for complex datasets or perception systems. Our project page is available at: https://lxkim814.github.io/ManipGPT_website/ I INTRODUCTION Robotic manipulation is crucial for enabling robots to perform in wide variety of tasks autonomously. Despite its importance, it remains a complex and challenging problem largely due to the variability in objects, mechanisms, and environmental constraints. A promising approach to address this challenge is visual actionable affordance, which reframes robotic manipulation as a perception task. This method uses visual cues to predict optimal interaction points, which streamlines manipulation tasks. Traditional affordance map generation [ 1 , 2 , 3 ] involves sampling pixels on an object, interacting at each point, and recording scores to construct a map. This process is repetitive, computationally intensive, and further burdened by reliance on point cloud data. Additionally, the interaction data collected from simulated objects have physical and visual discrepancies from real objects, creating a significant sim-to-real gap. Fig. 1: ManipGPT processes an RGB image with a visual prompt to generate an affordance mask, which determines the contact point and manipulation direction. Recent advancements leverage large-scale datasets and large language models (LLMs) to enhance task execution [ 4 , 5 ] . However, these methods often result in high computational costs and increased complexity, limiting their practicality in resource-constrained environments. Additionally, a gap between language and vision modalities hinders the seamless integration of visual and language-based commands, affecting the execution of complex tasks. To overcome these limitations, we present ManipGPT, a minimalist approach that leverages a fine-tuned large vision model in context to predict affordance masks for a wide range of indoor objects, significantly simplifying robotic manipulation. Our streamlined approach, using a single RGB image and a pair of prompt images to directly generate affordance masks, eliminates the need for iterative sampling techniques and enables one-shot manipulation without requiring robot-specific training. Additionally, we present a synthetic dataset of 9.9k images from both simulated and realistic environments, demonstrating that even minimal datasets with basic annotations can enhance robotic performance effectively. In this work, we define an affordance representation as any manipulatable geometry or region of an object that enables interaction to achieve a desired motion. By segmenting such regions, our approach generalizes across different object instances and enables visual affordance prediction without requiring explicit action annotations. Our system integrates affordance masks with a post-processing algorithm to ensure safe and efficient manipulation, including an optional impedance control mechanism for improved physical adaptability. Experimental results show that our framework predicts interaction points with human-like intuition, generating actionable affordance maps that enable robots to identify contact areas and motion directions. II RELATED WORKS II-A Articulated Object Manipulation Articulated object manipulation poses significant challenges due to geometric variability and the interactive nature of object parts. Existing methods largely fall into two categories: affordance-based policies and articulation estimation. Affordance-based policies identify actionable regions on objects by generating affordance maps that indicate the likelihood of moving object parts [ 1 , 2 , 3 , 6 ] . While these methods offer generalization to novel objects, they typically involve a dataset collection stage where a robot engages in trial-and-error interactions by sampling points in a point cloud or pixels in an image, aiming to identify the most actionable regions. This process is time-consuming and computationally intensive, especially in a 3D simulation environment. Moreover, the reliance on simulations introduces a sim-to-real gap, meaning the affordance map collected from simulated interactions may not accurately reflect real-world scenarios. Articulation estimation methods, on the other hand, predict 6D joint parameters directly from visual inputs such as a sequence of images [ 7 , 8 ] , RGB-D images [ 9 , 10 ] , or point clouds [ 11 , 12 ] . By segmenting images or point clouds and estimating part poses, these methods enable robust manipulation trajectories. In contrast, our approach simplifies the process by using a single RGB image and prompt inputs to generate an affordance mask. This eliminates the need for pointclouds or iterative sampling for affordance learning and achieves effective manipulation with minimal computational overhead. II-B Large Models and Applications in Robotics Recent advancements in large vision models [ 13 , 14 , 15 ] have advanced semantic and instance segmentation. These models allow robots to operate in diverse environments with minimal real-world training. By leveraging pre-trained models, robots can quickly generalize to new scenarios and improve operational efficiency. In robotics, such models are especially valuable for tasks that require fine-grained visual understanding, such as point cloud and 2D image segmentation [ 16 , 17 ] . LLMs like GPT [ 18 ] and LLaMA [ 19 ] have enhanced robotics by enabling natural language understanding and high-level reasoning. When integrated with Visual Language Models (VLMs), they strengthen robots to interpret language commands and execute tasks with contextual awareness and adaptability [ 4 , 5 ] . While large vision models have widely been used as supplementary tools for pre-processing or post-processing, with LLMs serving as the backbone, our approach diverges by implementing SegGPT [ 14 ] as the primary backbone model. Unlike resource-intensive LLMs, SegGPT’s smaller 370 million parameter architecture offers vivid segmentation and flexible visual prompting, making it an efficient application. II-C 2D Affordance Datasets Existing 2D affordance datasets primarily focus on small, hand-held objects with well-defined interaction points, such as kitchen utensils and tools [ 20 , 21 , 22 ] . While these datasets provide affordance annotations, they are limited to single-part objects and do not capture affordance regions on articulated objects with multiple manipulatable components. Articulated objects, such as drawers and refrigerators, require affordance segmentation that identifies distinct parts (e.g., doors, lids, drawers). This highlights the need for datasets that support part-level affordance segmentation in articulated objects. III Method Fig. 2: The pipeline is divided into two main modules: Affordance Predictor and Action Proposer. (a) The RGB image and category-specific prompt are input to generate an affordance mask. (b) The affordance mask highlights actionable parts of the object. (c) A surface normal map is used to understand the object’s surface orientation. (d) A vector gradient filter refines the normal map by filtering out non-ideal areas. (e) The optimal contact point and manipulation direction are identified. (f) The manipulation direction is set

Read the complete text