OVAL-Grasp: Open-Vocabulary Affordance Localization for Task Oriented Grasping
Edmond Tong 1, Advaith Balaji 1, Anthony Opipari 1, Stanley Lewis 1, Zhen Zeng 2, Odest Chadwicke Jenkins 1 · 2025
Read OVAL-Grasp: Open-Vocabulary Affordance Localization for Task Oriented Grasping by Edmond Tong 1, Advaith Balaji 1, Anthony Opipari 1, Stanley Lewis 1, Zhen Zeng 2, Odest Chadwicke Jenkins 1 (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2511-20841.txt.
Introduction
Edmond Tong and colleagues present in 2025 OVAL-Grasp, a system that lets a robot grasp objects by the right part for the job. Grabbing a knife by the blade fails the task even when the grip is firm. Geometry-only methods struggle with appearance-defined parts, blocked views, and unseen objects. OVAL-Grasp needs no training for new objects, working zero-shot from language and vision alone.
Given a photo and a task, a large language model names which parts to grasp and which to avoid. A vision-language model then segments those parts, and builds a heatmap of actionable regions for the gripper. On 20 household objects with three tasks each, it beat two grasping baselines. On the Fetch mobile manipulator it segmented the correct part 95 percent of the time and grasped the right area in 78.3 percent of attempts. Under clutter, part selection succeeded 80 percent of the time.
Catalog record
| CREATOR | Edmond Tong 1, Advaith Balaji 1, Anthony Opipari 1, Stanley Lewis 1, Zhen Zeng 2, Odest Chadwicke Jenkins 1 |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 19 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2511-20841
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
OVAL-Grasp: Open-Vocabulary Affordance Localization for Task Oriented Grasping Edmond Tong 1, Advaith Balaji 1, Anthony Opipari 1, Stanley Lewis 1, Zhen Zeng 2, Odest Chadwicke Jenkins 1 Abstract To manipulate objects in novel, unstructured environments, robots need task-oriented grasps that target object parts based on the given task. Geometry-based methods often struggle with visually defined parts, occlusions, and unseen objects. We introduce OVAL-Grasp , a zero-shot open-vocabulary approach to task-oriented, affordance based grasping that uses large-language models (LLM) and vision-language models (VLM) to allow a robot to grasp objects at the correct part according to a given task. Given an RGB image and a task, OVAL-Grasp identifies parts to grasp or avoid with an LLM, segments them with a VLM, and generates a 2D heatmap of actionable regions on the object. During our evaluations, we found that our method outperformed two task oriented grasping baselines on experiments with 20 household objects with 3 unique tasks for each. OVAL-Grasp successfully identifies and segments the correct object part 95% of the time and grasps the correct actionable area 78.3% of the time in real-world experiments with the Fetch mobile manipulator. Additionally, OVAL-Grasp finds correct object parts under partial occlusions, demonstrating a part selection success rate of 80% in cluttered scenes. We also demonstrate OVAL-Grasp’s efficacy in scenarios that rely on visual features for part selection, and show the benefit of a modular design through our ablation experiments. Our project webpage is available at https://ekjt.github.io/OVAL-Grasp/ . Keywords: Task-Oriented Grasping ⋅ \cdot Affordances † † tocauthor: Edmond Tong , Advaith Balaji, Anthony Opipari, Stanley Lewis, Zhen Zeng, Odest Chadwicke Jenkins † † institute: University of Michigan, Ann Arbor, MI, USA and J.P. Morgan AI Research Figure 1: OVAL-Grasp at work on the Fetch mobile manipulator. The robot understands which parts of the object it should grasp and which parts should be avoided to fulfill the given tasks described by language. 1 Introduction & Related Work Robots in unstructured environments must identify objects and understand their affordances —the actions they enable. Affordance grounding links actions to object geometry. Most current methods are end-to-end, relying on supervised training with fixed affordances, which limits adaptability 2 . Recent work explores one or zero-shot (open vocabulary) affordance localization 9 ; 10 . However, they still require some fine-tuning, which can be costly and dataset-dependent. We present our method OVAL-Grasp , a modular and training free approach to task oriented, affordance-based grasping, that enables a robot to grasp an open set of objects at the correct part according to a task described by language. This paper presents our method, along with an evaluation of its performance compared to two state-of-the-art (SOTA) baselines, ShapeGrasp and GraspGPT 7 ; 19 . Figure 1 demonstrates OVAL-Grasp in action. Large Language Models (LLMs) and Vision Language Models (VLMs) have emerged as powerful open-vocabulary tools 4 , excelling in robotics applications like planning and navigation 15 ; 14 . Prior affordance localization approaches used fine-tuned LLMs 13 or text encoders 10 . In contrast, we explore affordance localization using pre-trained foundation models without fine-tuning. Task-Oriented Grasping (TOG) selects grasps based on task input and affordances 5 . Deep learning-based TOG methods 20 require supervision, while recent zero-shot approaches 19 ; 18 ; 7 aim to generalize but often still need training. In this work, we present the following contributions: 1. OVAL-Grasp , a method for zero-shot task-oriented grasping by leveraging LLMs and VLMs. 2. A modular design that enables performance improvements as individual components are enhanced. 3. A method that achieves superior performance over SOTA methods in real robotic experiments. 4. Experiments that demonstrate the capability of our method in novel TOG scenarios such as completing tasks that require both visual and geometric cues. 2 Problem Definition In order for the robot to perform a successful zero-shot task oriented grasp given an RGB-D image I\in\mathbb{R}^{H\times W\times C} as input, the robot must generate an end-effector pose g\in SE(3) that lies on a part of an object x that facilitates the execution of a task t described by natural language. We assume that the robot has an egocentric RGB-D camera as well as a manipulator arm with a parallel jaw gripper capable of navigating to the grasp target g . We also assume access to a Large Language Model L , a vision-language part segmentation model V , and a grasp proposal model G . TOG methods can be evaluated by measuring their part identification and grasp success rate 7 ; 18 ; 19 . A task oriented grasp is considered successful if the proposed grasp location fulfills the given task based on commonsense reasoning. Figure 2: System overview. The robot generates a task-oriented grasps by using an LLM to identify grasp-relevant object parts, a VLM to segment them, and a constructed heatmap to filter grasp candidates to produce a set of grasp that fulfill the given task. 3 OVAL-Grasp: Affordance-Prompting OVAL-Grasp processes an RGB image and a task description to localize affordances. As shown in Figure 2 , the language model L uses the task t to identify the object x , and decompose it into the desirable parts (to grasp), and the undesirable parts (to avoid). The part segmentation model V then segments these parts, which we use to generate a heatmap based on each part’s confidence score. The grasp generator G proposes candidate grasps for the entire object, which are subsequently scored using the heatmap. 3.0.1 Part Decomposition: We prompt L to identify and decompose the relevant object x into desirable and undesirable parts, where parts that facilitate task t are desirable, and vice-versa. We use the GPT-4o API 12 . 3.0.2 Part Localization: We prompt V with the list of identified parts, producing a binary mask and confidence score for each part. We use PartGLEE 6 for its SOTA open-vocabulary part segmentation capabilities. 3.0.3 Heatmap Generation: OVAL-Grasp filters grasp candidates by composing a heatmap from relevant part segments and confidence scores. We generate a heatmap with the same height and width as the input image to score potential grasps. The heatmap H is initialized to zero , and the whole object segmentation is added with a positive value. The desirable part segments are added to the heatmap and undesirable segments are subtracted from the heatmap and are both scaled by their respective confidence scores. The values in the heatmap H were scaled to the range [0,255] and then smoothed using a Gaussian blur with a 3\times 3 kernel to remove any noise from adding segments. 3.0.4 Grasp Generation and Filtering: Based on the previous modules’ outputs, OVAL-Grasp scores candidate grasps using the task-specific heatmap, selecting the highest-scoring pose for execution. We used ContactGraspNet 17 to generate the candidate grasps g_{c} , where the number of candidates ranges from 0 to N depending on the scene. Each candidate grasp is scored using H . We use two scores, the contact score ensures the grasp targets the correct region, while the z-axis score discourages approach angles that could obstruct task-relevant areas. The contact score S_{c}(g_{c_{i}}) is computed by reprojecting the gripper’s contact point (x_{c},y_{c}) , provided by ContactGraspNet, into the heatmap H . From the object point cloud P , and the line