AffordGrasp: In-Context Affordance Reasoning for Open-Vocabulary Task-Oriented Grasping in Clutter
Yingbo Tang, Shuaike Zhang, Xiaoshuai Hao, Pengwei Wang, Jianlong Wu, Zhongyuan Wang, Shanghang Zhang · 2025
Read AffordGrasp: In-Context Affordance Reasoning for Open-Vocabulary Task-Oriented Grasping in Clutter by Yingbo Tang, Shuaike Zhang, Xiaoshuai Hao, Pengwei Wang, Jianlong Wu, Zhongyuan Wang, Shanghang Zhang (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2503-00778.txt.
Introduction
Yingbo Tang and co-authors present AffordGrasp, a framework that lets robots grasp objects for a task, in clutter, in a 2025 paper. A good grasp depends on what the object is for: you hold a mug by its handle to pour, not by its rim. Older methods needed heavy training on fixed tasks and objects, so they stumbled on new scenes.
AffordGrasp leans on the reasoning power of vision-language models. It infers the task from loose, everyday instructions rather than strict commands, finds the task-relevant object, and grounds the right part to hold. Grasp poses are then generated inside that affordance region. The result is manipulation that respects both function and context, easing natural human-robot teamwork.
Catalog record
| CREATOR | Yingbo Tang, Shuaike Zhang, Xiaoshuai Hao, Pengwei Wang, Jianlong Wu, Zhongyuan Wang, Shanghang Zhang |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 24 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2503-00778
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
AffordGrasp: In-Context Affordance Reasoning for Open-Vocabulary Task-Oriented Grasping in Clutter Yingbo Tang, Shuaike Zhang, Xiaoshuai Hao, Pengwei Wang, Jianlong Wu, Zhongyuan Wang, Shanghang Zhang Abstract Inferring the affordance of an object and grasping it in a task-oriented manner is crucial for robots to successfully complete manipulation tasks. Affordance indicates where and how to grasp an object by taking its functionality into account, serving as the foundation for effective task-oriented grasping. However, current task-oriented methods often depend on extensive training data that is confined to specific tasks and objects, making it difficult to generalize to novel objects and complex scenes. In this paper, we introduce AffordGrasp , a novel open-vocabulary grasping framework that leverages the reasoning capabilities of vision-language models (VLMs) for in-context affordance reasoning. Unlike existing methods that rely on explicit task and object specifications, our approach infers tasks directly from implicit user instructions, enabling more intuitive and seamless human-robot interaction in everyday scenarios. Building on the reasoning outcomes, our framework identifies task-relevant objects and grounds their part-level affordances using a visual grounding module. This allows us to generate task-oriented grasp poses precisely within the affordance regions of the object, ensuring both functional and context-aware robotic manipulation. Extensive experiments demonstrate that AffordGrasp achieves state-of-the-art performance in both simulation and real-world scenarios, highlighting the effectiveness of our method. We believe our approach advances robotic manipulation techniques and contributes to the broader field of embodied AI. Project website: https://eqcy.github.io/affordgrasp/ . I INTRODUCTION For robots assisting in daily tasks, understanding object affordances and selecting grasp areas and poses are vital for effective execution. For example, when pouring water, a robot must identify a cup’s handle as the graspable region to ensure success. This highlights the need for task-oriented grasping based on affordances, akin to how humans intuitively interact with objects—like holding a spoon or racket handle—and generalize to new objects. However, enabling robots to replicate this ability is challenging, particularly in accurately identifying affordances without extensive task-specific training. Overcoming this is key to developing robots that adapt to diverse real-world scenarios. Fig. 1: Compared to existing task-oriented grasping methods, our approach offers three key advantages: (1) leveraging VLMs for affordance reasoning, enabling better understanding of user intentions from implicit language instructions and visual scene observations; (2) a training-free pipeline, eliminating the need for annotated data to train task-oriented grasp evaluators; and (3) the ability to handle cluttered scenes, rather than being limited to single-object grasping. Existing methods [ 1 , 2 , 3 , 4 , 5 , 6 ] transfer affordances [ 1 , 2 , 3 ] or task-oriented grasps [ 4 , 5 , 6 ] from training data to test objects by constructing affordance memories or learning task-specific skills. These approaches retrieve semantically and geometrically similar objects to transfer knowledge from seen to unseen instances. However, their reliance on time-consuming retrieval processes and limited affordance memory scales hinders their effectiveness in open-vocabulary scenarios with novel tasks and objects. Recent research shows growing interest in open-vocabulary grasping [ 7 , 8 , 9 , 10 ] . These methods generate grasp poses by jointly training vision-language grasping frameworks [ 7 , 8 , 9 ] or leveraging the visual grounding abilities of VLMs [ 10 ] . However, they focus solely on grasping entire objects, neglecting task-specific affordances. For finer-grained grasping, [ 11 ] uses open-vocabulary object detection [ 12 ] and part segmentation [ 13 ] to perform part-level grasps based on user’s part prompts. Other works [ 14 , 15 ] employ Large Language Models (LLMs) to reason about object parts or affordances from language instructions, localizing regions via VLMs. However, existing methods often require explicit object names in prompts and struggle with implicit instructions ( e.g. , “I am thirsty”) without visual scene context, lacking the ability to interpret tasks and object functionalities in complex open-vocabulary environments. Although existing task-oriented or open-vocabulary grasping methods perform well in some scenarios, inferring appropriate open-vocabulary task-oriented grasps in cluttered environments remains challenging. We summarize the key difficulties as follows: (1) Ambiguity in user instructions. Existing methods often require explicit object names and tasks, lacking flexibility. In practice, user instructions frequently contain implicit goals, demanding deeper semantic understanding. (2) Training dependency. Some methods [ 5 , 6 ] use LLMs to infer object properties and build task-object similarity, yet still rely on annotated data to train task-oriented grasp evaluators for generating desired poses. (3) Cluttered scenes. Current task-oriented grasping methods typically focus on single-object scenarios. Cluttered environments with multiple objects increase the complexity of identifying target objects, posing additional challenges. To address the aforementioned challenges, we propose AffordGrasp , a novel affordance reasoning framework for open-vocabulary task-oriented grasping. As illustrated in Fig. 1 , our method offers three key advantages over existing approaches. First, we tackle ambiguous user instructions by leveraging Vision-Language Models (VLMs) to reason about affordances from both language and visual inputs. By utilizing the inherent reasoning capabilities of VLMs, we establish relationships among tasks, objects, and affordances in complex scenes. Second, based on the affordance reasoning results, the target object and its visual affordance are grounded using VLMs for grasp generation, eliminating the need for additional training data. Finally, with a deep understanding of tasks and affordances, our method outperforms existing approaches in cluttered scenes, enabling robust and context-aware grasping. The main contributions of our work are as follows: • We introduce AffordGrasp, a novel framework for open-vocabulary task-oriented grasping in cluttered scenes. • Specifically, we propose an in-context affordance reasoning module that leverages the reasoning capabilities of Vision-Language Models (VLMs). This module extracts explicit tasks from implicit user instructions and infers object affordances from visual observations, significantly enhancing the understanding of tasks, objects, and their affordances through the integration of language and visual inputs. • Importantly, our method is entirely open-vocabulary, requiring only arbitrary user instructions and RGB-D images. It functions without the need for additional training or fine-tuning, which makes it highly flexible and scalable. • We validate the effectiveness of our approach through extensive experiments in both simulation and real-world robotic scenarios, demonstrating its robustness and practical applicability. II RELATED WORK II-A Visual Affordance Grounding Visual Affordance Grounding involves identifying and localizing specific areas of an object in an image that enable potential interactions. Researchers have explored various approaches to learn affordances, including annotated images [ 16 ] , demonstration videos [ 17 , 1 ] , and cross-modal inputs [ 18 ] . Methods like Robo-ABC [ 2 ] and RAM [ 3 ] employ semantic correspondence and retrieval-based paradigms to generalize affordances across objects and domains, utilizing diverse data sources for zero-shot transfer. Additionally, GLOVER [ 19 ] fine-tunes a multi-modal LLM to predict visual affordances. However, these approaches often require varying degrees of data training, limiting their applicability in fully open-vocabulary scenarios. II-B Task-Oriented Grasping Task-Oriented Grasping focuses on grasping specific object parts based on task constraints, enabling robots to perform subsequent manipulations. Traditional approaches rely on training with large-scale annotated datasets [ 20 , 21 ] . To enhance generalization to novel objects, methods like GraspGPT [ 5 ] and FoundationGrasp [ 6 ] leverage the open-ended semantic knowledge of LLMs to bridge connections between dataset objects and novel ones, though