GraspMAS: Zero-Shot Language-driven Grasp Detection with Multi-Agent System

Quang Nguyen, Tri Le, Huy Nguyen, Thieu Vo, Tung Ta, Baoru Huang, Minh Vu, Anh Nguyen · 2025

Read GraspMAS: Zero-Shot Language-driven Grasp Detection with Multi-Agent System by Quang Nguyen, Tri Le, Huy Nguyen, Thieu Vo, Tung Ta, Baoru Huang, Minh Vu, Anh Nguyen (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2506-18448.txt.

Introduction

Quang Nguyen and co-authors build a team of AI agents that finds grasps from spoken requests in a 2025 paper. Telling a robot 'pick up the red mug behind the box' is easy for people and hard for software, especially in clutter with no task-specific training.

Their system, GraspMAS, splits the job among three agents. A planner breaks down the request, a coder writes and runs vision code, and an observer judges the outcome and sends feedback for another pass. On two large data sets the team beat prior language-driven grasp methods without any fine-tuning. Tests on simulated and real robots confirmed the approach works off the page.

Catalog record

CREATORQuang Nguyen, Tri Le, Huy Nguyen, Thieu Vo, Tung Ta, Baoru Huang, Minh Vu, Anh Nguyen
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE29 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2506-18448

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

GraspMAS: Zero-Shot Language-driven Grasp Detection with Multi-Agent System Quang Nguyen, Tri Le, Huy Nguyen, Thieu Vo, Tung Ta, Baoru Huang, Minh Vu, Anh Nguyen Abstract Language-driven grasp detection has the potential to revolutionize human-robot interaction by allowing robots to understand and execute grasping tasks based on natural language commands. However, existing approaches face two key challenges. First, they often struggle to interpret complex text instructions or operate ineffectively in densely cluttered environments. Second, most methods require a training or fine-tuning step to adapt to new domains, limiting their generation in real-world applications. In this paper, we introduce GraspMAS, a new multi-agent system framework for language-driven grasp detection. GraspMAS is designed to reason through ambiguities and improve decision-making in real-world scenarios. Our framework consists of three specialized agents: Planner , responsible for strategizing complex queries; Coder , which generates and executes source code; and Observer , which evaluates the outcomes and provides feedback. Intensive experiments on two large-scale datasets demonstrate that our GraspMAS significantly outperforms existing baselines. Additionally, robot experiments conducted in both simulation and real-world settings further validate the effectiveness of our approach. I INTRODUCTION Grasp detection is a foundational aspect of robotic manipulation, enabling robots to determine feasible grasping configurations for effectively interacting with objects across various environments [ 1 , 2 ] . Traditional grasp detection approaches [ 3 ] have primarily emphasized securing the stability of identified grasp poses, frequently overlooking the integration of human intent in the process [ 4 , 5 , 6 , 7 , 8 ] . Recently, several works have integrated natural language into robotic grasping to enhance intuitive and flexible human-robot interaction, allowing robots to comprehend and execute context-aware instructions [ 9 , 10 ] . However, most existing grasp detection methods rely on a resource-intensive training process, which limits their adaptability to real-world scenarios, especially when faced with unseen data. Recently, progress in large-scale foundation models [ 11 , 12 , 13 , 14 ] has fueled significant growth in robotics research, particularly in enabling intuitive natural language interaction. Researchers have developed zero-shot approaches for different robotic tasks such as planning [ 15 , 16 ] , manipulation [ 17 , 18 ] , and navigation [ 19 , 20 ] . However, several existing works focus on long manipulation tasks while overlooking the foundational aspect of grasping. Some studies [ 21 , 22 , 23 ] design neural architectures to fuse visual and textual inputs, while others [ 9 , 24 , 25 ] adopt generative frameworks trained on extensive multimodal datasets [ 26 ] . Recent works utilize foundation models to enable zero-shot adaptation in robotics [ 27 , 28 , 29 ] . However, these methods still exhibit certain limitations. First, prior approaches lack the compositional reasoning needed to interpret open-ended language commands. For instance, a query like “Grasp the second bottle from the left” requires the model to spatially localize objects, count ordinal positions, and resolve directional references, while a command like “I need something to cut” demands inferring functional object properties (e.g., identifying knives or scissors as tools for cutting). Second, their reliance on end-to-end monolithic neural architectures necessitates training on large-scale datasets, a process that is resource-intensive and dataset-dependent. Furthermore, these models often lack generalization capabilities, requiring additional domain-specific training to achieve reliable performance when deployed across diverse scenarios [ 7 ] . (a) Traditional methods (b) Our method Fig. 1: Comparison between the traditional methods (a) and our approach (b) for language-driven grasp detection. The traditional method relies on a single trainable network, which may struggle with complex queries. In contrast, our approach leverages multi-agent collaboration, enabling more flexible and accurate grasp detection. To tackle real-world diverse scenarios, multi-agent systems that use compositional approaches [ 30 , 31 , 32 ] have emerged as powerful strategies to reason and make decisions collaboratively using several specialized subsystems (i.e., agents). Multi-agent systems have shown success in domains such as visual question answering [ 30 , 32 ] , video analysis [ 33 ] , and robotics [ 34 , 35 , 36 ] , where they integrate large language models and visual foundation models with minimal fine-tuning. In such frameworks, large language models usually serve as planners, code generators, or reasoning engines, while visual foundation models enable robust visual perception, fostering structured reasoning and task planning to enhance cross-domain generalization. Inspired by this paradigm, we propose a modular framework for language-conditioned grasp detection, designed to connect semantic language understanding with precise robotic physical execution in diverse real-world scenarios. In this paper, we propose GraspMAS, a zero-shot grasp detection method based on multi-agent systems. Our approach is a training-free, plug-and-play framework that leverages multi-agent collaboration and re-uses existing foundation vision language models (VLMs) to achieve robust language-driven grasp detection. Our framework comprises three core agents: Planner , Coder , and Observer . The Planner serves as a reasoning agent that generates detailed instructional plans. The Coder then translates these plans into executable Python code, dynamically interfacing with VLMs to detect objects and resolve spatial relation ambiguities. Finally, the Observer processes intermediate results from the executed code and generates a summary report, which is fed back to the Planner . This feedback mechanism enables the Planner to refine and adjust its instructional plans accordingly. In summary, our key contributions are: • We present GraspMAP, a new multi-agent approach for zero-shot language-driven grasp detection. • We validate our method across diverse settings, including benchmark datasets, simulated environments, and real robotic experiments. The results show that our method significantly outperforms other recent baselines. II Related Work Language-driven Grasp Detection. Traditional robotic grasping systems rely on analytical approaches that model object geometry and contact forces [ 37 , 38 , 39 ] , or on convolutional neural networks (CNNs) trained on labeled grasp datasets [ 40 , 41 , 42 , 5 , 26 ] . While effective in controlled settings, these methods lack contextual awareness and cannot interpret natural language instructions, limiting their adaptability in human-centric environments. Recent advancements address this gap by integrating language understanding with vision systems, enabling robots to localize and grasp objects specified through natural language [ 9 , 25 , 43 , 23 , 44 , 24 ] . A key innovation involves aligning textual and visual embeddings in a shared latent space, allowing models to correlate linguistic cues with object regions for grasp prediction. For example, Tziafas et al. [ 10 ] leverage linguistic references to synthesize grasps in cluttered scenes, while Chen et al. [ 45 ] fuse multimodal features to predict 2D grasp boxes from RGB images. Large language models (LLMs) further enhance this paradigm by enabling reasoning over indirect or abstract query. Jin et al. [ 46 ] , for instance, employ LLMs to infer grasp poses from verbal commands that lack explicit object references. Despite these advances, previous methods rely on simple text prompts and struggle with complex language, and often requiring additional fine-tuning. In contrast, our approach employs a multi-agent framework that re-uses existing foundation models to effectively interpret diverse language instructions without extra training. Multi-Agent Systems for Robotics. LLM-based agents have recently gained significant attention from researchers and are considered a key step toward artificial general intelligence

Read the complete text