ZeroDexGrasp: Zero-Shot Task-Oriented Dexterous Grasp Synthesis with Prompt-Based Multi-Stage Semantic Reasoning

Juntao Jian, Yi-Lin Wei, Chengjie Mou, Yuhao Lin, Xing Zhu, Yujun Shen, Wei-Shi Zheng, Ruizhen Hu, [2mm] Shenzhen University Sun Yat-sen University Ant Group · 2025

Read ZeroDexGrasp: Zero-Shot Task-Oriented Dexterous Grasp Synthesis with Prompt-Based Multi-Stage Semantic Reasoning by Juntao Jian, Yi-Lin Wei, Chengjie Mou, Yuhao Lin, Xing Zhu, Yujun Shen, Wei-Shi Zheng, Ruizhen Hu, [2mm] Shenzhen University Sun Yat-sen University Ant Group (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2511-13327.txt.

Introduction

Juntao Jian and 7 co-authors propose ZeroDexGrasp, a method that plans skillful multi-finger grasps for a stated task without task-specific training data. Most grasp systems lean on costly labeled examples and struggle with new objects or new instructions. A good grasp must fit both the object's use and its physical shape.

ZeroDexGrasp uses a multimodal language model in stages to reason from task and object meaning toward a first grasp and contact points. A contact-guided optimization step then refines the pose for physical feasibility. Tests showed strong zero-shot grasps across unseen object types and complex tasks, a step toward more general robot hands.

Catalog record

CREATORJuntao Jian, Yi-Lin Wei, Chengjie Mou, Yuhao Lin, Xing Zhu, Yujun Shen, Wei-Shi Zheng, Ruizhen Hu, [2mm] Shenzhen University Sun Yat-sen University Ant Group
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE36 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2511-13327

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

ZeroDexGrasp: Zero-Shot Task-Oriented Dexterous Grasp Synthesis with Prompt-Based Multi-Stage Semantic Reasoning Juntao Jian, Yi-Lin Wei, Chengjie Mou, Yuhao Lin, Xing Zhu, Yujun Shen, Wei-Shi Zheng, Ruizhen Hu, [2mm] Shenzhen University Sun Yat-sen University Ant Group Abstract Task-oriented dexterous grasping holds broad application prospects in robotic manipulation and human-object interaction. However, most existing methods still struggle to generalize across diverse objects and task instructions, as they heavily rely on costly labeled data to ensure task-specific semantic alignment. In this study, we propose ZeroDexGrasp , a zero-shot task-oriented dexterous grasp synthesis framework integrating Multimodal Large Language Models with grasp refinement to generate human-like grasp poses that are well aligned with specific task objectives and object affordances. Specifically, ZeroDexGrasp employs prompt-based multi-stage semantic reasoning to infer initial grasp configurations and object contact information from task and object semantics, then exploits contact-guided grasp optimization to refine these poses for physical feasibility and task alignment. Experimental results demonstrate that ZeroDexGrasp enables high-quality zero-shot dexterous grasping on diverse unseen object categories and complex task requirements, advancing toward more generalizable and intelligent robotic grasping. I INTRODUCTION Task-oriented dexterous grasp synthesis has long been a challenging problem in the fields of robotic manipulation and human-object interaction. Unlike general dexterous grasp synthesis primarily aims to ensure grasp stability or diversity, task-oriented dexterous grasping additionally requires producing multi-finger grasping poses that are well aligned with specific task objectives and object affordances (referred to as semantic requirements ). To better adapt grasping hands to align semantic requirements, recent studies typically annotate grasp poses with type-specific labels [ 1 , 2 ] , or further leverage large language models (LLMs) to generate diverse language instructions [ 3 , 4 , 5 ] . Both collect “object-semantic-grasp” data pairs that are then used to train conditional generative models for grasp synthesis. Although these methods have achieved progress in handling given task instructions, these methods face major limitations: they require extensive manual data collection and cover only finite object categories, which severely restricts generalization to unseen objects and tasks in open-set scenarios. Fig. 1: Given task instructions and target objects, ZeroDexGrasp synthesizes dexterous grasps in a zero-shot manner by reasoning contact information, hand position, orientation, grasp type, and followed by contact-guided refinement to ensure semantic alignment and physical feasibility. We aim to enhance the generalization of task-oriented dexterous grasping, particularly for open-set objects. This ambition introduces two primary challenges: (1) establishing reliable alignment between task instruction semantics and object affordances, and (2) bridging the significant conceptual gap from abstract, high-level semantic requirements to concrete, anthropomorphic dexterous grasp configurations. We observe that in typical grasping behaviors, the hand’s global properties, including final position, orientation, contact information, and overall grasp type, are primarily guided by semantic requirements, while the joints’ configurations are more directly influenced by the object’s local geometry. Besides, recent advancements in multimodal large language models (MLLMs) offer powerful capabilities in semantic understanding, spatial reasoning, and commonsense knowledge. These capabilities provide a feasible pathway for achieving zero-shot generalization of the hand’s global properties in dexterous grasping of diverse objects. To this end, we propose ZeroDexGrasp , a zero-shot task-oriented dexterous grasp synthesis framework that integrates MLLMs with grasp optimization to achieve human-like grasp pose generation for open-set objects and task instructions, as shown in Fig. 1 . ZeroDexGrasp comprises two key components, prompt-based multi-stage semantic reasoning and contact-guided grasp refinement. The first component bridges the gap between task semantics, object affordances, and grasp poses by operating on discretized semantic representation, thereby producing an initial grasp pose and contact information that satisfies the semantic requirements. Specifically, part-based and point-based visual prompts are used to infer discrete object regions for hand and functional fingers contacts, textual candidate prompts, drawn from discrete sets, are used to infer the initial relative hand position and grasp type, and an imagination-based visual prompt method is used to guide the MLLMs to reason the hand’s initial rotational pose. The second component leverages the inferred contact information to refine the initial grasp pose, synthesizing more reasonable, physically feasible, and task-aligned human-like dexterous grasps. Our contributions are summarized as follows: • We present ZeroDexGrasp, a zero-shot task-oriented dexterous grasp synthesis framework that integrates MLLMs with grasp optimization to generate high-quality dexterous grasps for open-set objects and tasks. • We develop a prompt-based multi-stage semantic reasoning based on discretized semantic representation to bridge task semantics, object affordances, and grasp poses, producing initial grasps and contact information that align with semantic requirements. • Extensive experiments on diverse open-set objects and tasks demonstrate the effectiveness and strong zero-shot generalization of ZeroDexGrasp. II RELATED WORK II-A Task-Oriented Semantic Dexterous Grasping Task-oriented dexterous grasp synthesis aims to produce grasps that support downstream manipulation. Some methods abstract tasks as action labels and learn task-conditioned grasp distributions from annotated datasets [ 1 , 2 ] . With the rapid progress of natural language processing and large language models, the diversity of task instructions has been enriched, and understanding unstructured task semantics has become feasible. Some works [ 3 , 5 ] encode task instructions and conditions a diffusion model on both text and object point clouds. SemGrasp [ 4 ] finetunes pretrained MLLMs [ 6 ] . Despite promising results, these approaches are constrained by limited training datasets and struggle to generalize to novel object categories. More recent works [ 7 , 8 , 9 , 10 ] have explored different strategies to enhance the generalization of dexterous grasp synthesis across unseen objects and diverse task instructions. Dexgraspvla [ 7 ] integrates large vision language models with grasp controllers to align grasp generation with task instructions. AffordDexGrasp [ 8 ] introduce generalizable affordance representations, and G-dexgrasp [ 10 ] retrieves generalizable priors to bridge semantics and dexterous actions. However, these methods still demand expensive hand-specific data and training. DexVLG [ 11 ] leverages geometry-based sampling and optimization to generate part-level grasp data, but focusing only on contact parts is insufficient to align high-level semantics requirements. Different from the above methods, our approach achieves zero-shot alignment of task semantics, object affordances, and grasp poses, enabling more holistic semantics-to-grasp reasoning with strong generalization. II-B Zero-Shot Robotic Grasping Zero-shot grasping synthesis refers to generating feasible grasps for diverse objects or tasks without requiring additional training or fine-tuning. Analytical approaches [ 12 , 13 , 14 ] leverage object geometry to synthesize stable or varied grasps for diverse objects. In addition, with the development of large-scale grasping datasets, some works [ 15 , 16 , 17 ] train foundation models for robotic grasping to improve performance on unseen objects. For low-degree-of-freedoms (Dofs) grippers, such as two-finger grippers, these models have been further adapted for task-oriented semantic grasping by leveraging Visual foundation Models [ 18 , 19 ] to build semantic feature fields [ 20 , 21 ] or to identify task-relevant contact regions [ 22 , 23 ] in a zero-shot manner. Nevertheless, directly extending such strategies to dexterous hands remains challenging, as their high DoFs make it difficult to ensure reasonable, semantically meaningful global hand poses and physically plausible finger configurations. II-C Visual Prompting in Multimodal Large Language Models MLLMs equip pre-trained LLMs with visual capabilities. While textual prompting [ 24 ] in LLMs has been widely studied, visual prompting [ 25 ]

Read the complete text