ORACLE-Grasp: Zero-Shot Task-Oriented Robotic Grasping using Large Multimodal Models

Avihai Giuili, Rotem Atari, Avishai Sintov · 2025

Read ORACLE-Grasp: Zero-Shot Task-Oriented Robotic Grasping using Large Multimodal Models by Avihai Giuili, Rotem Atari, Avishai Sintov (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2505-08417.txt.

Introduction

Avihai Giuili, Rotem Atari, and Avishai Sintov let a large multimodal model decide how a robot should grasp unfamiliar objects, in this 2025 paper. Service robots in homes and hospitals meet new items daily. Older grasp systems need dense training data or exact shape models, and both fail on novelty.

Their framework asks the model twice: first what the object is and does, then which regions suit its function. The image is split into candidate patches for the model to judge. Depth data refine the final grip when present, and an early stop saves compute. Tests cover varied scenes in color and depth.

Catalog record

CREATORAvihai Giuili, Rotem Atari, Avishai Sintov
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE35 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2505-08417

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

ORACLE-Grasp: Zero-Shot Task-Oriented Robotic Grasping using Large Multimodal Models Avihai Giuili, Rotem Atari, Avishai Sintov Abstract Grasping unknown objects in unstructured environments remains a fundamental challenge in robotics, requiring both semantic understanding and spatial reasoning. Existing methods often rely on dense training datasets or explicit geometric modeling, limiting their scalability to real-world tasks. Recent advances in Large Multimodal Models (LMMs) offer new possibilities for integrating vision and language understanding, but their application to autonomous robotic grasping remains largely unexplored. We present ORACLE-Grasp, a zero-shot framework that leverages LMMs as semantic oracles to guide grasp selection without requiring additional training or human input. The system formulates grasp prediction as a structured, iterative decision process, using dual-prompt tool calling to first extract high-level object context and then select task-relevant grasp regions. By discretizing the image space and reasoning over candidate areas, ORACLE-Grasp mitigates the spatial imprecision common in LMMs and produces human-like, task-driven grasp suggestions. Early stopping and depth-based refinement steps further enhance efficiency and physical grasp reliability. Experiments demonstrate that the predicted grasps achieve low positional and orientation errors relative to human-annotated ground truth and lead to high success rates in real-world pick up tasks. These results highlight the potential of combining language-driven reasoning with lightweight vision techniques to enable robust, autonomous grasping without task-specific datasets or retraining. I Introduction Robotic grasping remains a core challenge in autonomous manipulation, especially in unstructured and novel environments. Traditional approaches often depend on precise object models, depth-based geometry recognition, or supervised learning pipelines trained on large-scale datasets [ 1 , 2 ] . However, real-world deployment frequently requires interacting with novel objects exhibiting unpredictable shapes with task-based reasoning. In such scenarios, accurate grasp localization, identifying grasp points and orientations that are both feasible and task-appropriate, becomes critical [ 3 ] . While recent deep learning methods have advanced grasp quality prediction [ 4 ] , they primarily focus on geometric properties and neglect semantic reasoning about the object or intended task. This gap motivates the development of new frameworks that combine high-level semantic understanding with spatial reasoning, enabling more human-like, adaptive grasp selection in the real-world. Fig. 1: Overview of the ORACLE-Grasp framework. An RGB-D camera captures the scene. An LMM uses dual prompts over a grid to identify a graspable region, which is then refined with depth data for precise grasp prediction and executed by a robot. Previous approaches have typically relied on geometric analysis of depth data [ 5 , 6 ] , model-based grasp synthesis [ 7 ] , or extensive training of deep networks on large-scale grasp datasets [ 8 , 9 ] . Early work on learning-based methods for grasping novel objects, used machine learning techniques to predict effective grasp points directly from 2D images without requiring complete 3D reconstruction or object recognition [ 10 ] . However, deep learning-based approaches typically focus on learning and predicting the quality of candidate grasps, without integrating semantic reasoning about the task or the object’s context [ 11 ] . While recent methods have made significant advances in pixel-wise grasp prediction and affordance learning [ 12 , 13 ] , they often lack the semantic reasoning needed for task-oriented grasping, especially in zero-shot settings where the object has not been seen before. Large Language Models (LLMs) have significantly advanced robotics by enabling natural language understanding for task planning, human-robot interaction, and high-level decision-making [ 14 , 15 ] . These models allow robots to interpret abstract instructions (e.g., ”organize the desk”) and generate structured action sequences with explainable reasoning. Building on these capabilities, recent developments have introduced Large Multimodal Models (LMMs), which integrate vision and language processing to tackle embodied tasks [ 16 , 17 ] . By jointly reasoning over visual inputs and textual prompts, LMMs extend the functionality of LLMs, enabling robots to perceive, interpret, and act within their environments with greater semantic understanding. Prominent examples of LMMs include CLIP [ 18 ] , Flamingo [ 19 ] , and LLaVA [ 20 ] , which have demonstrated impressive capabilities in grounding visual scenes through natural language reasoning [ 21 ] . These models are typically trained on large-scale datasets comprising paired image and text data, allowing them to align visual and linguistic representations within a shared semantic space. As a result, LMMs can generalize across domains, leverage contextual information, and infer high-level semantic relationships that are critical for flexible and task-driven robotic behaviors [ 22 ] . Recently, LMMs have begun to be applied in robotics to bridge perception, reasoning, and action [ 16 , 23 , 24 ] , enabling new capabilities such as instruction following, semantic scene understanding, and object-centric manipulation in real-world environments [ 25 , 26 ] . By leveraging their ability to reason jointly over visual and linguistic modalities, LMMs offer a promising path toward more adaptable, scalable, and intuitive robotic systems that can operate with minimal task-specific training and additional data collection. Despite notable advances in high-level robotic task planning and embodied reasoning, the application of LMMs to robotic grasping remains relatively underexplored. Most existing work leveraging LMMs in robotics focuses on semantic instruction following, policy learning, or general manipulation planning, rather than on fine-grained grasp localization and selection. Among the few exceptions, PhyGrasp [ 27 ] introduces an LMM that integrates 3D point clouds and language to inform grasping decisions based on estimated physical properties such as material, fragility, and mass. However, PhyGrasp requires detailed 3D shape modeling and the availability of physical property annotations. It also depends on large-scale data collection and does not explicitly address task-driven, human-like grasp selection from single-view visual inputs. Another emerging line of work employs LMMs to direct grasping based on human verbal instructions, providing external context and object knowledge to guide the robot’s actions [ 28 , 29 , 30 , 31 ] . While these approaches support task-specific grasping, they depend on human guidance at inference time and often require pretraining on large, specialized grasp datasets, limiting their scalability for fully autonomous deployment. Moreover, they focus on selecting which object to grasp, rather than identifying task-driven, human-like grasp locations on the object itself. In this letter, we explore the capability of LMMs to infer feasible, task-relevant grasps from single images. We introduce ORACLE-Grasp (O-Grasp), a novel grasping framework (Figure 1 ) that treats an LMM as a semantic oracle to guide grasp selection. Rather than relying on low-level image features or geometric heuristics, O-Grasp reformulates the grasping problem as an iterative, prompt-driven reasoning process, enabling human-like, task-aware grasp predictions even for novel and amorphous objects. Our approach requires no data collection, dataset curation, or pretraining, operating in a fully zero-shot manner by leveraging an existing, general-purpose LMM. Through a dual-prompt approach, O-Grasp extracts both high-level scene context and localized candidate regions, framing grasp selection as a structured, context-driven decision rather than free-form generation tasks. The prompts are designed to be general and autonomous, allowing O-Grasp to operate without human intervention at inference time, in contrast to prior approaches that require human guidance through language commands. Critically, the use of chain-of-thought [ 32 ] reasoning initiated by the first prompt encourages the model to produce task-driven, human-like grasps that align with intuitive manipulation strategies. Although

Read the complete text