GraspMolmo: Generalizable Task-Oriented Grasping via Large-Scale Synthetic Data Generation

Abhay Deshpande, Yuquan Deng, Arijit Ray, Jordi Salvador, Winson Han, Jiafei Duan, Kuo-Hao Zeng, Yuke Zhu, Ranjay Krishna, Rose Hendrix · 2025

Read GraspMolmo: Generalizable Task-Oriented Grasping via Large-Scale Synthetic Data Generation by Abhay Deshpande, Yuquan Deng, Arijit Ray, Jordi Salvador, Winson Han, Jiafei Duan, Kuo-Hao Zeng, Yuke Zhu, Ranjay Krishna, Rose Hendrix (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2505-13441.txt.

Introduction

Abhay Deshpande and co-authors present GraspMolmo in this 2025 paper, a model that tells robots how to grasp objects for a stated task. Given a spoken instruction and one depth-camera view, it picks a grip that fits the job: a teapot by the handle for pouring, not by the body. Prior methods trained on small sets with simple wording and tidy scenes.

The team built PRISM, a synthetic set of 379,000 samples with cluttered scenes and varied, realistic task text. They fine-tuned a vision-language model on it. In hard real-world tests the model hit about seventy percent success, double the next best rival. It also predicted two-handed grasps for new tasks without extra training. Data, code, and benchmarks are public.

Catalog record

CREATORAbhay Deshpande, Yuquan Deng, Arijit Ray, Jordi Salvador, Winson Han, Jiafei Duan, Kuo-Hao Zeng, Yuke Zhu, Ranjay Krishna, Rose Hendrix
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE52 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2505-13441

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

GraspMolmo: Generalizable Task-Oriented Grasping via Large-Scale Synthetic Data Generation Abhay Deshpande, Yuquan Deng, Arijit Ray, Jordi Salvador, Winson Han, Jiafei Duan, Kuo-Hao Zeng, Yuke Zhu, Ranjay Krishna, Rose Hendrix Abstract We present GraspMolmo , a generalizable open-vocabulary task-oriented grasping (TOG) model. GraspMolmo predicts semantically appropriate, stable grasps conditioned on a natural language instruction and a single RGB-D frame. For instance, given “pour me some tea,” GraspMolmo selects a grasp on a teapot handle rather than its body. Unlike prior TOG methods, which are limited by small datasets, simplistic language, and uncluttered scenes, GraspMolmo learns from PRISM , a novel large-scale synthetic dataset of 379k samples featuring cluttered environments and diverse, realistic task descriptions. We fine-tune the Molmo visual-language model on this data, enabling GraspMolmo to generalize to novel open-vocabulary instructions and objects. In challenging real-world evaluations, GraspMolmo achieves state-of-the-art results, with a 70% prediction success on complex tasks, compared to the 35% achieved by the next best alternative. GraspMolmo also successfully demonstrates the ability to predict semantically correct bimanual grasps zero-shot. We release our synthetic dataset, code, model, and benchmarks to accelerate research in task-semantic robotic manipulation, which, along with videos, are available at this URL . Keywords: Robots, Learning, Task-Oriented Grasping, Manipulation Figure 1: A. We introduce a large dataset of task-semantic-annotated tasks in virtual scenes with a large variety of target objects, which we name P urpose-driven R obotic I nteraction in S cene M anipulation. B. We use PRISM to train GraspMolmo , which represents grasps as points and matches them to grasp proposals. C. We show strong transfer and generalization in real-world complex scenes with interesting task semantics using a 7-DoF Franka FR2 arm, and additionally show zero-shot adaptation to semantically-appropriate bimanual grasping. 1 Introduction Robotic grasping has come a long way, with models now capable of robustly predicting stable grasps for a wide range of objects. Yet most existing methods operate in an object-centric fashion: they predict grasps that are stable for a given object but oblivious to the task at hand. In the real-world, however, how an object should be grasped depends crucially on what the robot is trying to do with it. Consider a kitchen knife. A task-agnostic grasping system might simply choose a stable grasp anywhere on the knife, including the blade. While the blade might be an appropriate grasp site when handing the knife safely to another person, it would be ineffective for the purposes of chopping vegetables. These task-specific differences highlight the need for task-oriented grasping (TOG): The same physical object requires dramatically different grasps depending on the intended task [ 1 , 2 ] . Recent work has introduced datasets and benchmarks [ 2 , 3 , 4 ] , but they are limited in realism and diversity. Some contain relatively simple scenes without clutter and lack diverse object categories [ 2 ] . Others often templatize (e.g., “grasp the mug to pour” or “grasp the pan to clean”). These simplifications fail to capture the complexity of real-world environments and the wide variety of language humans use to specify tasks, such as “mince some garlic” or “do the dishes”. Furthermore, many models require pre-segmented point clouds [ 4 ] or multi-view observations [ 3 ] , restricting real-time deployment. To address these limitations, we propose GraspMolmo : a generalizable task-conditioned grasping model trained entirely on synthetic data. To overcome the brittleness of prior datasets, we create P urpose-driven R obotic I nteraction in S cene M anipulation (or PRISM for short), a large-scale suite of task-grasp pairs with 10,000 scenes, 2,356 object instances, and task descriptions ranging from simple (“cut the apple”) to compositional or nuanced (“mince some garlic for a salad”). This dataset spans diverse cluttered scenes with realistic textures, occlusions, and multiple objects per task. Using this dataset, we fine-tune Molmo [ 5 ] , a recent open-weight vision-language model (VLM), to predict 6-DoF grasps from a single RGB-D frame and a task instruction. By grounding task semantics in the visual scene, GraspMolmo learns to disambiguate grasps that would otherwise be indistinguishable from a purely geometric perspective. GraspMolmo achieves state-of-the-art results on the TaskGrasp benchmark (76.7% versus the next best prior work at 72.3%). GraspMolmo also achieves significant performance boost on our benchmark of simulated cluttered scenes (62.5% versus 40.0%). Naturally, simulated evaluations are only a proxy, but we further demonstrate GraspMolmo transfers zero-shot to the real world in cluttered scenes, including objects and tasks completely unseen in training (70.4% prediction and 61.1% overall success rates). GraspMolmo also zero-shot transfers to semantically-appropriate bimanual grasping in qualitative testing. We will release our dataset, model, code, and benchmarks to enable further research in task-semantic manipulation. 2 Related work Task-oriented grasping. The field of Task-Oriented Grasping (TOG) addresses how robots should grasp objects based on intended tasks rather than just stability. Recent approaches have shown diverse strategies with specific limitations. GraspGPT [ 6 ] and FoundationGrasp [ 7 ] are most similar to our method, but due to their reliance on TaskGrasp, have limited pretraining with simple scenes and tasks. LERF-TOGO [ 8 ] uses language-embedded radiance fields to localize grasp points in a scene, but has costly requirements such as multiple views and long inference times, which make it impractical for use. RTAGrasp [ 9 ] transfers grasping behavior from human demonstrations but depends heavily on task-grounded data. Methods such as CROG and GraspCLIP [ 10 , 11 ] represent a family of methods that perform inference directly on the image, outputting per-pixel predictions for grasp poses. These methods are limited to predicting only 4-DoF grasps, and strictly require a top-down camera view. TOGNet [ 12 ] extracts “patches of interest” in the image for grasping, but only accepts object-level rather than task-level grounding, resulting in coarse semantics that are ill-suited for diverse tasks. Underpinning many of these works, the TaskGrasp benchmark [ 13 ] provides evaluation standards and training data, but features simple scenes with limited task diversity. Our work addresses these collective limitations through comprehensive synthetic data generation with complex scenes, diverse and natural task specifications, and large data scale. Vision-language models in robotics Recent years have witnessed significant progress in leveraging the strong vision-language commonsense capabilities of foundation multimodal language models (MLMs) [ 14 , 15 , 16 , 17 ] for taking actions in the physical world [ 18 , 19 , 20 , 21 , 22 ] . Some studies employ pre-trained MLMs directly in a prompt-based manner [ 23 , 24 , 25 , 26 ] , while others further fine-tune or instruction-tune models for specific capabilities, including action policies [ 27 , 19 , 21 , 28 ] , physics [ 29 ] , navigation [ 30 , 31 ] , and spatial reasoning [ 32 , 33 , 34 ] . Our approach aligns with the latter, fine-tuning a state-of-the-art MLM [ 15 ] on task descriptions and grounding points specifically for object grasping tasks. We advance beyond concurrent methods that separately address physics or multi-step reasoning limited to judging object relationships [ 33 , 29 ] or precise pointing [ 32 ] without multi-step reasoning, by integrating both aspects (e.g., needing to reason about the precise point to grasp a

Read the complete text