VISO-Grasp: Vision-Language Informed Spatial Object-centric 6-DoF Active View Planning and Grasping in Clutter and Invisibility

Yitian Shi*, Di Wen*, Guanqi Chen, Edgar Welte, Sheng Liu, Kunyu Peng, Rainer Stiefelhagen, Rania Rayyes · 2025

Read VISO-Grasp: Vision-Language Informed Spatial Object-centric 6-DoF Active View Planning and Grasping in Clutter and Invisibility by Yitian Shi*, Di Wen*, Guanqi Chen, Edgar Welte, Sheng Liu, Kunyu Peng, Rainer Stiefelhagen, Rania Rayyes (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2503-12609.txt.

Introduction

Yitian Shi and co-authors present a 2025 robotics paper on VISO-Grasp, a system that helps a robot grip objects it can barely see. In heavy clutter the target may hide behind other items, and a gripper aimed at a guess often fails. The system builds a spatial map of which object sits where, updating it as the scene is explored.

Big foundation models supply the spatial reasoning behind the map. From it, the system plans its next best viewpoint, moving the camera to reveal what matters before reaching in. The authors show the loop of look, reason, and reposition lifts grasp success in severely blocked scenes.

Catalog record

CREATORYitian Shi*, Di Wen*, Guanqi Chen, Edgar Welte, Sheng Liu, Kunyu Peng, Rainer Stiefelhagen, Rania Rayyes
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE38 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2503-12609

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

VISO-Grasp: Vision-Language Informed Spatial Object-centric 6-DoF Active View Planning and Grasping in Clutter and Invisibility Thanks: Karlsruhe Institute of Technology, Karlsruhe, Germany. Email: [email protected]. This work is sponsored by the DFG SFB-1574 Circular Factory project. Thanks: *Equal contribution Yitian Shi*, Di Wen*, Guanqi Chen, Edgar Welte, Sheng Liu, Kunyu Peng, Rainer Stiefelhagen, Rania Rayyes Abstract We propose VISO-Grasp, a novel vision-language-informed system designed to systematically address visibility constraints for grasping in severely occluded environments. By leveraging Foundation Models (FMs) for spatial reasoning and active view planning, our framework constructs and updates an instance-centric representation of spatial relationships, enhancing grasp success under challenging occlusions. Furthermore, this representation facilitates active Next-Best-View (NBV) planning and optimizes sequential grasping strategies when direct grasping is infeasible. Additionally, we introduce a multi-view uncertainty-driven grasp fusion mechanism that refines grasp confidence and directional uncertainty in real-time, ensuring robust and stable grasp execution. Extensive real-world experiments demonstrate that VISO-Grasp achieves a success rate of 87.5 % 87.5\% in target-oriented grasping with the fewest grasp attempts outperforming baselines. To the best of our knowledge, VISO-Grasp is the first unified framework integrating FMs into target-aware active view planning and 6-DoF grasping in environments with severe occlusions and entire invisibility constraints. Code is available at: https://github.com/YitianShi/vMF-Contact I Introduction Robotic grasping in unstructured, cluttered environments remains a significant challenge, particularly for executing target-oriented grasps under partial or complete occlusions [ 1 ] . Humans instinctively overcome occlusion during targeted searches by adjusting their viewpoints and intuitive reasoning about spatial relationships to infer potential target locations. In comparison, leveraging the recent success of deep neural networks and contributions to large-scale training data [ 2 , 3 , 4 , 5 ] , existing learning-based robotic grasp detectors [ 1 ] rely primarily on static viewpoint observations, assuming sufficient visibility of the target object from single or pre-defined multi-view perspectives [ 6 , 7 , 8 ] . This constraint reduces flexibility in scenarios where the target object is highly occluded or entirely unobservable. Meanwhile, recent active grasping frameworks [ 9 , 10 , 11 ] have made initial attempts to integrate Next-Best-View (NBV) planning into grasping, which prioritize 3D reconstruction and rendering over target-driven view search for grasping, relying heavily on pre-defined search spaces. However, without such prior knowledge, these methods may fail to handle heavily cluttered scenarios and target invisibility. We argue that effectively reaching the target, even under invisibility, requires a structured active vision and sequential grasping strategy. This shall leverage object relationships to systematically optimize sensor viewpoints, remove obstructions when necessary, and incrementally reveal the target for efficient and successful grasp execution. To this end, we propose VISO-Grasp, a Vision-language Informed Spatial Object-centric grasping framework that integrates off-the-shelf Foundation Models (FMs) [ 12 , 13 , 14 ] with active vision and occlusion-aware \mathrm{SE(3)} grasp planning, which leverages the inherent prior of VLMs to achieve human-like decision-making. In contrast to recent works [ 15 , 16 , 17 , 18 ] , which focus on planar 4-DoF grasp reasoning with VLMs while overlooking their fundamental limitations in accurate 3D spatial understanding and real-time execution, VISO-Grasp introduces evolving spatial reasoning that continuously integrates spatial and occlusion relationships. By incorporating an online grasp fusion mechanism, our approach dynamically refines target visibility and substantially improves grasp efficiency in heavy occlusions. Specifically, when direct grasping is infeasible, a sequential grasp strategy guided by VLMs is employed, leveraging the observable relational hierarchy to determine the optimal grasp sequence for decluttering and removing potential obstructions. In summary, our main contributions are threefold: Fig. 1: VISO-Grasp, a unified system integrating Foundation Models (FMs) into target-aware active view planning and uncertainty-driven real-time 6-DoF grasp fusion. - Vision-language-driven unified scene understanding for robotic perception , which integrates scene reasoning from the VLM [ 12 ] to infer spatial relationships between objects. This facilitates occluder identification through Visual-In-Context Learning (VICL), and determines optimal sequential grasp orders in highly occluded environments. - Target-aware view planning for object-centric grasping , a novel NBV planner that constructs object-centric continuous velocity fields using 3D instance segmentations from existing foundation models [ 14 , 13 ] to optimize sensor viewpoints and maximize target visibility in cluttered environments. - Uncertainty-driven online grasp fusion , a real-time grasp fusion framework that integrates multi-view grasp predictions through pose-centric categorization and real-time Bayesian updates. This leverages a pre-trained uncertainty-aware grasp generator [ 19 ] , exclusively on simulated data, to enhance grasp robustness and reliability. Fig. 2: Overview of VISO-Grasp. Our system consists of: 1) Adaptive Multi-View Open-Vocabulary 3D Object Detector (AMOV3D, Sec. IV-B ): It leverages VLM (Qwen2.5-VL [ 12 ] ), Grounding DINO [ 14 ] and Segment Anything 2 [ 13 ] to extract open-vocabulary 3D object representations. 2) Target-Guided View Planning (TGV-Planner, Sec. IV-C ): The TGV-Planner dynamically adjusts viewpoints using spatial reasoning and Velocity-field-based NBV (V-NBV) to enhance object discovery. 3) Real-Time Uncertainty-guided Multi-view Grasp Fusion (RT-UMGF, Sec. 4 ) to enable occlusion-aware object-centric grasping, RT-UMGF fuses inferred grasps through real-time uncertainty-aware updates in 10Hz. II Related works II-A Active View Search for Grasp Detection Recent advancements in scene-level \mathrm{SE(3)} grasp detection frameworks prioritize generalization to unseen, unstructured environments with diverse objects. While end-to-end approaches [ 20 , 21 , 7 , 22 ] employ deep neural networks to infer grasp configurations from point clouds [ 20 , 21 ] or volumetric representations [ 7 , 22 ] , the robustness diminishes in the presence of sim-to-real gap and heavy occlusions, where incomplete geometric information significantly impairs prediction reliability. To this end, multi-view strategies [ 23 , 6 , 8 , 7 ] integrate observations across multiple viewpoints, while, either bringing the cost of target-agnostic view exploration or relying on pre-defined perspectives by sacrificing flexibility. Within this context, active vision [ 24 ] facilitated by NBV strategies, emerges as a principled approach in vision-based decision-making such as navigation [ 25 , 26 ] and 3D reconstruction [ 27 , 28 ] . By incorporating historical observations, active vision employs a closed-loop optimization process to iteratively select viewpoints and maximize scene visibility for downstream tasks. Recent active grasping frameworks [ 9 , 10 , 11 ] primarily formulate the NBV problem as a correlated reconstruction and rendering task. For instance, they leverage strategies such as maximization of volumetric Information Gain (IG) derived from ray casting [ 9 ] and minimization of neural graspness or rendering inconsistency [ 11 , 10 ] . These approaches are constrained by discrete viewpoint sampling, pre-defined search space prior (e.g., the bounding box around the targets), target-agnostic grasp planning, and the prioritization of reconstruction or rendering over grasping. Consequently, these limitations hinder adaptability for optimal target-oriented grasp and view planning in unstructured environments or even invisibility. In comparison, we consider involving spatial reasoning between objects to realize object-centric active view planning. II-B FMs in Robotic Perception and Planning Foundation Models (FMs), including VLMs, have significantly improved robotic perception and planning by zero-shot object detection [ 29 ] , multi-modal reasoning [ 30 , 17 , 31 ] , and decision-making [ 32 , 33 ] , reducing reliance on task-specific data collection while enhancing adaptability in real-world environments [ 34 , 35 ] . Recent works leverage FMs to enable

Read the complete text