Visuomotor Grasping with World Models for Surgical Robots

Hongbin Lin, Bin Li, and Kwok Wai Samuel Au · 2025

Read Visuomotor Grasping with World Models for Surgical Robots by Hongbin Lin, Bin Li, and Kwok Wai Samuel Au (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2508-11200.txt.

Introduction

Hongbin Lin, Bin Li, and Kwok Wai Samuel Au teach a surgical robot to grasp objects it has never seen, guided by its cameras, in a 2025 paper. Automating grasping could cut surgeon workload in robot-assisted surgery, but earlier systems leaned on explicit object tracking or handcrafted features, failing on new or soft objects. The team's framework, GASv2, learns visuomotor control inside a world model and pairs it with a surgical perception pipeline and hybrid controller for safe execution.

The policy trains in simulation with heavy randomization, then transfers to the robot using only the standard stereo endoscope pair. It reached a sixty-five percent success rate in both phantom and ex vivo tissue settings. One policy handled diverse unseen objects, even different grippers, and adapted to disturbances. The authors present this as evidence that world-model visuomotor learning can meet surgery's precision and safety demands.

Catalog record

CREATORHongbin Lin, Bin Li, and Kwok Wai Samuel Au
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE101 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2508-11200

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Visuomotor Grasping with World Models for Surgical Robots Hongbin Lin, Bin Li, and Kwok Wai Samuel Au Abstract Grasping is a fundamental task in robot-assisted surgery (RAS), and automating it can reduce surgeon workload while enhancing efficiency, safety, and consistency beyond teleoperated systems. Most prior approaches rely on explicit object pose tracking or handcrafted visual features, limiting their generalization to novel objects, robustness to visual disturbances, and the ability to handle deformable objects. Visuomotor learning offers a promising alternative, but deploying it in RAS presents unique challenges, such as low signal-to-noise ratio in visual observations, demands for high safety and millimeter-level precision, as well as the complex surgical environment. This paper addresses three key challenges: (i) sim-to-real transfer of visuomotor policies to ex vivo surgical scenes, (ii) visuomotor learning using only a single stereo camera pair—the standard RAS setup, and (iii) object-agnostic grasping with a single policy that generalizes to diverse, unseen surgical objects without retraining or task-specific models. We introduce Grasp Anything for Surgery V2 (GASv2), a visuomotor learning framework for surgical grasping. GASv2 leverages a world-model-based architecture and a surgical perception pipeline for visual observations, combined with a hybrid control system for safe execution. We train the policy in simulation using domain randomization for sim-to-real transfer and deploy it on a real robot in both phantom-based and ex vivo surgical settings, using only a single pair of endoscopic cameras. Extensive experiments show our policy achieves a 65% success rate in both settings, generalizes to unseen objects and grippers, and adapts to diverse disturbances, demonstrating strong performance, generality, and robustness. \corrauth Hongbin Lin, Kwok Wai Samuel Au keywords Surgical Robots, Reinforcement Learning and World Models 1 Introduction Grasping is a fundamental and frequent action in surgery, essential for tasks such as handling needles (see Fig. 1 ), gauze, thread, sponges, and manipulating tissue during procedures like debridement. In modern robot-assisted surgery (RAS), these tasks are executed via teleoperation, where a surgeon controls the robotic gripper to replicate hand movements. However, teleoperation demands significant physical and cognitive effort due to the repetitive and tedious nature of grasping, contributing to surgeon fatigue. Moreover, outcomes vary widely depending on individual surgeon skill and experience [ 1 ] . Automating surgical grasping could reduce surgeon workload and improve efficiency, safety, and consistency beyond current teleoperated systems. Researchers have extensively explored surgical grasping automation using pose-based methods [ 2 ] and visual-feature-based methods [ 3 ] . These approaches track low-dimensional spatial states—such as object poses or handcrafted image-space feature coordinates–from visual input and feed them into downstream controllers. However, they face three key limitations. First, they often rely on full geometric models or assumed shapes, which limits generalization to unseen objects and introduces assumed geometric bias. Second, state estimation is sensitive to visual disturbances, including occlusion, fogging, and blood. Third, low-dimensional states struggle to represent deformable object configurations. Some approaches use robot proprioception, such as joint measurements, to infer gripper pose without vision. Yet, most surgical robots (e.g., da Vinci Surgical System) suffer from inaccurate forward kinematics due to potentiometer-based sensing, hysteresis, and mechanical slack, which degrades control accuracy [ 4 ] . Visuomotor learning offers a promising alternative to pose-based and visual-feature-based methods, showing strong performance in general robotic grasping [ 5 ; 6 ] . These methods learn end-to-end policies that map visual observations directly to robot actions, implicitly capturing spatial features that are often richer than explicit representations, such as pose or hand-craft visual features. As a result, visuomotor approaches demonstrate improved generalization to out-of-distribution (OOD) objects [ 6 ] , enhanced robustness to visual noise [ 7 ] , and the ability to handle deformable objects [ 8 ] . Despite the success of visuomotor learning in general robotic grasping, applying it to RAS faces several domain-specific challenges. First, in typical RAS camera settings, the task-related pixels on the observed image are extremely scarce, with the ratio of task object pixels (i.e., signal) to background pixels (i.e., noise) being approximately 1:99 [ 9 ] . Such a low signal-to-noise ratio poses significant challenges for training the policy’s encoder. Second, surgical grasping requires millimeter-level precision and safe trajectories to prevent tissue damage, unlike general grasping with coarser centimeter-scale tolerances and lower safety demands. Third, surgical environments are significantly more complex, with patient-specific anatomy, dynamic scene changes, and frequent visual occlusions (e.g., blood, smoke), as well as unpredictable tissue dynamics, all of which challenge both perception and control [ 1 ] . Only a few works have explored visuomotor learning in RAS. A transformer-based policy trained via imitation learning has been applied to surgical scenes in both phantom [ 4 ] and ex vivo settings [ 1 ] , but it requires extensive real-robot data collection, incurring high labor and operational costs. Moreover, it relies on additional wrist-mounted cameras, increasing hardware complexity beyond standard RAS setups (which typically use only endoscopic cameras) and raising concerns about feasibility in in vivo settings due to port constraints. To circumvent real-robot data collection, Scheikl et al. train a visuomotor policy in simulation using PPO and transfer it to phantom scenes via pixel-level domain randomization [ 8 ] . However, its applicability to ex vivo settings remains unclear, especially given its reliance on a RealSense depth camera, which may not be suitable for surgical environments. In this paper, we aim to address three unsolved challenges beyond prior work: (i) sim-to-real transfer of visuomotor policies to ex vivo surgical scenes, (ii) visuomotor learning using only a single stereo camera pair—the standard RAS setup, and (iii) object-agnostic grasping with a single policy that generalizes to diverse, unseen surgical objects without retraining or task-specific models. To this end, we propose a visuomotor learning framework, G rasp A nything for S urgery V2 (GASv2), for surgical grasping (see Fig. 2 ). GASv2 adopts a world-model-based architecture to learn a visuomotor policy. A surgical perception pipeline is leveraged for visual observations of the policy. A hybrid control system integrates the learned policy with traditional controllers for safe and precise execution. We train the policy in simulation with domain randomization for sim-to-real transfer, and deploy it on a real robot in both phantom-based and ex vivo surgical scenes—using only a single pair of endoscopic cameras, consistent with standard RAS settings. In summary, our primary contributions are: 1. A data-driven framework for sim-to-real learning of object-agnostic visuomotor policy for surgical grasping. We present the first successful deployment of a visuomotor policy transferred from simulation to an ex vivo setting, using only visual input from a standard pair of endoscopic cameras. 2. A novel image representation method for visuomotor policy learning in surgical grasping. Our approach encodes depth images, segmentation masks, and system states into a compact 64\times 64\times 3 image. It preserves high-resolution detail in task-critical local regions via dynamic zooming, while meeting image size constraints imposed by the limited capacity of our world-model-based policy. 3. Extensive experiments in simulation and on real robots to evaluate the performance, generalization, and robustness of our learned visuomotor policy. Our policy achieves strong performance, with a 65% success rate in both phantom-based and ex vivo scenes. It generalizes to unseen objects and grippers, and adapts to diverse disturbances, demonstrating its generality and robustness. Figure 1:

Read the complete text