Confidence-based Intent Prediction for Teleoperation in Bimanual Robotic Suturing

Zhaoyang Jacopo Hu, Haozheng Xu, Sion Kim, Yanan Li, Ferdinando Rodriguez y Baena, Etienne Burdet · 2025

Read Confidence-based Intent Prediction for Teleoperation in Bimanual Robotic Suturing by Zhaoyang Jacopo Hu, Haozheng Xu, Sion Kim, Yanan Li, Ferdinando Rodriguez y Baena, Etienne Burdet (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2504-20761.txt.

Introduction

Zhaoyang Jacopo Hu and co-authors built a helper system in 2025 for surgeons who guide robot arms through stitches from a console. Remote control gives the surgeon fine motion but can suffer from delay and weak feedback. Full automation, in turn, struggles with messy, changing scenes.

Their system guesses the surgeon's intent at two levels. A gesture model reads the surgeon's moves as they happen, while a control layer blends robot motion with the guessed plan based on how sure it is. Sensors record the robot's motion and the forces at work. In trials with users of mixed skill, the helper cut task time and raised user satisfaction compared with plain remote control.

Catalog record

CREATORZhaoyang Jacopo Hu, Haozheng Xu, Sion Kim, Yanan Li, Ferdinando Rodriguez y Baena, Etienne Burdet
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE35 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2504-20761

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Confidence-based Intent Prediction for Teleoperation in Bimanual Robotic Suturing Zhaoyang Jacopo Hu, Haozheng Xu, Sion Kim, Yanan Li, Ferdinando Rodriguez y Baena, Etienne Burdet Abstract Robotic-assisted procedures offer enhanced precision, but while fully autonomous systems are limited in task knowledge, difficulties in modeling unstructured environments, and generalisation abilities, fully manual teleoperated systems also face challenges such as delay, stability, and reduced sensory information. To address these, we developed an interactive control strategy that assists the human operator by predicting their motion plan at both high and low levels. At the high level, a surgeme recognition system is employed through a Transformer-based real-time gesture classification model to dynamically adapt to the operator’s actions, while at the low level, a Confidence-based Intention Assimilation Controller adjusts robot actions based on user intent and shared control paradigms. The system is built around a robotic suturing task, supported by sensors that capture the kinematics of the robot and task dynamics. Experiments across users with varying skill levels demonstrated the effectiveness of the proposed approach, showing statistically significant improvements in task completion time and user satisfaction compared to traditional teleoperation. I Introduction In traditional teleoperation the human operator fully controls the robot’s movements [ 1 ] . While this offers clear roles and responsibilities, it under-utilizes both the human and robot capabilities. Robots like the da Vinci Surgical System are equipped with sensors and models offering valuable local information inaccessible to the human operator, such as during visual occlusions or operations with different sensory modalities. By spanning across the spectrum between traditional fully manual teleoperation and full autonomy, shared control leverages the benefits of both to enhance teleoperation with the robot’s sensory data and control [ 2 ] . While demonstrated for suturing assistance [ 3 , 4 ] , these methods overlook the impact on positional uncertainty, environmental unknowns, or instrument errors. For example, robotic surgery cameras are frequently occluded by body tissues or parts of the robot [ 5 ] . Fig. 1: Bimanual robotic suturing with four key surgemes: (1) Needle tip positioned at the entry point; (2) Needle pushed/inserted reaching the exit point; (3) Needle pulled out; (4) Needle passed to the starting manipulator. To address these shortcomings, this paper develops an interaction control framework for the da Vinci Surgical System, framing shared control as an information exchange between human user and robot agent based on the user’s intent [ 6 ] . Inspired by studies on human-human collaboration [ 7 , 8 ] , this approach shares not only current positions but also motion plans between the partners, enabling them to anticipate each other’s actions and optimize movement control [ 9 ] . At task level , the robot identifies the operator’s desired high-level actions through specific hand gestures, or surgemes [ 10 ] . These surgemes can be used to infer the surgeon’s task throughout the procedure (Fig. 1 ). For instance, online gesture classification has been used to automate camera motion [ 11 ] or switch assistance modes [ 12 ] . After recognizing a surgeme, the robot estimates the operator’s motion intention at a lower control level to adapt seamlessly during collaborative tasks, akin to the interaction observed in connected humans carrying out a tracking task [ 9 , 6 ] . In [ 13 , 14 , 15 , 16 ] the authors suggest that adaptation and incorporation of motion intentions can minimize the operator’s effort. The intention assimilation controller (IAC) [ 13 ] enables the robot to identify the operator’s motion plan and combine it with its own based on task requirements and sensor input. However, [ 13 ] did not specify the criteria for transitioning between interaction strategies, which may be regulated based on the robot’s confidence in its sensors [ 6 , 17 ] . Confidence scores can help avoid collisions [ 18 ] or improve trajectory tracking [ 19 ] , facilitating human-robot negotiation based on information received. In this paper, we assume camera data to be more accurate than forward kinematics for position estimation, but prone to occlusions and fluctuations. We introduce an interaction control framework that integrates task-level prediction with confidence-based IAC (C-IAC), implemented on the da Vinci Research Kit (dVRK). The dVRK offers high-definition cameras but limited sensory feedback, making tasks like suturing tedious, time consuming, and attention-intensive [ 20 ] . We hypothesize that, by combining sensing and motion planning of the human and robot, one can use sensory augmentation to enhance suturing performance and comfort. To test this, we developed a kinematic suturing dataset and trained a Transformer model to infer suturing activity in real-time on the dVRK by decomposing it into surgemes. We also developed a mechanical device to automatically orient the needle relative to the manipulator. We then designed an IAC-based controller with confidence level computed from Bayesian inference analyzing kinematic and camera data. Two user studies evaluated the entire system’s performance. The main contributions of this paper are as follows: • A novel needle holder for facilitated grasping. • Integration of shared control with a gesture recognition model for bimanual robotic suturing using the dVRK. • Implementation of a dynamic parameter to compute the confidence component based on Bayesian inference. • Development of a confidence-based IAC controller used to teleoperate the da Vinci robot under shared control. • User studies to evaluate the performance of the C-IAC . Fig. 2: Design of the Compact Holder for Enhanced Needle Alignment (CHENA). (a-b) The structure comprises of walls that create a concavity where the needle is attracted and maintained fixed by a magnet. (c-d) The needle self-aligns using to two independent acting forces: the jaw pushing and the magnet pulling the needle towards the cavity. II Surgical Setting II-A Compact Holder for Enhanced Needle Alignment During suturing, operators often require multiple manipulations of the needle to achieve the optimal orientation [ 21 , 22 ] . Previous attempts in solving this problem include [ 23 ] , which created a component whose geometry enabled needle alignment and that could be easily attached to the existing surgical tool without requiring a complete replacement. However, the device increases the size of the end-effector, making needle handling awkward. We developed a low-cost Compact Holder for Enhanced Needle Alignment (CHENA), improving prior designs [ 23 , 22 ] by facilitating needle grabbing and holding (Fig. 2 ). The aligner in [ 23 ] was intended for autonomous suturing and required a large rear wall and catching area to guide the needle. The necessity of these parts force the components to extrude over the opposite side of the surgical tool, limiting its maneuverability and maximum jaw open angle. In contrast, CHENA self-aligns and positions the needle without the need for large extrusion in the catching area by using a minimalistic geometry and a magnet to create a driving force for the needle to align even without closing the jaws. II-B dVRK Setup For the experiments, we use the dVRK equipped with two Patient Side Manipulators (PSMs): right PSM1 with DeBakey Forceps and CHENA, and left PSM2 with Large Needle Drivers. The needle used is a 1/2 circle, 24 mm chord length with triangular cutting point. Robot control is performed in Cartesian space

Read the complete text