Occlusion‐robust markerless surgical instrument pose estimation Authors: Haozheng Xu, Stamatia Giannarou Published: 2024 ABSTRACT AbstractThe estimation of the pose of surgical instruments is important in Robot‐assisted Minimally Invasive Surgery (RMIS) to assist surgical navigation and enable autonomous robotic task execution. The performance of current instrument pose estimation methods deteriorates significantly in the presence of partial tool visibility, occlusions, and changes in the surgical scene. In this work, a vision‐based framework is proposed for markerless estimation of the 6DoF pose of surgical instruments. To deal with partial instrument visibility, a keypoint object representation is used and stable and accurate instrument poses are computed using a PnP solver. To boost the learning process of the model under occlusion, a new mask‐based data augmentation approach has been proposed. To validate the model, a dataset for instrument pose estimation with highly accurate ground truth data has been generated using different surgical robotic instruments. The proposed network can achieve submillimeter accuracy and the experimental results verify its generalisability to different shapes of occlusion.Keywords: endoscopes, medical robotics, pose estimation INTRODUCTION Robot‐assisted minimally invasive surgery (RMIS) has evolved significantly in the last decades thanks to the advances in Artificial Intelligence (AI) and surgical robotics such as the da Vinci™platform, which provide surgical assistance through enhanced visualisation and feedback control. An important task in RMIS is the tracking of surgical tools. This involves the estimation of the 3D position and orientation of the tool as it moves with six degrees of freedom (6DoF). In surgical tracking tasks, external hardware such as depth cameras, and optical and electromagnetic trackers have been widely used [1, 2]. These methods may require markers to be attached to surgical tools and introduce extra equipment to the operating theatre. However, this can be costly, and impractical and it requires additional hardware calibration and software installation. In contrast, vision‐based methods provide a practical and cost‐effective approach to tool tracking without requiring any modifications on the hardware setup or the attachment of external markers. Early vision‐based methods for surgical tool tracking include marker‐based and markerless approaches. Since most of the surgical instruments including those of the Da Vinci™ surgical robotic system are cylindrical objects, emphasis has been given to the design of cylindrical markers [3, 4]. These patterns consist of an array of blobs and dots which are detected on the 2D image plane to estimate the 6DoF pose of the instrument. One limitation of these methods is that the marker needs to be kept in the camera's field of view (FoV). In addition, the attachment of external markers requires sterilization and calibration. This has steered the research focus into the development of markerless methods based on computer vision algorithms. Most surgical instrument tracking methods [5, 6, 7] consist of two steps, (i) instrument segmentation and (ii) pose estimation. These methods first segment the 2D mask of the instrument on the image, then estimate the 3D pose given the prior knowledge of the 3D model and geometry primitives. However, these two‐stage methods make it difficult to accurately estimate the rotation along the central axis of instruments, since the region change on the image is insensitive to the axial rotation. These two‐stage methods rely on the accurate detection of image features like the tip or center line of the instrument, which is unstable in low‐light or high‐reflection scenarios due to endoscopic illumination. Recently, some deep learning methods were proposed to directly estimate the 3D pose of the object in natural scenes but they have several challenges in surgical scenarios [8, 9, 10, 11, 12, 13]. The first challenge is the partial visibility of the instrument. The surgical camera needs to be very close to the instrument and tissue due to the limited operating space. In this case, only part of the instrument will be in the FOV which may affect the performance of pose estimation methods based on object detection. The second challenge is the occlusion of the tracked instrument. The surgical tool will frequently interact with organs and tissue, which can occlude the tip of the instrument. The visual features of the tip are vital for the pose estimation and occlusions can make the pose estimation unstable. Furthermore, the varying lighting conditions in the surgical environment and specular reflections on the tissue and the tool, affect significantly the appearance and texture of the instrument. Another challenge is the difficulty of acquiring vast and accurate training data. Although the pose of the instrument can be acquired from the kinematic information of a surgical robotic arm, the error of this estimated pose is in the range of several mm. Hence, robot kinematics can not be used to acquire ground truth tool poses and another reliable method needs to be developed to acquire a large amount of image and tool pose data. In this work, we propose a vision‐based framework to estimate the 6DoF pose of surgical instruments without relying on external markers. To enable our method to efficiently deal with partial object visibility, a keypoint object representation is used. For this purpose, a keypoint prediction module is introduced to detect 2D keypoints on the shaft of the instrument. These keypoints correspond to 3D points sampled from the CAD model of the instrument. Stable and accurate object pose is computed using the PnP solver [14] based on the 2D‐3D correspondences. Our contributions are: 1.A feature backbone has been designed by adapting the HRNet to extract features at multiple resolutions and achieve high performance on high‐resolution endoscopic images. A feature backbone has been designed by adapting the HRNet to extract features at multiple resolutions and achieve high performance on high‐resolution endoscopic images. 2.A new mask‐based data augmentation approach has been proposed to increase the robustness of the pose estimation to partial instrument occlusion. A new mask‐based data augmentation approach has been proposed to increase the robustness of the pose estimation to partial instrument occlusion. 3.To validate our model, a dataset for 6DoF instrument pose estimation with highly accurate ground truth data has been generated using different surgical robotic instruments. The dataset will become publicly available upon publication of this work. To validate our model, a dataset for 6DoF instrument pose estimation with highly accurate ground truth data has been generated using different surgical robotic instruments. The dataset will become publicly available upon publication of this work. The proposed network can achieve sub‐millimeter accuracy. Also, our experimental results verify its generalisability and robustness to different shapes of occlusion. METHODS Our approach is designed to determine the 6DoF pose of a surgical instrument utilizing a single RGB image. Given an image I and a collection of n sparse 3D points {zi}i=1n located on the instrument, our initial step is projecting the sparse 3D points onto the image, generating a corresponding set of 2D keypoints {xi}i=1n onto the image. These 3D points are sampled from the instrument's CAD model by applying the farthest point sampling (FPS) algorithm [15]. Subsequently, the instrument's pose is derived through a RANSAC‐based Perspective‐n‐Point (PnP) methodology, which utilizes the 2D‐3D point correspondences for pose estimation. Moreover, our method attains resilience against partial occlusion of the instrument by implementing mask‐based data augmentation. Keypoint prediction module This module aims to identify on the instrument the 2D keypoints {xi}i=1n. We propose a two‐stage pipeline. First, the keypoint prediction module predicts the 2D location of each keypoint, then the PnP algorithm is used to estimate the 6DoF pose given the known 3D location of each keypoint. For this purpose, high‐resolution feature maps which contain rich semantic and texture information are extracted by adapting the HRNet V2 model [16]. The HRNet is composed of multiple branches with different resolutions. This allows the network to extract both high‐level semantic representations and low‐level spatial features from the image data. The stage 4 outputs of HRNet V2 consist of 4 different scale feature maps representing high‐to‐low level image features. In the original HRNet V2 paper, only the largest feature map is used for segmentation. The other feature maps are discarded due to size incompatibility. In our framework, the 4 different scale feature maps are upsampled to the size of the largest map first. Then two 1x1 convolution kernels are added to fuse the feature maps to maintain high‐resolution image representations through the whole process. Our experiments have shown that combining the 4 different scale feature maps increases the model's performance. The concatenated feature map is fed to the instrument segmentation branch and the vector‐field prediction branch. These branches consist of 1×1 convolutions. The proposed model is shown in Figure 1. The overview of our proposed pose estimation network. RGB images labeled with semantic masks and 2D keypoint locations are used to train the model. Rather than directly regressing the 2D keypoint localization like [9], in this work a special format of unit vector maps [10] is used to represent the keypoint localization. It is based on the property of a rigid body where the relative position between different parts of a rigid body is fixed no matter how it translates or rotates. Once one part of the rigid body is visible, the rest of the rigid body parts can be inferred even under occlusion. For an image containing M pixels, the semantic segmentation branch outputs the segmentation map {segj}j=1M and the vector‐field prediction branch outputs the unit vector maps {vi,j}i=1,j=1n,M. To localize keypoints, the outputs from the two branches are combined to generate the filtered unit vector map {vi,j}i=1,j=1n,m where m