Video-based Surgical Tool-tip and Keypoint Tracking using Multi-frame Context-driven Deep Learning Models

Bhargav Ghanekar, Lianne R. Johnson, Jacob L. Laughlin, Marcia K. O’Malley, Ashok Veeraraghavan · 2025

Read Video-based Surgical Tool-tip and Keypoint Tracking using Multi-frame Context-driven Deep Learning Models by Bhargav Ghanekar, Lianne R. Johnson, Jacob L. Laughlin, Marcia K. O’Malley, Ashok Veeraraghavan (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2501-18361.txt.

Introduction

Bhargav Ghanekar, Lianne Johnson, Jacob Laughlin, Marcia O'Malley, and Ashok Veeraraghavan track the keypoints of surgical tools in robotic surgery videos. Automated tracking of tool tips and bases underpins downstream uses such as skill assessment, expertise assessment, and delineating safety zones. Deep learning has produced many instrument segmentation systems, but tracking specific keypoints received less attention.

The authors propose a multi-frame, context-driven deep learning framework that localizes and tracks keypoints across video frames rather than treating each frame alone. Trained and tested on annotated frames from the twenty fifteen EndoVis Challenge dataset, it reaches state-of-the-art performance, with high detection accuracy and pixel-level localization error. On a self-annotated surgical skills dataset with more challenging scenarios, the multi-frame models again tracked tool-tip and tool-base keypoints accurately. Temporal context is key: an occluded tool remains findable from neighboring frames.

Catalog record

CREATORBhargav Ghanekar, Lianne R. Johnson, Jacob L. Laughlin, Marcia K. O’Malley, Ashok Veeraraghavan
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE18 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2501-18361

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Video-based Surgical Tool-tip and Keypoint Tracking using Multi-frame Context-driven Deep Learning Models Bhargav Ghanekar, Lianne R. Johnson, Jacob L. Laughlin, Marcia K. O’Malley, Ashok Veeraraghavan Abstract Automated tracking of surgical tool keypoints in robotic surgery videos is an essential task for various downstream use cases such as skill assessment, expertise assessment, and the delineation of safety zones. In recent years, the explosion of deep learning for vision applications has led to many works in surgical instrument segmentation, while lesser focus has been on tracking specific tool keypoints, such as tool tips. In this work, we propose a novel, multi-frame context-driven deep learning framework to localize and track tool keypoints in surgical videos. We train and test our models on the annotated frames from the 2015 EndoVis Challenge dataset, resulting in state-of-the-art performance. By leveraging sophisticated deep learning models and multi-frame context, we achieve 90% keypoint detection accuracy and a localization RMS error of 5.27 pixels. Results on a self-annotated JIGSAWS dataset with more challenging scenarios also show that the proposed multi-frame models can accurately track tool-tip and tool-base keypoints, with < 4.2 {<}4.2 -pixel RMS error overall. Such a framework paves the way for accurately tracking surgical instrument keypoints, enabling further downstream use cases. Project and dataset webpage: https://tinyurl.com/mfc-tracker † † address: Rice University, Houston TX USA 1 Introduction Robotic minimally invasive surgery (MIS) has seen significant growth over the past two decades [ 1 ] . With more and more robotic surgeries performed every year, there is a need for surgeons to be trained and skilled in robotic surgery. Most of the learning feedback is given through the process of viewing robotic surgery videos by experts, which is time-consuming and subjective. Automated detection and tracking of robotic surgical tools can enable automated surgical skill and learning assessment. Real-time tool tracking can enable computer-assisted interventions, such as marking safe working zones, etc. While there have been recent works in surgical instrument detection and instrument segmentation [ 2 , 3 ] , there have been only a few works focusing on tracking keypoints on surgical tools and instruments, including tool-tips, tool jaw base, and tool shaft point. Estimating keypoints present on surgical tools while performing tasks is a challenging and interesting computer vision task. The ability to track tool-tips in an automated manner from videos can enable downstream video analytics and analysis of surgical tool movement, which can be very useful for surgical skill assessment and judging expertise of motor skill movements [ 4 ] . With these use cases in mind, surgical tool keypoint localization and tracking can become challenging due to changing lighting conditions, temporary occlusions, frame focusing issues, motion blur artifacts, and awkward tool positions and orientations. In this work, we propose the use of deep learning-based (DL-based) supervised models that can reliably track surgical tool keypoints such as tool-tips, tool-base, and other relevant keypoints. We pose the tracking problem as a segmentation task to localize small segments (regions) around tool keypoints. By leveraging sophisticated deep learning segmentation architectures [ 5 , 6 ] , and considering optical flow and depth predictions from state-of-the-art optical flow and monocular depth estimation models [ 7 , 8 ] , our proposed tracking method (see Fig. 1 ) is able to use a multi-frame context to segment out keypoint regions and subsequently track keypoints of surgical tools from video frames. Figure 1: Proposed keypoint tracking workflow. We perform tracking by (i) segmenting out keypoint regions (keypoint ROIs) using a deep learning-based segmentation model, and (ii) estimating the centroids of segmented output blobs as tool keypoint locations. Figure 2: Multi-frame context (MFC) model design for keypoint ROI segmentation. Predictions of K consecutive frames from a trained single-frame context model are computed. Alongside the segmentation maps, K{-}1 optical flow maps and K depth maps are also computed. These maps are passed into MFCNet to aggregate multi-frame context and thus estimate accurate keypoint ROI segmentation predictions. 2 Related Work Surgical instrument detection and segmentation have been of significant interest in recent years, with several works deploying supervised deep learning solutions [ 2 , 3 , 9 ] . Although significant progress has been made on surgical instrument segmentation, less progress has been made on developing vision-based methods for tracking keypoints of surgical tools. Tracking keypoints and pose are a well-known problem in computer vision. However, most of the modern deep learning solutions focus on estimating human poses and require training on large amounts of data [ 10 , 11 ] . Non-human keypoint/pose estimation works mainly focus on animal poses [ 12 ] . Estimation and tracking for surgical instruments is more challenging, particularly due to scarcity of open-source annotated datasets. Some early works used traditional machine learning-based methods [ 13 , 14 ] , and more recent works [ 15 , 16 , 17 ] have looked at tracking various tool keypoints using convolutional neural networks. Tracking tool-tips and other keypoints in videos can be challenging due to occlusions, motion blur, and video compression artifacts. Our proposed tracking method aims to build an effective tracking method that is capable of handling awkward tool poses and rapid tool motions using only a small annotated dataset. 3 Methods We aim to track various keypoints of surgical tools from videos using novel DL-based trackers. We propose to address the keypoint tracking problem in a two-step manner: 1. We first aim to segment out small regions around keypoints (henceforth referred as keypoint regions or keypoint ROIs). This is modeled as a per-pixel multi-class segmentation problem, with each keypoint region of each tool (left/right) being assigned a different class. We aim to train deep learning-based segmentation models that can successfully segment small keypoint regions. 2. Once segmentation outputs delineating the keypoint regions are obtained, we localize the keypoints by estimating contiguous segmented regions (blobs) and estimating their centroid as the keypoint coordinate. Our workflow is illustrated in Fig. 1 . 3.1 Keypoint region segmentation models In the proposed framework, we group our proposed keypoint region segmentation models into two categories: • Single-frame context (SFC) models . In SFC models, we predict a segmentation output given an input frame at a particular time instance. The segmentation model outputs are independent of past/future video frames. Any deep learning model that performs 2D semantic segmentation [ 2 , 5 , 6 ] given an input image can be thought of as a SFC model. • Multi-frame context (MFC) models . To make our tracking more robust, we propose a keypoint ROI segmentation model that takes into consideration a multi-frame, moving window context. K consecutive video frames are input, each processed by a single-frame context model (SFC model) to produce keypoint ROI segmentation maps for all K frames. These outputs are refined through an additional refinement model, which we refer to as MultiFrameNet (MFCNet). To provide a better multi-frame context, we propose using optical flow maps and monocular depth estimation maps as auxiliary inputs. Optical flow maps represent the 2D pixel displacement between consecutive video frames, capturing the motion and direction of objects within the scene. Monocular depth estimation maps predict the per-pixel depth from a single RGB frame. Both estimates offer a temporal and geometric context for keypoint positioning. We use

Read the complete text