Towards Real-Time Autonomous Navigation: Transformer-Based Catheter Tip Tracking in Fluoroscopy

Harry Robertshaw, Yanghe Hao, Weiyuan Deng, Benjamin Jackson, Nikola Fischer, Tom Vercauteren Alejandro Granados, Thomas C. Booth Surgical & Interventional EngineeringSchool of Biomedical Engineering & Imaging SciencesKings College LondonLondonS.M.Hadi SadatiSchool of Engineering & Materials ScienceQueen Mary LondonLondon · 2026

Read Towards Real-Time Autonomous Navigation: Transformer-Based Catheter Tip Tracking in Fluoroscopy by Harry Robertshaw, Yanghe Hao, Weiyuan Deng, Benjamin Jackson, Nikola Fischer, Tom Vercauteren Alejandro Granados, Thomas C. Booth Surgical & Interventional EngineeringSchool of Biomedical Engineering & Imaging SciencesKings College LondonLondonS.M.Hadi SadatiSchool of Engineering & Materials ScienceQueen Mary LondonLondon (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2605-14253.txt.

Introduction

Robertshaw and eight co-authors develop a live catheter tip tracking pipeline under fluoroscopy for autonomous navigation. Mechanical thrombectomy improves stroke outcomes but is limited by local treatment access. Reinforcement learning robots could ease this gap, but need live tip tracking. The pipeline handles low contrast, noise and occlusion with frame reading, preprocessing and inference. U-Net, U-Net+Transformer and SegFormer were benchmarked in two-class and three-class forms. Two-class SegFormer reached a mean error of 4.44 mm, beating U-Net and three-class models.

Catalog record

CREATORHarry Robertshaw, Yanghe Hao, Weiyuan Deng, Benjamin Jackson, Nikola Fischer, Tom Vercauteren Alejandro Granados, Thomas C. Booth Surgical & Interventional EngineeringSchool of Biomedical Engineering & Imaging SciencesKings College LondonLondonS.M.Hadi SadatiSchool of Engineering & Materials ScienceQueen Mary LondonLondon
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE32918
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2605.14253

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Towards Real-Time Autonomous Navigation: Transformer-Based Catheter Tip Tracking in Fluoroscopy Thanks: Citation : Robertshaw, H., Hao, Y., Deng, W. et al. Toward real-time autonomous navigation: transformer-based catheter tip tracking in fluoroscopy. Int J CARS (2026). https://doi.org/10.1007/s11548-026-03647-7 Harry Robertshaw, Yanghe Hao, Weiyuan Deng, Benjamin Jackson, Nikola Fischer, Tom Vercauteren Alejandro Granados, Thomas C. Booth Surgical & Interventional EngineeringSchool of Biomedical Engineering & Imaging SciencesKings College LondonLondonS.M.Hadi SadatiSchool of Engineering & Materials ScienceQueen Mary LondonLondon Abstract Purpose: Mechanical thrombectomy (MT) improves stroke outcomes, but is limited by a lack of local treatment access. Widespread distribution of reinforcement learning (RL)-based robotic systems can be used to alleviate this challenge through autonomous navigation, but current RL methods require live device tip coordinate tracking to function. This paper aims to develop and evaluate a real-time catheter tip tracking pipeline under fluoroscopy, addressing challenges such as low contrast, noise, and device occlusion. Methods: A multi-threaded pipeline was designed, incorporating frame reading, preprocessing, inference, and post-processing. Deep learning segmentation models, including U-Net, U-Net+Transformer, and SegFormer, were trained and benchmarked using two-class and three-class formulations. Post-processing involved two-step component filtering, one-pixel medial skeletonization, and greedy arc-length path following with contour fall-back. Results: On manually-labeled moderate complexity fluoroscopic video data, the two-class SegFormer achieved a mean absolute error of 4.44 mm, outperforming U-Net (4.60 mm), U-Net+Transformer (6.20 mm) and all three-class models (5.19-7.74 mm). On segmentation benchmarks, the system exceeded state-of-the-art CathAction results with improvements of up to +5% in Dice scores for three-segmentation. Conclusion: The results demonstrate that the proposed multi-threaded tracking framework maintains stable performance under challenging imaging conditions, outperforming prior benchmarks, while providing a reliable and efficient foundation for RL-based autonomous MT navigation. K eywords catheter tip tracking \cdot fluoroscopy \cdot mechanical thrombectomy \cdot deep learning segmentation 1 Introduction Mechanical thrombectomy (MT) is an essential endovascular intervention in treating stroke due to large vessel occlusion [ 9 ] . However, its efficacy declines beyond 7.3 h of stroke onset, contributing to low treatment rates despite up to 15% of patients being eligible [ 19 , 20 ] . MT can be associated with vessel perforations (1%), procedure-related vessel dissections (2%), or distal embolization of thrombus (9%) [ 2 ] . Cumulative X-ray exposure endangers clinicians with cancer and cataract risks, compounded by the musculoskeletal burden of lead garments [ 11 , 12 ] . Endovascular robotic platforms mitigate radiation exposure and ergonomic strain [ 5 ] , but their controller-operator design incurs high cognitive load and is susceptible to human error [ 13 ] . Recent research has investigated AI-based autonomous MT navigation, using reinforcement learning (RL) and robotics [ 16 , 15 ] . The current state-of-the-art utilizes a ‘state-based’ RL approach, requiring information such as the device tip coordinates, target coordinates, and previous actions [ 17 ] . However, current methods for device tracking and tip localization are unproven under clinically realistic conditions, and it is unknown whether existing deep learning models can generalize across variable imaging, such as camera/fluoroscopy, while maintaining robust tip localization when it is hidden or poorly visible in real-time. To realize the benefits of previous autonomous endovascular navigation research, it is necessary to develop accurate and robust device tip localization methods, which can provide input to RL agents as they are transferred from in silico to in vitro , ex vivo , and in vivo testbeds. Furthermore, reliable tracking of both guidewire and catheter tip (i.e. 3-class formulation) will have a higher impact on training RL agents [ 16 ] on environments related to real interventions. This will allow continuous development of autonomous tele-operated robotic platforms, promoting increased accessibility of expert-level treatment to underserved regions. Current device tracking methods rely on deep networks such as U-Net and instance-segmentation models, aiming to deliver full-catheter masks at \sim 8 fps with <1 mm tip error without manual setup [ 18 , 1 ] . Two-stage detectors with multi-task centerline/tip predictors and unsupervised optical-flow have been used to improve robustness when poorly visible due to low image contrast or when hidden by overlapping objects (occlusion) [ 14 ] . Additionally, ‘weakly supervised’ learning has been shown to reduce annotation needs [ 21 ] , while transformers have been used to capture broader context [ 6 ] . However, the absence of a large, open-source training dataset has limited performance when testing on complicated and realistic clinical environments. The CathAction dataset was recently introduced with \sim 500{,}000 annotated frames and 25{,}000 masks with separate instrument classes, facilitating pixel-level tracking [ 8 ] . Segmentation results on this dataset are currently benchmarked using U-Net [ 18 ] , TransUNet [ 4 ] , SwinUNet [ 3 ] , and SegViT [ 23 ] . Most existing models have been evaluated primarily on in silico or in vitro phantom datasets, but it is unclear whether they generalize to in vitro live-camera data used in experimental testbeds, or to real in vivo clinical fluoroscopy. Furthermore, to our knowledge, the CathAction dataset has not yet been used to develop a live, real-time catheter tip detection or tracking algorithm across any experimental medium ( in vitro through to in vivo ), representing an important opportunity for advancing autonomous endovascular navigation. The aim of this study was to propose a deep learning-based catheter and guidewire segmentation and tip-tracking method, leveraging the CathAction dataset, to enable accurate, real-time device localization under clinically challenging conditions. The primary objective was to demonstrate that modern segmentation architectures (U-Net, U-Net+Transformer, and SegFormer) can be effectively integrated into a downstream tracking pipeline, rather than evaluated as standalone segmentation models, to achieve high accuracy and low tip-localization error. The secondary objectives were to evaluate the generalizability of the proposed tracking framework across clinically relevant footage, and to benchmark two-class (instrument vs. background) versus three-class (catheter vs. guidewire vs. background) segmentation in the context of downstream tracking accuracy rather than segmentation performance alone. Our contributions are: 1) We introduce a novel guidewire and catheter tip-tracking pipeline based on skeletonization and multi-point sampling, improving localization in the presence of low image contrast, partial occlusion, and overlapping instruments; 2) we demonstrate that, when embedded within this tracking framework, modern segmentation architectures achieve improved performance compared to the existing benchmarks on the CathAction dataset, extending beyond prior segmentation-only evaluations; 3) We validate the proposed method across in vitro phantom, in vivo animal, and in vivo clinical fluoroscopic data, demonstrating its feasibility for real-time catheter tip tracking and its potential role in enabling AI-driven endovascular navigation systems. 2 Methods 2.1 Dataset Training data: The action-based dataset from CathAction was used for training, focusing on the segmentation task component [ 8 ] . The dataset comprised 25{,}000 pixel-level reference standard masks for manually-segmented catheters and guidewires. These devices were captured in 569 fluoroscopic videos derived from two different sources: 1) five in vitro human-like vascular silicone phantoms, offering homogeneous imaging conditions with consistent contrast and minimal motion artifacts; 2) five in vivo porcine subjects, giving heterogeneous imaging conditions with cardiac motion, vessel deformation, and variable contrast agent distribution. As in the CathAction study, we use the phantom data for training. All videos were processed at 500\times 500 resolution and 24 fps. Evaluation data: Four types of data were used during evaluation: 1)

Read the complete text