Evaluating Large Vision–language Models for Surgical Tool Detection
Nakul Poudel, Richard Simon, Cristian A. Linte · 2026
Read Evaluating Large Vision–language Models for Surgical Tool Detection by Nakul Poudel, Richard Simon, Cristian A. Linte (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2601-16895.txt.
Introduction
Nakul Poudel, Richard Simon, and Cristian Linte test sight-and-text AI for surgery help. Surgery is a hard craft with fast, high-stakes calls. AI could back those calls with live guidance. Most surgery AI sees in one mode, sight or text. That lone view caps its grasp of the work flow. Their 2026 preprint grades big sight-text models on tool spotting.
Tool spotting marks each tool in the surgery view. The team checks how well the big models do it. Sound grades show if such models are fit for the room. They also show where lone-mode tools still win. Safe help in surgery needs proof of this sort.
Catalog record
| CREATOR | Nakul Poudel, Richard Simon, Cristian A. Linte |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 17 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2601-16895
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Evaluating Large Vision–language Models for Surgical Tool Detection Nakul Poudel, Richard Simon, Cristian A. Linte Abstract Surgery is a highly complex process, and artificial intelligence has emerged as a transformative force in supporting surgical guidance and decision-making. However, the unimodal nature of most current AI systems limits their ability to achieve a holistic understanding of surgical workflows. This highlights the need for general-purpose surgical AI systems capable of comprehensively modeling the interrelated components of surgical scenes. Recent advances in large vision–language models that integrate multimodal data processing offer strong potential for modeling surgical tasks and providing human-like scene reasoning and understanding. Despite their promise, systematic investigations of VLMs in surgical applications remain limited. In this study, we evaluate the effectiveness of large VLMs for the fundamental surgical vision task of detecting surgical tools. Specifically, we investigate three state-of-the-art VLMs, Qwen2.5, LLaVA1.5, and InternVL3.5, on the GraSP robotic surgery dataset under both zero-shot and parameter-efficient LoRA fine-tuning settings. Our results demonstrate that Qwen2.5 consistently achieves superior detection performance in both configurations among the evaluated VLMs. Furthermore, compared with the open-set detection baseline Grounding DINO, Qwen2.5 exhibits stronger zero-shot generalization and comparable fine-tuned performance. Notably, Qwen2.5 shows superior instrument recognition, while Grounding DINO demonstrates stronger localization. Keywords: Image-guided surgery, surgical robotics, human-robot collaboration, holistic scene understanding, vision-language model, future directions in surgical AI 1 Introduction Surgical environments are inherently complex, high-risk, and cognitively demanding. As procedures increasingly shift toward minimally invasive and robot-assisted techniques, the integration of Artificial Intelligence (AI) has emerged as a transformative force. Surgical AI systems provide critical intraoperative guidance and decision support, enhancing both surgical precision and patient safety [ 5 ] . However, most current surgical AI systems are unimodal and task-specific, with limited modeling of interactions among surgical tasks. Developing such capabilities is important for achieving comprehensive scene understanding, augmenting entire surgical workflows, and enabling future autonomous robotic assistants [ 8 , 12 ] . Recent advances in vision-language models (VLMs) take multimodal input, i.e., images and rich textual descriptions, thereby enabling a more comprehensive understanding of the input data and modeling interactions among tasks. In the surgery domain, such models can encode information from surgical images and textual annotations, including the presence and type of instruments, anatomical structures, surgical actions, and their spatial relationships. This approach can enable a single model to perform multiple tasks, including instrument and tissue detection, surgical phase recognition, action recognition, step recognition, and critical view of safety. Moreover, the interdependencies among these tasks enable mutual reinforcement, leading to a more holistic understanding of surgical scenes. Despite their promise, the effectiveness of VLMs in surgical environments depends on their visual perception capability. Errors in early visual perception can propagate to higher-level reasoning modules, potentially leading to unsafe or misleading predictions. Among all surgical tasks, surgical tool detection and localization can be considered the fundamental component, as they directly support downstream tasks such as action recognition, phase recognition, and skill assessment. However, surgical scenes present substantial visual challenges, including occlusion, blood, smoke, motion blur, illumination variation, and poor contrast, which significantly degrade visual perception performance. The application of VLMs in the surgical domain remains in its early stages. Several studies [ 12 , 15 ] have developed surgery-specific VLMs by constructing large-scale surgical knowledge datasets, demonstrating encouraging results in multimodal reasoning. Other works [ 13 , 6 ] have explored the use of general-purpose VLMs for surgical instrument classification, showing promising potential. However, to the best of our knowledge, there is no systematic study that evaluates how general-purpose VLMs perform on the fundamental visual perception task of surgical instrument detection, particularly under both zero-shot and fine-tuned settings. In this study, we present the comprehensive evaluation of three state-of-the-art large general-purpose vision-language models for surgical instrument detection. We investigate two practical deployment scenarios: (1) a zero-shot setting, in which off-the-shelf VLMs are directly prompted to classify and localize surgical instruments without any task-specific training, and (2) a fine-tuned setting, in which models are adapted to the surgical tool detection task using parameter-efficient LoRA fine-tuning. Through extensive experiments, we analyze the strengths and limitations of general-purpose VLMs for surgical visual perception and examine their potential applicability to future surgical AI systems. 2 METHODOLOGY 2.1 Problem Definition Let I\in\mathbb{R}^{H\times W\times 3} denote an input surgical image. The goal is to predict a set of surgical instrument instances \mathcal{O}=\{(c_{i},b_{i})\}_{i=1}^{N} , where c_{i} represents the instrument category, b_{i} denotes the corresponding bounding box, and N is the total number of instruments present in the image. We evaluate this task using three large VLMs, specifically Qwen2.5-7B [ 2 ] , LLaVA1.5-7B [ 9 ] , and InternVL3.5-8B [ 14 ] across both zero-shot and fine-tuned settings. 2.2 Dataset We used frames from the GraSP dataset [ 1 ] , extracted at 1 fps from 13 Robot-Assisted Radical Prostatectomy videos. Following the original dataset splits, we used frames extracted from five videos for testing, and the remaining videos for fine-tuning the models. The frames are annotated at 35 frames interval, yielding 1125 frames for testing and 2324 frames for fine-tuning. There are a total of 7 instrument categories, with each frame containing as few as 1 and as many as 5 instances. The distribution of instrument instances per instrument category is summarized in Table 1 . Table 1: Class-wise distribution of instrument instances in the training and test sets. Instrument Category Train Test Total Bipolar Forcep (BF) 1,694 809 2,503 Prograsp Forcep (PF) 741 330 1,071 Large Needle Driver (LND) 896 449 1,345 Monopolar Curved Scissor (MCS) 1,765 844 2,609 Suction Instrument (SI) 816 310 1,126 Clip Applier (CA) 64 38 102 Laparoscopic Grasper (LG) 194 81 275 All Instruments 6,170 2,861 9,031 2.3 Experimental Setup We experimented with three VLMs for the surgical tool detection task on the GraSP dataset. We imposed two settings—zero-shot and fine-tuned. For zero-shot inference, each model was prompted as-is, using its original pretrained weights to predict the instrument category and its corresponding bounding box on the test set. For the fine-tuned setting, we first adapted each model using supervised finetuning with Rank-8 LoRA adaptation [ 7 ] for 5 epochs, a batch size of 4 , a gradient accumulation step of 4 , and a learning rate of 1\times e-4 on the training set, and evaluated on the test set. The prompts used to query the model for fine-tuning and zero-shot inference are presented in Table 2 . The prompt for zero-shot inference includes a list of potential instrument categories to guide the model. However, after fine-tuning, the model has already learned the instrument categories directly from the training data, and therefore, no such guidance is required. All the VLM experiments were performed using the Swift framework 4 4 4 https://github.com/modelscope/ms-swift on an NVIDIA A100 (40GB) GPU [ 11 ] . As a baseline, Grounding DINO [ 10 ] was fine-tuned for 15 epochs with a batch size of 4 and a learning rate of 1\times e-4 using the Open-GroundingDino framework 5 5 5 https://github.com/longzw1997/Open-GroundingDino . During inference, a bounding box confidence threshold of 0.28 was applied to filter predicted bounding boxes. Table 2: Prompts used for zero-shot and fine-tuned VLM inference. Setting Prompt Zero-shot <image> From the following list of surgical instruments: