GroundedSurg: A Multi-Procedure Benchmark for Language-Conditioned Surgical Tool Segmentation

Tajamul Ashraf †, Abrar Ul Riyaz*, Wasif Tak* ‡, Tavaheed Tariq*, Sonia Yadav, Moloud Abdar, Janibul Bashir · 2026

Read GroundedSurg: A Multi-Procedure Benchmark for Language-Conditioned Surgical Tool Segmentation by Tajamul Ashraf †, Abrar Ul Riyaz*, Wasif Tak* ‡, Tavaheed Tariq*, Sonia Yadav, Moloud Abdar, Janibul Bashir (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2603-01108.txt.

Introduction

Ashraf and six colleagues introduce GroundedSurg, the first benchmark that asks models to find one specific surgical tool from a spoken-style description, not just every tool of a class. Each item pairs an image with language targeting a single instrument, plus box and point grounding labels, spanning eye, laparoscopic, robotic, and open surgery. Tests reveal large gaps in modern segmentation and vision-language models, underscoring the need for clinically grounded reasoning in surgical AI.

Catalog record

CREATORTajamul Ashraf †, Abrar Ul Riyaz*, Wasif Tak* ‡, Tavaheed Tariq*, Sonia Yadav, Moloud Abdar, Janibul Bashir
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE23119
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2603.01108

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

GroundedSurg: A Multi-Procedure Benchmark for Language-Conditioned Surgical Tool Segmentation Tajamul Ashraf †, Abrar Ul Riyaz*, Wasif Tak* ‡, Tavaheed Tariq*, Sonia Yadav, Moloud Abdar, Janibul Bashir Abstract Clinically reliable perception of surgical scenes is essential for advancing intelligent, context-aware intraoperative assistance such as instrument handoff guidance, collision avoidance, and workflow-aware robotic support. Existing surgical tool benchmarks primarily evaluate category-level segmentation, requiring models to detect all instances of predefined instrument classes. However, real-world clinical decisions often requires resolving references to a specific instrument instance based on its functional role, spatial relation, or anatomical interaction capabilities not captured by current evaluation paradigms. We introduce GroundedSurg , the first language conditioned, instance-level surgical grounding benchmark. Each instance pairs a surgical image with a natural-language description targeting a single instrument, accompanied by structured spatial grounding annotations including bounding boxes and point-level anchors. The dataset spans ophthalmic, laparoscopic, robotic, and open procedures, encompassing diverse instrument types, imaging conditions, and operative complexities. By jointly evaluating linguistic reference resolution and pixel-level localization, GroundedSurg enables a systematic and realistic evaluation of vision–language models in clinically realistic multi-instrument scenes. Extensive experiments demonstrate substantial performance gaps across modern segmentation and VLMs, highlighting the urgent need for clinically grounded vision–language reasoning in surgical AI systems. Code and data is publically available at https://github.com/gaash-lab/GroundedSurg † † footnotetext: Corresponding author: [email protected] * * footnotetext: Equal Contribution ‡ ‡ footnotetext: Work done while an intern at Gaash Lab, NIT Srinagar GroundedSurg: A Multi-Procedure Benchmark for Language-Conditioned Surgical Tool Localization 1 Introduction Accurate interpretation of surgical scenes has traditionally been framed as the segmentation and recognition of predefined instrument categories [ 1 , 2 , 15 ] . These category-level formulations have enabled important downstream applications, including workflow analysis, skill assessment, and context-aware assistance systems [ 23 , 29 , 28 ] . However, surgical scene understanding extends beyond class identification. In real operative environments, multiple visually similar instruments often coexist within the same field of view. Their clinical relevance depends not only on category, but also on functional role, spatial configuration, and interaction with surrounding anatomy [ 6 , 26 ] . Distinguishing the instrument actively dissecting tissue from another instrument of the same type that is idle or retracting therefore requires resolving a context-dependent reference to a specific instance. Current evaluation paradigms do not assess this capability [ 6 , 8 , 23 ] . Category-level tool segmentation benchmarks measure recognition performance but do not require resolving which specific instance satisfies a procedural description when multiple candidates coexist. As summarized in Table 1 , existing surgical tool datasets primarily focus on class-level segmentation and lack integrated support for language conditioning, structured spatial grounding, and instance-level disambiguation across diverse procedures. Conversely, general vision–language grounding benchmarks (e.g., RefCOCO) [ 33 , 18 , 17 ] do not reflect the visual complexity, occlusions, fine-grained instrument morphology, and domain-specific constraints inherent to surgical environments. As a result, grounded, instance-level reasoning in operative scenes remains largely untested. Bridging this gap requires a benchmark that explicitly couples natural-language reference resolution with precise pixel-level instrument localization. Such evaluation must occur under realistic intraoperative conditions, as illustrated in Fig. 1 . Such a formulation must support multi-instrument scenes with ambiguity, structured spatial grounding annotations to enforce localization consistency, and evaluation protocols that quantify both grounding accuracy and segmentation precision. Table 1: Comparison of surgical tool datasets with respect to grounding capabilities. Our benchmark supports multi-procedure, language conditioning, explicit spatial grounding, instance-level disambiguation, and prompt-based evaluation. Dataset Domain Multi-Proc. Language Spatial Disambig. Prompt CaDIS [ 8 ] Cataract ✗ ✗ ✗ ✗ ✗ AutoLaparo [ 30 ] Laparoscopic ✗ ✗ ✗ ✗ ✗ SISVE [ 32 ] Gastrectomy ✗ ✗ ✓ ✓ ✗ EndoVis [ 26 ] Robotic ✗ ✗ ✗ ✗ ✗ CholecSeg8k [ 10 ] Cholecystectomy ✗ ✗ ✗ ✗ ✗ CholecInstanceSeg [ 1 ] Cholecystectomy ✗ ✗ ✗ ✓ ✗ Robust-MIS [ 22 ] MIS ✗ ✗ ✗ ✗ ✗ InSeg1 [ 6 ] Ophthalmic ✗ ✗ ✓ ✓ ✗ InSeg2 [ 6 ] Ophthalmic ✗ ✗ ✓ ✓ ✗ GroundedSurg (Ours) Multi-procedure ✓ ✓ ✓ ✓ ✓ Figure 1: Overview of GroundedSurg . (a) Existing datasets focus on category-level segmentation without language conditioning or instance-level grounding. (b) GroundedSurg introduces natural-language queries with structured spatial annotations for query-conditioned instrument localization. (c) Baseline results reveal substantial performance gaps, highlighting the challenges of grounded surgical perception. To address this need, we introduce GroundedSurg , a grounding-based surgical tool segmentation benchmark consisting of 612 surgical im- ages and 1,071 tool-level annotations, that reformulates surgical tool perception as a language-conditioned, instance-level segmentation task. Given a surgical image and a natural-language query describing a specific instrument through its functional role, spatial relation, or anatomical interaction, the objective is to localize and segment the instrument instance satisfying the description. This formulation departs from conventional category-level segmentation by requiring explicit disambiguation among visually similar instruments and grounding of contextual references to a single spatially localized instance. GroundedSurg incorporates structured spatial annotations, including bounding boxes and center points, to quantify localization accuracy at multiple levels of granularity. Bounding boxes enable coarse instance verification, while pixel-level masks assess fine-grained delineation under realistic surgical challenges such as occlusion, specular reflections, motion blur, and instrument overlap. Each image–query pair is treated as an independent evaluation unit, enabling controlled instance-level metric computation in multi-instrument scenes. By unifying natural-language reference, structured spatial grounding, and multi-procedure diversity within a standardized evaluation framework, GroundedSurg establishes a principled benchmark for grounded surgical perception, thereby enabling the next generation of grounding-aware surgical AI systems. Contributions. (1) We reconceptualize surgical tool perception as a grounded vision–language task requiring resolution of context-dependent references to specific instrument instances. (2) We introduce GroundedSurg, a surgical benchmark that systematically couples natural-language descriptions with explicit spatial grounding annotations, including bounding boxes, center points, and pixel-level masks to enable rigorous evaluation of language-conditioned, instance-level localization and segmentation. (3) We curate a diverse, multi-procedure dataset spanning heterogeneous surgical domains and imaging conditions, providing a clinically realistic and reproducible testbed for grounding-aware intraoperative vision systems. 2 GroundedSurg Problem Formulation: GroundedSurg formalizes surgical tool perception as a language-conditioned, instance-level segmentation task. Each benchmark instance consists of a surgical image paired with a natural-language description referring to a single target instrument, along with structured spatial grounding annotations and a pixel-level segmentation mask. Formally, let I\in\mathbb{R}^{H\times W\times 3} denote a surgical image of height H and width W . For each image, a single target instrument instance is defined by: (i) a natural-language query T describing the instrument through its procedural role, spatial relation, or interaction, (ii) a bounding box B=(x_{\min},y_{\min},x_{\max},y_{\max}) , (iii) a center point C=(x_{c},y_{c}) , and (iv) a binary segmentation mask M\in\{0,1\}^{H\times W} . The objective is to predict a segmentation mask \hat{M} corresponding to the instrument described by T , i.e., to learn a mapping f(I,T,B,C)\rightarrow\hat{M} . Each image–query pair corresponds to exactly one instrument instance. In scenes containing multiple tools, separate instances are constructed for each reference to ensure unambiguous supervision and controlled instance-level evaluation. The spatial annotations (B,C) act as auxiliary grounding cues to reduce ambiguity and strengthen alignment between linguistic reference and visual localization. Figure 2: GroundedSurg benchmark pipeline. Surgical images are paired with initial prompts and processed by a vision–language

Read the complete text