Zero-Shot Gaze-based Volumetric Medical Image Segmentation

Tatyana Shmykova, Leila Khaertdinova, Ilya Pershin · 2025

Read Zero-Shot Gaze-based Volumetric Medical Image Segmentation by Tatyana Shmykova, Leila Khaertdinova, Ilya Pershin (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2505-15256.txt.

Introduction

Tatyana Shmykova, Leila Khaertdinova, and Ilya Pershin let doctors segment scans with their eyes in a 2025 paper. Outlining anatomy in three-dimensional medical images usually needs clicks and boxes drawn by hand. That manual prompting slows clinical work. The team instead feeds eye-gaze points into two interactive segmentation models, testing both synthetic gaze and real tracked gaze from users.

Gaze prompts worked faster than bounding boxes, with slightly lower outline quality. The trade-off favors gaze as a helping hand rather than a full replacement for careful manual work. Readers get a first proof that eye tracking can steer three-dimensional medical segmentation in practice.

Catalog record

CREATORTatyana Shmykova, Leila Khaertdinova, Ilya Pershin
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE16 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2505-15256

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Zero-Shot Gaze-based Volumetric Medical Image Segmentation Tatyana Shmykova, Leila Khaertdinova, Ilya Pershin Abstract Accurate segmentation of anatomical structures in volumetric medical images is crucial for clinical applications, including disease monitoring and cancer treatment planning. Contemporary interactive segmentation models, such as Segment Anything Model 2 (SAM-2) and its medical variant (MedSAM-2), rely on manually provided prompts like bounding boxes and mouse clicks. In this study, we introduce eye gaze as a novel informational modality for interactive segmentation, marking the application of eye-tracking for 3D medical image segmentation. We evaluate the performance of using gaze-based prompts with SAM-2 and MedSAM-2 using both synthetic and real gaze data. Compared to bounding boxes, gaze-based prompts offer a time-efficient interaction approach with slightly lower segmentation quality. Our findings highlight the potential of using gaze as a complementary input modality for interactive 3D medical image segmentation. 1 Introduction Accurate anatomical segmentation is vital for cancer diagnosis, treatment planning, and disease monitoring. Although automated segmentation methods have shown promise, they often struggle with the variability of medical images, necessitating interactive approaches that incorporate user input for enhanced accuracy and adaptability. Recent advancements in interactive segmentation have demonstrated the potential to combine automated algorithms with different forms of user guidance. For instance, Segment Anything Model (SAM) kirillov2023segment and its medical variant, MedSAM ma2024segment , leverage user prompts such as bounding boxes, clicks, and masks to refine segmentation results, enhancing both precision and efficiency. Various prompting techniques have been explored for the medical image segmentation task, including bounding boxes (MedSAM medsam and MedSAM-2 medsam2 ), text prompts (MedCLIP-SAMv2 medclip ), and gaze-based inputs (GazeSAM gazesam and gazeassisted ; ieeegaze ). Existing approaches often rely on fine-tuning to adapt to new prompting modalities, such as gaze inputs. However, these methods require additional labeled data, which may be scarce for narrow-domain tasks. Foundation models, such as MedSAM medsam and MedSAM-2 medsam2 , take a medical image as input along with a specifies region of interest. Unlikely, multimodal models, including MedCLIP-SAMv2 medclip integrate information from multiple modalities (images and text), allowing them to interpret visual data within the context of textual descriptions. Despite the MedCLIP-SAMv2 claimed performance in zero-shot settings medclip , we observed that text prompts perform poorly when segmenting abdominal organs in the WORD dataset luo2022word . This highlights the need for an alternative zero-shot approach that enables MedSAM to be used effectively on specific medical segmentation tasks without additional fine-tuning. Our proposed method addresses this limitation by providing an efficient and accurate segmentation solution tailored to narrow-domain applications. One of the major challenges in medical imaging is the transition from segmentation of 2D medical images to 3D images, as annotation of volumetric scans performed per slice can be extremely time-consuming. Recent models, such as SAM-2 sam2 and MedSAM-2 medsam2 , have introduced memory mechanisms to facilitate interactive 3D segmentation by incorporating prior contextual information. However, traditional input methods like manual clicks and bounding boxes remain discrete and require manual effort, which can be inefficient for large-scale volumetric data. Figure 1 : The process of generating segmentation masks based on the annotator’s gaze stream involves the following steps: (1) tracking of eye gaze, (2) gaze heatmaps generation, (3) heatmaps post-processing, (4) bounding box-based prompt construction, and (5) segmentation using a pre-trained interactive segmentation model, e.g., MedSAM-2, with frozen parameters. In this study, we propose gaze as a novel informational modality for interactive 3D medical image segmentation. Unlike conventional user prompts, gaze is a continuous and passive input that does not require explicit manual intervention, making it a more natural and intuitive interaction method. While gaze has been previously explored as a prompting technique for segmentation of 2D images, its potential for 3D segmentation remains largely unexplored. We bridge this gap by integrating gaze-based inputs into interactive segmentation models and comparing their performance and efficiency with traditional prompting techniques. To assess the feasibility and effectiveness of our approach, we conduct experiments using a workstation equipped with a Tobii Eye Tracker 4C, allowing an expert radiologist to interactively segment medical images using gaze-based prompts. Our results demonstrate that gaze-based segmentation offers a faster and more seamless alternative to bounding boxes-based method with minimal loss in accuracy. This study presents eye gaze as an innovative modality for interactive 3D medical image segmentation, paving the way for more intuitive and efficient approach for human-AI collaboration in the medicine domain. 2 Related work The zero-shot segmentation method in MedCLIP-SAMv2 integrates multimodal learning to achieve precise medical image segmentation without requiring manual annotations. The approach begins with BiomedCLIP, a specialized vision-language model, which extracts both image and text embeddings. These embeddings are refined through the Multi-modal Information Bottleneck (M2IB) mechanism, which enhances relevant visual information while suppressing noise. The resulting saliency maps undergo a postprocessing step with K-Means clustering to generate an initial segmentation mask. This mask serves as the basis for creating bounding box or point-based prompts for the Segment Anything Model (SAM), which further refines the segmentation output. The authors conduct experiments to assess the impact of text prompt design on zero-shot segmentation. They compare six different types of text prompts: P0 and P1 use class names, P2 and P3 feature descriptive prompts, and P4 and P5 employ ensembles of 20 prompts. P0, P2, and P4 are generic, whereas P1, P3, and P5 specify object subtypes. For example, in Breast Ultrasound, P0 is ”breast tumor,” while P1 specifies ”malignant” or ”benign” tumors. P2 provides a single descriptive sentence, whereas P3 details tumor subtypes. P4 and P5 follow the same structure but use ensemble embeddings. GPT-4 generates all descriptive prompts. The authors evaluate their approach on various datasets, including Lung CT, where the lowest Dice score is 69.89 when using P0 prompts, which contain only the class name. With P2 prompts, the performance improves to 80.38. To assess the performance of the zero-shot segmentation algorithm MedCLIP-SAMv2, we use two types of text prompts: P0, which use the class name of the organ, and P2, which consists of descriptive sentence prompts in the format: ”A medical CT scan showing the class name of the organ . Although the authors of MedCLIP-SAMv2 claim that their method demonstrates strong zero-shot capabilities, its performance on the WORD dataset is notably poor. Our evaluation is conducted on 600 slices of the WORD dataset, employing MedCLIP-SAMv2 approach. While the model achieves a Dice score of 0.88 when using bounding boxes derived from ground truth mask boundaries, its performance drops significantly when using P0 and P2 text prompts, yielding scores of 0.16 and 0.2, respectively. These results indicate that the proposed approach is not effective across all datasets. 3 Methods 3D interactive medical image segmentation focuses on accurately delineating organs or pathological structures across all slices of a volumetric scan. The input is a 3D medical image represented as a stack of slices, and the output consists of segmentation masks for each slice. Unlike fully automated segmentation, interactive segmentation incorporates user input, such as bounding boxes, clicks, or other prompts to refine and guide the process. These prompts may be applied to all slices or only a subset. The challenge lies in ensuring accurate, consistent, and efficient segmentation with minimal user interaction while adapting to variations in anatomy and pathology. Our method

Read the complete text