Eye image segmentation using visual and concept prompts with Segment Anything Model 3 (SAM3) DOI: XXXXXXX.XXXXXXX Conference: Make sure to enter the correct conference title from your rights confirmation email; June 03–05, 2018; Woodstock, NY ISBN: 978-1-4503-XXXX-X/2018/06 CCS: Computing methodologies Video segmentation CCS: Human-centered computing CCS: Computing methodologies Neural networks

Diederick C. Niehorster, Marcus Nyström · 2026

Read Eye image segmentation using visual and concept prompts with Segment Anything Model 3 (SAM3) DOI: XXXXXXX.XXXXXXX Conference: Make sure to enter the correct conference title from your rights confirmation email; June 03–05, 2018; Woodstock, NY ISBN: 978-1-4503-XXXX-X/2018/06 CCS: Computing methodologies Video segmentation CCS: Human-centered computing CCS: Computing methodologies Neural networks by Diederick C. Niehorster, Marcus Nyström (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2603-17715.txt.

Introduction

Niehorster and Nyström test SAM3 against SAM2 for eye image segmentation. Earlier work showed vision foundation models segment eye images well with no training. SAM3 adds a new concept prompting mode driven by text. Lab videos and the wild TEyeD set form the test bed. In most cases SAM3 matched or trailed SAM2, they find. SAM2 also ran faster. Their verdict keeps SAM2 as the best pick for this task.

Catalog record

CREATORDiederick C. Niehorster, Marcus Nyström
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE28400
LICENSEhttp://creativecommons.org/licenses/by-sa/4.0/
RIGHTSCC BY SA
SOURCEarXiv

Source: https://arxiv.org/abs/2603.17715

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Eye image segmentation using visual and concept prompts with Segment Anything Model 3 (SAM3) DOI: XXXXXXX.XXXXXXX Conference: Make sure to enter the correct conference title from your rights confirmation email; June 03–05, 2018; Woodstock, NY ISBN: 978-1-4503-XXXX-X/2018/06 CCS: Computing methodologies Video segmentation CCS: Human-centered computing CCS: Computing methodologies Neural networks Diederick C. Niehorster, Marcus Nyström Abstract . Previous work has reported that vision foundation models show promising zero-shot performance in eye image segmentation. Here we examine whether the latest iteration of the Segment Anything Model, SAM3, offers better eye image segmentation performance than SAM2, and explore the performance of its new concept (text) prompting mode. Eye image segmentation performance was evaluated using diverse datasets encompassing both high-resolution high-quality videos from a lab environment and the TEyeD dataset consisting of challenging eye videos acquired in the wild. Results show that in most cases SAM3 with either visual or concept prompts did not perform better than SAM2, for both lab and in-the-wild datasets. Since SAM2 not only performed better but was also faster, we conclude that SAM2 remains the best option for eye image segmentation. We provide our adaptation of SAM3’s codebase that allows processing videos of arbitrary duration. 1. Introduction There are several approaches to performing gaze estimation, that is, determining where a person looks based on images from their eyes ( 40 ) . Besides appearance-based methods that often rely on end-to-end neural networks ( 12 ) , most other approaches require features, such as the center or edges of the pupil, the iris and the corneal reflection (CR) of the eye tracker’s illuminator, to be detected and localized in the eye images. Such features are then used for regression-based gaze estimation techniques such as P-CR ( 43 ; 55 ; 35 ; 9 ; 3 , e.g.,) or techniques involving geometric eye models [ 24 ; 13 ; 2 ; 57 ; 15 ; 52 ; see 25 for an overview]. How are these features detected in eye images and their centers localized? While traditional image processing approaches to detecting these features ( 48 ; 56 ; 51 ; 19 ; 20 , e.g.,) remain popular, deep learning approaches are also being developed [ 22 ; 17 ; 23 ; 36 ; 14 ; 12 ; 33 ; see 1 for an overview]. One recurring problem with deep learning approaches is that if they do not work out of the box, retraining requires large datasets that are expensive to acquire and manually annotate ( 7 ; 6 , e.g.,) . Vision foundation models, such as the Segment Anything Model family of models ( 34 ; 49 ; 8 ) , offer a potential solution to eye image segmentation that may drastically reduce the amount of manual work required. Specifically, both SAM ( 42 ; 14 ) and SAM 2 ( 41 ; 47 ) have been shown to offer segmentation performance that is competitive with the state-of-the-art for eye images in a range of different settings. The eye image segmentation performance of the latest model, SAM 3 ( 8 ) has however not yet been examined. In this work, we examine how the recently released SAM 3 performs compared to SAM 2 for the task of eye image segmentation. Specifically, first we ask whether the visual prompting (manually indicating an object in the image to be segmented) performance of SAM 3 is superior to that of SAM 2. While SAM 3 has the same model architecture as SAM 2 for visual prompting tasks, the HieraDet image encoder ( 50 ; 5 ) of SAM 2 was replaced with the Perception Encoder ( 4 ) , which may lead to altered performance. Second, we ask how SAM 3’s new concept prompting abilities (i.e., prompting the model using a short string such as “pupil”) compares to visual prompting in SAM 2 and 3. Concept prompting would remove all manual work from using SAM (no prompts have to be placed manually on the various eye features), but it also offers potential benefits for, for instance, blink recovery. To provide insight into how SAM 2 and SAM 3 perform in eye image segmentation across a wide range of eye tracking domains, we perform our evaluation using both datasets containing high-resolution and high-quality eye images obtained in controlled lab settings, and datasets of eye images obtained from head-worn (glasses, VR and AR) eye trackers obtained in unconstrained settings. Across datasets, we prompt the models to segment the pupil, iris and sclera, and for the high-resolution lab datasets we additionally segment the CR. For the lab datasets, since they lack a ground truth, we examine the RMS-S2S ( 46 ; 45 ) precision of the resulting feature signals, thereby probing which method yields signals with the lowest noise levels, and data loss. For the in-the-wild dataset, TEyeD ( 18 ) is used, which consists of various datasets ( 58 ; 37 ; 33 ; 19 ; 20 ; 21 ; 32 ) and ground truth labels for the pupil, iris and eyelid aperture. As such, for these datasets, model performance is assessed as the overlap of model output with the ground truth segmentation, using the intersection-over-union (IoU) metric. Finally, as part of this work, we have adapted the SAM 3 code to be able to run on videos of arbitrary length. This code is available here: https://github.com/dcnieho/sam3 . 2. Methods 2.1. Datasets 2.1.1. High-resolution lab datasets Two different datasets were used, both recorded using the FLEX setup ( 27 ; 48 ; 7 ; 6 ; 30 ; 59 ) . The first dataset was recorded from four experienced participants (all male) at 1000 Hz and consisted of 106 trials during which participants performed various short and long fixation and saccade sequences (this dataset has been used previously, see 6 ; 47 , for further description) . The second dataset consisted of recordings taken from 17 participants (five females, 11 males, one non-binary). Each dataset contained forty trials consisting of fixation and saccade sequences recorded at 1000 Hz (the dataset was previously used, see 7 , for further details) . The datasets contain 2.87 million images. 2.1.2. Unconstrained datasets TEyeD ( 18 ) consists of NVGaze ( 33 ) , two sets of eye images collected with a VR (14 participants) and an AR (42 participants) device; Gaze-in-wild ( 37 ) , 19 participants performing everyday tasks with a wearable eye tracker; Labelled pupils in the wild ( 58 ) , 22 participants performing everyday tasks with a wearable eye tracker; and a series of Dikablis datasets, which are a combination of the datasets from 19 , 20 , 21 and 32 . TEyeD provides ground truth pupil and iris ellipses as well as eyelid polygons for all datasets. The datasets contain 14.44 million images. 2.2. Models and prompting For SAM 2 ( 49 ) , the largest model ( sam2.1_hiera_large ) was used since it performs best for eye image segmentation ( 47 ) . For SAM 3 ( 8 ) , only one model size is available ( sam3.pt from 19 Nov 2025). Below, we describe how prompting was performed, Figure 1 shows example prompts and segmentation masks. Inference was run on three

Read the complete text