Eye-Tracking, Mouse Tracking, Stimulus Tracking, and Decision-Making Datasets in Digital Pathology
Veronica Thai, Rui Li, Meng Ling, Shuning Jiang, Jeremy Wolfe, Raghu Machiraju, Yan Hu, Zaibo Li, Anil Parwani, Jian Chen · 2025
Read Eye-Tracking, Mouse Tracking, Stimulus Tracking, and Decision-Making Datasets in Digital Pathology by Veronica Thai, Rui Li, Meng Ling, Shuning Jiang, Jeremy Wolfe, Raghu Machiraju, Yan Hu, Zaibo Li, Anil Parwani, Jian Chen (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2510-24653.txt.
Introduction
Veronica Thai and nine co-authors share a rich data set on how eye doctors read huge slide shots of tissue. Reading these shots is slow and hard, and right calls average about seventy in a hundred. A second reader does not fix the lack of match by much, so the field needs proof of where reads go wrong.
Their set, PathoGaze1.0, holds close to nineteen hours of eye, mouse, and screen track data from nineteen readers. In all it logs more than one hundred seventy thousand gaze stops and far more mouse moves. The data may help train both new doctors and AI aids.
Catalog record
| CREATOR | Veronica Thai, Rui Li, Meng Ling, Shuning Jiang, Jeremy Wolfe, Raghu Machiraju, Yan Hu, Zaibo Li, Anil Parwani, Jian Chen |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 50 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2510-24653
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Eye-Tracking, Mouse Tracking, Stimulus Tracking, and Decision-Making Datasets in Digital Pathology Veronica Thai, Rui Li, Meng Ling, Shuning Jiang, Jeremy Wolfe, Raghu Machiraju, Yan Hu, Zaibo Li, Anil Parwani, Jian Chen Abstract Interpretation of giga-pixel whole-slide images (WSIs) is an important but difficult task for pathologists. Their diagnostic accuracy is estimated to average around 70 % 70\% . Adding a second pathologist does not substantially improve decision consistency. The field lacks adequate behavioral data to explain diagnostic errors and inconsistencies. To fill in this gap, we present PathoGaze1.0, a comprehensive behavioral dataset capturing the dynamic visual search and decision-making processes of the full diagnostic workflow during cancer diagnosis. The dataset comprises 18.69 hours of eye-tracking, mouse interaction, stimulus tracking, viewport navigation, and diagnostic decision data (EMSVD) collected from 19 pathologists interpreting 397 WSIs. The data collection process emphasizes ecological validity through an application-grounded testbed, called PTAH. In total, we recorded 171,909 fixations, 263,320 saccades, and 1,867,362 mouse interaction events. In addition, such data could also be used to improve the training of both pathologists and AI systems that might support human experts. All experiments were preregistered at https://osf.io/hj9a7 , and the complete dataset along with analysis code is available at https://go.osu.edu/pathogaze . Background & Summary Visual search expertise in digital pathology depends on how pathologists allocate attention across complex tissue landscapes [ 1 , 2 , 3 ] . Collecting gaze data can provide insight into the causes of diagnostic inconsistency [ 4 ] , support decision-making [ 5 ] , model visual search errors [ 6 ] , and enable the development of machine learning methods to assist pathologists in daily diagnostic tasks [ 7 ] , or to create a digital twin of a pathologist’s diagnostic process [ 8 ] . However, multimodal behavior analyses remain scarce in digital pathology. Existing datasets such as CAMELYON16 [ 9 ] , were designed for understanding neural network behaviors, but have not been analyzed together with pathologists’ visual search. Here, we captured comprehensive behavioral datasets that integrate gaze and diagnostic actions that can be used to explain human pathologists’ visual and cognitive processes. Collecting gaze data can also contribute to the community’s efforts to compare, replicate, and extend findings on observer behaviors. Pathologists’ gaze emphasizes task-specific attentional behaviors, thus both bottom-up and top-down processes are involved [ 10 ] , where participants use both the stimuli and their domain knowledge to allocate regions that stand out. Together, this large data collection enables analyses of visual search behaviors [ 10 ] such as “Looked But Failed to See” errors [ 11 ] , failures to recognize [ 2 ] , or being “satisfied” too early [ 12 ] , behaviors that have also been reported in non-pathology medical imaging domains. Furthermore, our data can be used to validate whether mouse-action and eye-gaze are correlated as in other imaging modalities [ 13 ] , or help identify the interaction between image features, actions, individual differences, and tasks. The primary contribution of this paper is a collection of eye-gaze behaviors, mouse tracking, image stimuli, and associated decisions ( Figure 1 ). Our data collection uses application-grounded evaluation, where data collection is conducted within a replication of application tools: our testbed, called PTAH, has assembled features from the state-of-the-art clinical diagnosis platforms. We recruited 19 pathologists across two experiments (10 in Experiment I and 9 in Experiment II), designed to study complementary aspects of diagnostic reasoning. Experiment I (P10S60T600) captured the behavior of 10 pathologists viewing the same set of 60 unique WSIs (15 benign, 45 with metastases) and recorded their annotated region that led to their final decisions. As a result, we can measure behavior consistency between pathologists. In contrast, Experiment II, P9D397T540, asked participating pathologists to look at as many WSIs as possible for broad behavior monitoring. Nine pathologists read in 540 trials over 9.14 hours, which led to 85,605 fixations on 397 CAMELYON16 WSIs. Figure 1 : Examples of gaze behaviors from pathologists in PathoGaze1.0. Each whole-slide image (WSI) contains approximately one billion pixels (gigapixel scale). The axis tick labels are scaled down by a factor of 32\times 32 relative to the original image size. Dots represent fixation points, with color and size indicating fixation duration, while connecting lines denote saccades (scan paths). Both fixations and saccades are overlaid on the corresponding original WSIs. Observations. The behavior data captured distinct viewing strategies. For example, for cases with large tumor regions (top row), Participant P5 made a diagnostic decision without scanning all tumor areas, whereas Participant P10 examined most regions before responding. In challenging small-tumor cases, one participant correctly identified a tumor in one slide but misclassified another. Participant P8 exhibited a search error by failing to fixate on the tumor regions (highlighted in red). In general, we observe mouse movement (blue lines and arrows) aligns with gaze points, such as the examples from Participant P10. However, there are a few cases, such as Participant P5, where the mouse does not follow the gaze, in this case, spanning a wider area than the gaze. Methods Experimental Design We recorded perception-centered gaze behaviors, action-centered navigation behaviors, and subsequent diagnostic decisions in both experiments. In Experiment I, participants also marked tumor regions. The perception-centered data captured gaze behaviors that informed search, selection, filtering, and decision processes, while the action-centered data tracked pathologists’ interactive behaviors such as mouse activities reflecting zooming, panning, dragging, and changes in viewport position. Data Collection Goals and Generic Data Choices Figure 2 : Viewport and screen coordinates. All data are calibrated in the WSI image coordinates, where the upper-left corner of the WSI is (0, 0) and bottom right corner is largest ( x_{image} , y_{image} ) of the WSI in its full pixel resolution. A viewport is the rectangular area on the screen where the WSI is actually rendered and displayed. It excludes surrounding user interface elements. Each viewport was recorded as its corresponding position on the full WSI. Specifically, we stored the pixel coordinates of the viewport’s upper-left corner in the WSI coordinate system, as illustrated in the navigator overview . Fixations were captured in the screen coordinates and were subsequently transformed to the WSI image coordinates. We chose to use the CAMELYON16 competition data [ 9 ] , a standard benchmark for digital pathology of breast cancer lymph node samples taken from patients at two medical centers in the Netherlands (University Medical Center Utrecht and Radboud University Medical Center). We used it for several reasons: it was carefully curated and contained cases of varying sizes and difficulty levels. Specifically, among the 399 cases, 239 were benign and 160 were cancer metastases with ground-truth locations. Moreover, the dataset includes multiple zoom levels and tumor sizes, allowing us to compare search behaviors across stimuli of different scales. These WSIs were scanned at either 20 \times magnification for a pixel size of 0.243 \mu m \times 0.243 \mu m, or 40 \times for a pixel size of 0.226 \mu m \times 0.226 \mu m. To label tumor sizes, we followed clinical practice to categorize slides based on tumor diameter and identified eight isolated (tumor regions < 0.2mm in diameter), 93 micrometastases (tumor regions \in [0.2, 2) mm), and 59 macrometastases (tumor