MedSPOT: A Workflow-Aware Sequential Grounding Benchmark for Clinical GUI
Rozain Shakeel, Abdul Rahman Mohammad Ali, Muneeb Mushtaq, Tausifa Jan Saleem, Tajamul Ashraf · 2026
Read MedSPOT: A Workflow-Aware Sequential Grounding Benchmark for Clinical GUI by Rozain Shakeel, Abdul Rahman Mohammad Ali, Muneeb Mushtaq, Tausifa Jan Saleem, Tajamul Ashraf (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2603-19993.txt.
Introduction
Shakeel and co-authors build MedSPOT, a benchmark for grounding language in clinical software screens. Medical tasks unfold as sequences of steps across changing interfaces. Older benchmarks test single, isolated grounding queries instead. MedSPOT uses 216 task videos with 597 annotated keyframes. Each task has two or three linked grounding steps in real medical workflows. Scoring is strict: a task stops at the first wrong prediction. This design exposes how errors spread through multi-step work. It tests procedural skill, not just single clicks.
Catalog record
| CREATOR | Rozain Shakeel, Abdul Rahman Mohammad Ali, Muneeb Mushtaq, Tausifa Jan Saleem, Tajamul Ashraf |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 84721 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2603.19993
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
MedSPOT: A Workflow-Aware Sequential Grounding Benchmark for Clinical GUI Rozain Shakeel, Abdul Rahman Mohammad Ali, Muneeb Mushtaq, Tausifa Jan Saleem, Tajamul Ashraf Abstract Despite the rapid progress of Multimodal Large Language Models ( MLLMs ), their ability to perform reliable visual grounding in high-stakes clinical software environments remains underexplored. Existing GUI benchmarks largely focus on isolated, single-step grounding queries, overlooking the sequential, workflow-driven reasoning required in real-world medical interfaces, where tasks evolve across interdependent steps and dynamic interface states. We introduce MedSPOT , a workflow-aware sequential grounding benchmark for clinical GUI environments. Unlike prior benchmarks that treat grounding as a standalone prediction task, MedSPOT models procedural interaction as a sequence of structured spatial decisions. The benchmark comprises 216 task-driven videos with 597 annotated keyframes, in which each task comprises 2–3 interdependent grounding steps within realistic medical workflows. This design captures interface hierarchies, contextual dependencies, and fine-grained spatial precision under evolving conditions. To evaluate procedural robustness, we propose a strict sequential evaluation protocol that terminates task assessment upon the first incorrect grounding prediction, explicitly measuring error propagation in multi-step workflows. We further introduce a comprehensive failure taxonomy, including edge bias, small-target errors, no prediction, near miss, far miss, and toolbar confusion, to enable systematic diagnosis of model behavior in clinical GUI settings. By shifting evaluation from isolated grounding to workflow-aware sequential reasoning, MedSPOT establishes a realistic and safety-critical benchmark for assessing multimodal models in medical software environments. Code and data are available at https://github.com/Tajamul21/MedSPOT Project Page: https://rozainmalik.github.io/MedSPOT_web/ 1 Introduction Table 1 : Comparison of MedSPOT against existing GUI grounding benchmarks. Green cells indicate the presence of a capability, red cells indicate absence. MedSPOT is the only benchmark that jointly evaluates workflow-aware sequential grounding, early-termination protocol, structured failure taxonomy, and medical software environments. Benchmark Size Multi-Step Step Eval. Sequential Failure Medical Tasks Protocol Taxonomy Software ScreenSpot [ 7 ] 1,272 × × × × × ScreenSpot-Pro [ 20 ] 1,581 × ✓ × × × OSWorld-G [ 48 ] 564 ✓ × × × × OmniACT [ 18 ] 9,802 ✓ ✓ × × × AssistGUI [ 11 ] 100 ✓ ✓ × × × Mind2Web [ 8 ] 2,350 ✓ × × × × MedSPOT (Ours) 216 / 597 ✓ ✓ ✓ ✓ ✓ (a) Medical GUI interface (b) General GUI interface Figure 1 : Comparison of interface complexity. (a) Medical GUI environments are densely structured, hierarchically organized, and semantically rich with domain-specific terminology. (b) General GUI environments typically involve simpler layouts and short-horizon navigation tasks. Graphical User Interfaces ( GUIs ) are the primary medium through which users interact with modern software systems. GUI grounding refers to the task of mapping natural language instructions to precise visual and interactive elements within an interface [ 7 , 45 ] . Recent advances in Vision-Language Models ( VLMs ) and Multimodal Large Language Models ( MLLMs ) [ 3 , 35 , 23 , 38 , 39 ] have significantly improved multimodal reasoning capabilities, enabling systems to interpret screenshots, detect interface elements, reason over contextual cues, and predict executable actions. Despite this progress, existing research predominantly focuses on general-purpose environments such as web navigation and desktop automation. Benchmarks such as ScreenSpot [ 7 ] , Mind2Web [ 8 ] , OmniAct [ 18 ] , and related works [ 20 , 54 , 12 , 2 ] evaluate grounding in short-horizon, single-step scenarios. These settings treat grounding as an isolated prediction problem, overlooking the structured, sequential reasoning required in real-world workflows. This limitation becomes critical in high-stakes domains such as clinical software systems. Modern medical workflows rely heavily on complex graphical interfaces, including diagnostic imaging viewers, treatment planning systems, and health record dashboards. As illustrated in Fig. 1 , medical GUIs are densely structured, hierarchically organized, and semantically rich with domain-specific terminology. Within such environments, accurate grounding requires not only visual alignment but also contextual awareness of interface hierarchies, multi-step procedural reasoning, and strict precision. An early grounding error can invalidate subsequent actions, potentially leading to severe operational consequences. Yet, despite the rapid growth of both GUI grounding and medical AI research, there exists no benchmark specifically designed to evaluate workflow-aware sequential grounding in clinical software environments. Existing GUI datasets focus on general-purpose settings, while medical benchmarks [ 51 , 5 , 15 ] primarily evaluate static image reasoning rather than interactive procedural workflows. As a result, the intersection of sequential grounding and safety-critical medical interfaces remains largely unexplored. As summarized in Table 1 , existing GUI grounding benchmarks primarily focus on general-purpose environments and short-horizon tasks. While several datasets support multi-step interactions, none incorporate a strict sequential evaluation protocol, structured failure analysis, or domain-specific clinical software settings. In contrast, MedSPOT is the first benchmark to jointly evaluate workflow-aware grounding under early-termination scoring in safety-critical medical interfaces. To address this gap, we introduce MedSPOT , a benchmark for evaluating workflow-aware spatial grounding in clinical GUI environments. MedSPOT spans 10 functionally diverse medical imaging software platforms and consists of 216 task-driven videos with 597 annotated keyframes , as illustrated in Fig. 2 . Each task involves multiple interdependent grounding steps that reflect realistic clinical procedures, where decisions evolve across dynamic interface states. Beyond dataset construction, we propose a failure-aware sequential evaluation protocol that terminates task scoring upon the first incorrect grounding prediction. This strict early-termination strategy explicitly measures error propagation across multi-step workflows, providing a more realistic assessment of procedural robustness. Furthermore, we introduce a structured six-class failure taxonomy, including edge bias, small-target errors, no prediction, near miss, far miss, and toolbar confusion, to systematically diagnose grounding behavior in complex medical interfaces. More details regarding MedSPOT are given in Supplementary Material A . Our contributions are summarized as follows: • We introduce MedSPOT , the first benchmark for workflow-aware sequential grounding in clinical GUI environments, spanning 10 medical software platforms with 216 tasks and 597 annotated keyframes. • We propose a failure-aware sequential evaluation protocol with early termination, capturing realistic error propagation in safety-critical multi-step workflows. • We define a structured six-class failure taxonomy to systematically analyze grounding errors in densely structured medical interfaces. • We conduct a comprehensive evaluation of 16 state-of-the-art MLLMs , revealing a substantial performance gap between general-purpose models and GUI -specialized architectures in clinical environments. 2 Related Work 2.1 GUI Grounding Benchmarks and Agent Models Recent benchmarks have advanced GUI grounding in general-purpose environments. ScreenSpot [ 7 ] introduced a cross-platform grounding dataset across mobile, desktop, and web interfaces, while ScreenSpot-Pro [ 20 ] extended evaluation to high-resolution professional applications, highlighting degradation on small, dense UI elements. VenusBench-GD [ 55 ] and MMBench-GUI [ 42 ] further expanded coverage with hierarchical evaluation protocols, and web-centric datasets such as Mind2Web [ 8 ] and OmniACT [ 18 ] incorporated multi-step browsing tasks. Parallel to benchmark development, GUI grounding models have progressed rapidly. UI-TARS [ 30 ] integrates perception, unified action modeling, and System-2 reasoning at scale; Agent-X [ 2 ] emphasizes structured multi-step reasoning; and GUI-Actor [ 45 ] proposes coordinate-free grounding via a dedicated <ACTOR> token. General-purpose backbones such as Qwen2.5-VL [ 38 ] , Qwen3-VL [ 39 ] , and GPT-style models remain strong baselines. Despite this progress, existing benchmarks focus on general-purpose software, evaluate largely independent