FineState-Bench: Benchmarking State-Conditioned Grounding for Fine-grained GUI State Setting

Fengxian Ji, Jingpu Yang, Zirui Song, Yuanxi Wang, Zhexuan Cui, Yuke Li, Qian Jiang, Xiuying Chen · 2026

Read FineState-Bench: Benchmarking State-Conditioned Grounding for Fine-grained GUI State Setting by Fengxian Ji, Jingpu Yang, Zirui Song, Yuanxi Wang, Zhexuan Cui, Yuke Li, Qian Jiang, Xiuying Chen (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2604-27974.txt.

Introduction

Ji and colleagues introduce FineState-Bench, a test of whether computer-using agents can set a screen control to an exact intended state rather than merely finish a task. With thousands of cases across desktop, web and mobile, it diagnoses failure stage by stage, from finding a control to reaching its target state. Leading models still succeed exactly only a minority of the time, though hints about where to look lift scores notably.

Catalog record

CREATORFengxian Ji, Jingpu Yang, Zirui Song, Yuanxi Wang, Zhexuan Cui, Yuke Li, Qian Jiang, Xiuying Chen
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE44896
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2604.27974

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

FineState-Bench: Benchmarking State-Conditioned Grounding for Fine-grained GUI State Setting Fengxian Ji, Jingpu Yang, Zirui Song, Yuanxi Wang, Zhexuan Cui, Yuke Li, Qian Jiang, Xiuying Chen Abstract Despite the rapid progress of large vision-language models (LVLMs), fine-grained, state-conditioned GUI interaction remains challenging. Current evaluations offer limited coverage, imprecise target-state definitions, and an overreliance on final-task success, obscuring where and why agents fail. To address this gap, we introduce FineState-Bench , a benchmark that evaluates whether an agent can correctly ground an instruction to the intended UI control and reach the exact target state. FineState-Bench comprises 2,209 instances across desktop, web, and mobile platforms, spanning four interaction families and 23 UI component types, with each instance explicitly specifying an exact target state for fine-grained state setting. We further propose FineState-Metrics , a four-stage diagnostic pipeline with stage-wise success rates: Localization Success Rate (SR@Loc), Interaction Success Rate (SR@Int), Exact State Success Rate at Locate (ES-SR@Loc), and Exact State Success Rate at Interact (ES-SR@Int), and a plug-and-play Visual Diagnostic Assistant (VDA) that generates a Description and a bounding-box Localization Hint to diagnose visual grounding reason via controlled w/ vs. w/o comparisons. On FineState-Bench, exact goal-state success remains low: ES-SR@Int peaks at 32.8% on Web and 22.8% on average across platforms. With VDA localization hints, Gemini-2.5-Flash gains +14.9 ES-SR@Int points, suggesting substantial headroom from improved visual grounding, yet overall accuracy is still insufficient for reliable fine-grained state-conditioned interaction Github. rmTeXGyreTermesX [*devanagari]rmLohit Devanagari [*arabic]rmNoto Sans Arabic 1 Introduction Figure 1: FineState-Bench overview and VDA. (a) Static data creation pipeline (filtering, supplementation, annotation, instruction/state drafting, verification). (b) VDA-assisted evaluation that appends target-region localization hints for controlled comparisons. (c) Fine-grained interaction taxonomy with four families and 23 UI component types. (d) Example instances with precise target boxes and normalized interaction points. Recent advances in LVLMs have enabled a new class of GUI agents that can execute natural-language instructions on real-world software interfaces 29 ; 41 . By integrating visual understanding with language-conditioned decision making, such agents have shown promising capability in operating complex applications across desktop, web, and mobile environments. Representative systems such as CogAgent 15 and AppAgent further demonstrate the potential of this paradigm for practical human computer interaction. To support systematic progress, prior benchmarks including AITW 14 ; 27 and ScreenSpot 50 ; 35 have provided standardized testbeds for evaluation. Despite rapid progress in LVLM-based GUI agents 54 ; 27 , achieving fine-grained state-conditioned interaction remains challenging in practice. In many real applications, a single instruction requires setting a UI control to an exact target state using only the agent’s first predicted interaction point, such as adjusting a slider to a precise value, selecting an exact date/time, or choosing a specific color. However, agents that perform strongly under coarse success criteria or standard grounding benchmarks such as ScreenSpot/SeeClick 8 , Android in the Wild 37 , and VisualWebArena 17 can still fail to reliably reach precise target states. A key limitation lies in current evaluation practices. First, at the benchmark and task-definition level, existing evaluations for GUI and web/mobile agents predominantly focus on end-to-end task completion or click-level grounding. This design under-represents state-conditioned interaction scenarios, and the target specifications are often insufficiently precise to enable unambiguous verification of intermediate or final target states 37 ; 10 ; 61 ; 17 ; 26 ; 51 ; 4 . Second, at the evaluation-protocol and metric level, many studies primarily report aggregate outcomes such as final task success or overall accuracy. Such coarse metrics collapse the entire perception–grounding–interaction pipeline into a single score, obscuring where failures occur and preventing fine-grained diagnostic analysis and failure attribution 17 ; 45 ; 19 ; 21 ; 47 . As a consequence, prior work may arrive at inconsistent conclusions regarding failure sources. To address these limitations, we introduce FineState-Bench, a cross-platform benchmark of 2,209 instances for single-step, fine-grained state-conditioned GUI exact state setting with exact goal-state verification. Rather than modeling full interactive trajectories, FineState-Bench focuses on a controlled static single-step setting to isolate fine-grained state-conditioned grounding and exact state-setting ability. Agents are evaluated under a single-step, point-based protocol, as illustrated in Figure 1 and Figure 4 . Unlike proxy-based evaluations, each instance provides exact goal-state labels and dual-region annotations (a control-extent locate box and an interactable-core box) under current/target configurations, enabling unambiguous verification of exact goal-state attainment. Built on these annotations, we propose FineState-Metrics, a stage-wise diagnostic pipeline (SR@Loc, SR@Int, ES-SR@Loc, ES-SR@Int) that decomposes performance from component grounding to interactable-core grounding and point precision, and finally exact goal-state attainment. To further quantify the grounding errors, we introduce a plug-and-play Visual Diagnostic Assistant (VDA) that optionally appends a Description and/or a Localization Hint; experiments show that while current agents exhibit a low success floor, VDA yields substantial gains, indicating insufficient accuracy for broad fine-grained state-conditioned interactions. Our main contributions are: (1) We define and study fine-grained, state-conditioned GUI state setting with explicit goal-state labels and exact verification in desktop/web/mobile platforms. (2) We build and release FineState-Bench featuring dual-region annotations under current/target configurations for component localization and interactable-core precision, enabling fine-grained, reproducible evaluation. (3) We introduce FineState-Metrics and VDA to decompose failures and perform controlled input-augmentation analysis, and we benchmark 8 representative agents to quantify how much performance is limited by interactable-core localization and point precision. 2 Related Work GUI Agents. GUI agents have advanced rapidly with large vision-language and multimodal models 52 ; 30 . Early studies often relied on general-purpose models for GUI operation 46 ; 1 ; 59 , while later work increasingly builds GUI-specialized agents and UI grounding models 15 ; 40 ; 22 ; 42 ; 18 ; 13 ; 6 ; 34 . These systems span mobile and desktop platforms 39 ; 31 ; 16 ; 23 ; 11 ; 53 ; 48 ; 49 , yet reliably achieving state-conditioned exact state setting remains challenging. In practice, small execution errors can accumulate into user-visible failures, especially for precise controls such as sliders, pickers, and professional UI widgets, motivating evaluations that stress fine-grained state manipulation rather than only task completion. GUI Agent Evaluation. Existing evaluations include interactive end-to-end benchmarks 61 ; 17 ; 36 ; 44 ; 57 ; 5 ; 55 and offline grounding benchmarks on screenshots or professional software 10 ; 58 ; 33 ; 9 . While these benchmarks improve realism or grounding assessment, they often under-cover state-conditioned interaction scenarios and lack precise target-state specifications for unambiguous verification. Moreover, many protocols emphasize aggregate success rates, limiting deeper analysis 60 ; 56 . Recent benchmarks probe robustness or distribution shifts, but they typically do not isolate failures from mis-perception, mis-localization, or incorrect state outcomes when exact goal-state attainment is required. Diagnosis and Failure Attribution. Recent work highlights that coarse metrics can obscure bottlenecks and yield divergent conclusions about failure sources 38 . Related efforts improve grounding or data coverage 7 or analyze reliability issues 25 ; 20 , but controlled diagnosis for disentangling visual grounding from non-visual interaction and state-control factors remains limited. 3 FineState-Bench 3.1 Problem Definition FineState-Bench targets fine-grained, state-conditioned GUI state setting with exact goal states. Given a screenshot x and two instructions (I^{0},I^{1}) , an agent predicts two points (p^{0},p^{1}) : p^{0} indicates the target control’s location in the current UI, and p^{1} indicates

Read the complete text