HalluClear: Diagnosing, Evaluating and Mitigating Hallucinations in GUI Agents

Chao Jin, Wenkui Yang, Hao Sun, Yuqi Liao, Qianyi Jiang, Kai Zhou, Jie Cao, Ran He, Huaibo Huang · 2026

Read HalluClear: Diagnosing, Evaluating and Mitigating Hallucinations in GUI Agents by Chao Jin, Wenkui Yang, Hao Sun, Yuqi Liao, Qianyi Jiang, Kai Zhou, Jie Cao, Ran He, Huaibo Huang (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2604-17284.txt.

Introduction

Jin and eight colleagues study a neglected flaw in AI agents that operate computer screens. These agents invent actions and targets, and one bad step can cascade into real mistakes. Their HalluClear suite sorts these errors into types, grades them with expert-checked judging, and counters them with step-by-step checks plus light retraining. Training on only 9,000 examples cut such errors sharply across agents and benchmarks.

Catalog record

CREATORChao Jin, Wenkui Yang, Hao Sun, Yuqi Liao, Qianyi Jiang, Kai Zhou, Jie Cao, Ran He, Huaibo Huang
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE115858
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2604.17284

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

HalluClear: Diagnosing, Evaluating and Mitigating Hallucinations in GUI Agents Chao Jin, Wenkui Yang, Hao Sun, Yuqi Liao, Qianyi Jiang, Kai Zhou, Jie Cao, Ran He, Huaibo Huang Abstract While progress in GUI agents has been largely driven by industrial-scale training, ungrounded hallucinations often trigger cascading failures in real-world deployments. Unlike general VLM domains, the GUI agent field lacks a hallucination-focused suite for fine-grained diagnosis, reliable evaluation, and targeted mitigation. To bridge this gap, we introduce HalluClear , a comprehensive suite for hallucination mitigation in GUI agents as a complement to computation-intensive scaling. HalluClear comprises: (1) a GUI-specific hallucination taxonomy derived from empirical failure analysis; (2) a calibrated three-stage evaluation workflow which enhances VLM-as-a-judge reliability via expert-annotated benchmarking and ensemble credibility estimation; and (3) a mitigation scheme based on closed-loop structured reasoning, enabling lightweight continual post-training with cold-start initialization for both generalist and GUI-specialist agents. Experiments across representative agents and public benchmarks demonstrate that post-training on only 9K samples within our suite can significantly reduce hallucinations, thereby improving grounding and action fidelity, offering a compute-efficient pathway to robust GUI automation. 1 Introduction Recent strides in Vision-Language Models (VLMs) qwen25vl ; qwen3vl have accelerated the development of visual GUI agents, a trend further bolstered by advanced teacher-guided distillation gpt4o ; gemini3flash , expansive offline corpora wu2024os-atlas ; wang2025opencua , and self-evolving schemes deepseekR1 ; qin2025uitars ; ye2025guiowl inspired by reinforcement learning (RL) ppo ; grpo ; dapo . While industrial efforts prioritize scaling data and computation to push performance frontiers ( Figure 2 , green arrow ), the hallucinations underlying step-wise failures remain underexplored. Consequently, even state-of-the-art agents may “ punch below their weight ”, as their decisions occasionally slip into the Hallucinated Zone . Complementarily, targeted mitigation offers a computationally efficient way to scale ( Figure 2 , blue arrow ). By shifting decisions back into the valid Tolerance Zone , it elevates the statistical performance lower bound and overall reliability without necessitating further upper-bound expansion. However, while existing VLM suites li2023pope ; guan2024hallusionbench ; bang2025hallulens ; hallusurvey have laid a solid foundation for static, perception-centric evaluation, adapting them to better reflect the dynamic, action-centric nature of GUI agents remains non-trivial. Specifically, distinguishing hallucinations from inevitable errors caused by partial observability (implied by the Privileged Zone ) requires specialized protocols beyond current capabilities. Figure 1 : Overview of HalluClear Suite. HalluClear aims to provide a comprehensive solution for diagnosing, evaluating, and mitigating hallucinations in GUI agents. (1) Constructed via bottom-up clustering and abstraction of failure cases from offline datasets, the case-driven taxonomy precisely categorizes diverse hallucination modes. (2) To ensure trustworthiness, the three-stage evaluation workflow (bottom left) qualifies VLM judges via credibility estimation on the expert-annotated JQ-Bench before measurement. (3) Addressing these issues, the mitigation scheme (right) incorporates a closed-loop OODA -style reasoning pattern during lightweight post-training. Motivated by this gap, we introduce HalluClear , a comprehensive suite for diagnosing, evaluating and mitigating hallucinations in GUI agents. As shown in Figure 1 , it consists of three core components. First, we systematically collect and analyze failure cases of advanced agents on open-source offline datasets. By employing bottom-up inductive clustering and abstraction of similar cases, while drawing on hallucination taxonomies from other domains, we construct a hallucination taxonomy tailored for GUI agents. Second, leveraging this taxonomy, we design an automated workflow for quantitative hallucination assessment that proceeds in three stages. Initially, human experts construct a fine-grained benchmark annotated with golden references. This benchmark then serves to calibrate and filter VLM judges based on their credibility, adhering to a “strict-in, strict-out” selection principle. Only qualified judges are permitted to participate in the final evaluation, where their credibility scores and the detected hallucination rates are jointly reported to ensure trustworthiness. Finally, inspired by the interdisciplinary Observe-Orient-Decide-Act (OODA) loop Johnson2022AutomatingOODA , we augment ReAct yao2022react with a closed-loop structured reasoning pattern incorporating a built-in reflection mechanism, and perform lightweight continual post-training with cold-start initialization and RL on agents. Experiments show that the OODA-inspired agents significantly reduce hallucinations, and therefore achieve better grounding and action performance on public benchmarks, providing a complementary perspective to large-scale post-training while remaining largely seamlessly integrable into existing systems. In brief, our contributions can be summarized as follows: • We construct a GUI-specific hallucination taxonomy via a systematic, bottom-up analysis and abstraction of real-world failure cases, categorizing prevalent failure modes into eight distinct subtypes. • We design a three-stage automated evaluation workflow that combines an expert-annotated benchmark with VLM-as-a-judge signals, thereby revealing the hallucination behavioral profiles across diverse agents. • We implement incremental, lightweight post-training mitigation that significantly reduces hallucinations and consequently improves grounding and action fidelity. 2 Preliminaries, Terminology & Taxonomy GUI agents interact with the environment sequentially under partial observability. We model the underlying environment as a discrete-time Partially Observable Markov Decision Process (POMDP) pomdp , where each u ser task u\in\mathcal{U} induces an underlying POMDP \mathcal{M}_{u}=\langle\mathcal{S},\mathcal{A},\mathcal{O},\mathbb{T},\mathbb{O},\mathcal{R}\rangle with the state space \mathcal{S} , the action space \mathcal{A} , the observation space \mathcal{O} , the transition kernel \mathbb{T}(s_{t+1}|s_{t},a_{t}) , the observation kernel \mathbb{O}(o_{t}|s_{t}) , and the reward function \mathcal{R}(\tau) given a specific trajectory/episode \tau . The recent advent of VLMs chen2024internvl ; qwen2vl ; qwen25vl ; qwen3vl has led to the rapid adoption of a pure-vision, end-to-end paradigm with explicit System-2-style reasoning wei2022cot ; yao2022react , which has emerged as a prominent approach for GUI agents xu2024aguvis ; wu2024os-atlas ; qin2025uitars ; wang2025uitars2 ; ye2025guiowl ; wang2025opencua . Instead of maintaining explicit beliefs in a POMDP, modern VLM-based GUI agents directly operate on the information state \tilde{s}_{t} based on history h_{t}=(o_{1:t-1},a_{1:t-1}) . However, the inherent hallucination of foundation models hallusurvey ; kalai2025language has not been systematically addressed in this domain. 2.1 Hallucination Definition Figure 2 : Mitigating Hallucination in GUI Agents. Massive training raises the capability ceiling, whereas hallucination mitigation elevates the performance floor by correcting avoidable hallucinations. This complementary gain is visualized by the red arrow, depicting the upward shift in expected returns (midline) from Agents w/o HalluClear to Agents w/ HalluClear . Policy optimization aims to maximize expected returns in a POMDP. In the context of GUI interaction, the expected returns are fundamentally constrained by partial observability due to the underspecified GUI front-end. We therefore define hallucination relative to the available information state \tilde{s}_{t} , distinguishing agent errors from the inevitable performance gap caused by partial observability. Definition. Given a policy \pi and state \tilde{s}_{t}\in\tilde{\mathcal{S}} , an action a_{t} is \delta -hallucinated if it overconfidently prefers a bad choice : \begin{cases}\pi(a_{t}|\tilde{s}_{t})>\pi(a^{*}|\tilde{s}_{t})\\ Q^{\pi^{*}_{\tilde{s}}}(\tilde{s}_{t},a^{*})-Q^{\pi^{*}_{\tilde{s}}}(\tilde{s}_{t},a_{t})>\delta\end{cases} (1) where \pi^{*}_{\tilde{s}}\in\text{argmax}_{\pi\in\Pi_{\tilde{s}}}\mathbb{E}_{\tau(\pi,\mathcal{M}_{u})}[\mathcal{R}(\tau)] is optimal within the search space \Pi_{\tilde{s}} , a^{*}\in\text{argmax}_{a}Q^{\pi^{*}_{\tilde{s}}}(\tilde{s}_{t},a) , and \delta>0 is the tolerance. By construction, \pi^{*}_{\tilde{s}} itself exhibits minimal hallucination under Equation 1 , just as shown in Figure 2 . Notice that \pi^{*}_{\tilde{s}} differs from the “oracle” policy \pi^{ora}_{\tilde{s}^{p}}\in\Pi_{\tilde{s}^{p}} , implied by ground truth in ideal error-free offline datasets with access to extended p rivileged state \tilde{s}^{p}\in\tilde{\mathcal{S}}^{p} (e.g., DOM and hindsight replay). Now, under the surjective mapping \phi:\tilde{\mathcal{S}}^{p}\to\tilde{\mathcal{S}} and its kernel equivalence relation, we have: \Pi_{\tilde{s}}\cong\left\{\pi\in\Pi_{\tilde{s}^{p}}\mid\forall\tilde{s}^{p}_{1}\sim_{\phi}\tilde{s}^{p}_{2},\ \pi(\cdot\mid\tilde{s}^{p}_{1})=\pi(\cdot\mid\tilde{s}^{p}_{2})\right\} (2) Equation 2 represents the measurability constraint . Consequently, we inherently have: \max_{\pi\in\Pi_{\tilde{s}}}\mathbb{E}_{\tau(\pi,\mathcal{M}_{u})}[\mathcal{R}(\tau)]\leq\mathbb{E}_{\tau(\pi^{ora}_{\tilde{s}^{p}},\mathcal{M}_{u}^{p})}[\mathcal{R}(\tau)]=J_{\text{ora}} (3) The condition for the inequality to be strict is perceptual aliasing where optimal actions differ despite inducing the same observation. In such cases, the oracle upper bound J_{\text{ora}} is

Read the complete text