DocOS: Towards Proactive Document-Guided Actions in GUI Agents

Jingjing Liu, Ziye Huang, Zihao Cheng, Zeming Liu, Jiahong Wu, Yuhang Guo, Kehai Chen, Yunhong Wang, Haifeng Wang · 2026

Read DocOS: Towards Proactive Document-Guided Actions in GUI Agents by Jingjing Liu, Ziye Huang, Zihao Cheng, Zeming Liu, Jiahong Wu, Yuhang Guo, Kehai Chen, Yunhong Wang, Haifeng Wang (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2605-18048.txt.

Introduction

Jingjing Liu and eight colleagues introduce proactive document-guided action for GUI agents in open-web settings. GUI agents lean on static knowledge from pre-training, limiting long-tailed tasks needing explicit procedural knowledge. The new paradigm lets agents search alone for relevant documentation to resolve long-tailed tasks. The team proposes DocOS, a benchmark requiring agents to browse, locate documentation, and ground instructions into GUI actions. Tests reveal dual bottlenecks: locating relevant information during proactive search, and grounding retrieved instructions into precise actions.

Catalog record

CREATORJingjing Liu, Ziye Huang, Zihao Cheng, Zeming Liu, Jiahong Wu, Yuhang Guo, Kehai Chen, Yunhong Wang, Haifeng Wang
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE47836
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2605.18048

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

DocOS: Towards Proactive Document-Guided Actions in GUI Agents Jingjing Liu, Ziye Huang, Zihao Cheng, Zeming Liu, Jiahong Wu, Yuhang Guo, Kehai Chen, Yunhong Wang, Haifeng Wang Abstract While Graphical User Interface (GUI) agents have shown promising performance in automated device interaction, they primarily depend on static parametric knowledge from pre-training or instruction tuning. This reliance fundamentally limits their ability to handle long-tailed tasks that require explicit procedural knowledge absent from model parameters, often forcing agents to resort to inefficient and brittle trial-and-error exploration. To mitigate this limitation, we introduce Proactive Document-Guided Action for GUI agents in dynamic, open-web environments, a novel paradigm that mirrors human problem-solving by enabling agents to autonomously search for relevant documentation to resolve long-tailed tasks. To evaluate agents’ capability in this paradigm, we propose DocOS , a benchmark designed to assess document-guided problem solving in fully interactive environments. DocOS requires agents to autonomously navigate a web browser, locate relevant online documentation, comprehend procedural instructions, and faithfully ground them into executable GUI actions. Extensive experiments reveal that progress is strictly constrained by dual bottlenecks: agents struggle to reliably locate relevant information during proactive search and frequently fail to faithfully ground retrieved instructions into precise actions, pointing toward document-guided interaction as a crucial pathway for enabling self-evolving GUI agents in dynamic environments 1 1 1 Dataset and codes are publicly available at https://github.com/BUAA-IRIP-LLM/DocOS . 1 Introduction Figure 1 : An example of a Proactive Document-Guided Action task. The workflow is divided into two distinct phases: Proactive Knowledge Retrieval (top row) and Document-Grounded Execution (bottom row). Recent advances in Multimodal Large Language Models (MLLMs) have driven rapid progress in Graphical User Interface (GUI) agents, enabling automated interaction across a wide range of applications ( Wang et al. 2024b ; Nguyen et al. 2025 ; Yin et al. 2024 ; Lu et al. 2025 ; Liu et al. 2025 ) , including web navigation ( Yao et al. 2022 ; Deng et al. 2023 ; Zhou et al. 2024 ) , desktop environments ( Xie et al. 2024 ) , and mobile devices ( Zhang et al. 2023 ; Wang et al. 2024a ) . These developments mark an important step toward general agents capable of assisting humans with diverse real-world tasks. Despite this progress, a fundamental limitation persists: existing GUI agents predominantly rely on static parametric knowledge acquired during pre-training or instruction tuning ( Xu et al. 2025a ; Liu et al. 2024 ; Zhou et al. 2025 ) . While sufficient for common interactions, such knowledge is inherently incomplete and quickly becomes brittle when agents are confronted with long-tailed, application-specific tasks. These tasks often require explicit procedural instructions, precise option semantics, or rarely used functionalities that are absent from the model’s fixed parameters. As illustrated in Figure 1 , for a specific instruction like "Create a run configuration from a python template," the agent lacks the inherent knowledge to navigate the PyCharm interface directly. Instead, it must first navigate the open web to retrieve the official "Run/Debug Configurations" guide, utilizing this external document to ground its subsequent actions in the software. This behavior increases execution failures and hallucinations, fundamentally undermining the reliability and deployability of GUI agents in dynamic environments. To mitigate this limitation, we introduce Proactive Document-Guided Action , a novel paradigm that mirrors human problem-solving by enabling agents to autonomously search the open web for documentation to resolve long-tailed tasks. Under this paradigm, agents are no longer restricted to static internal knowledge. Instead, they proactively navigate browsers to locate and comprehend documents in real time. This paradigm decouples an agent’s reasoning capability from the storage of vast, domain-specific knowledge. By dynamically acquiring explicit procedural guidance on the fly, agents can faithfully ground their actions in up-to-date documentation, enabling adaptation to long-tailed and evolving tasks without extensive retraining. Guided by this paradigm, we propose DocOS , a benchmark designed to assess document-guided problem solving in fully interactive environments. Specifically, we curate a suite of high-difficulty instructions targeting long-tailed functionalities that are typically unsolvable without external documents. Crucially, we establish a dynamic execution environment (i.e., Docker) ( Xie et al. 2024 ) where agents are not passively fed information and more details are in Appendix A . Instead, consistent with our paradigm, they are required to autonomously navigate a web browser, locate relevant online documentation, comprehend procedural instructions, and faithfully ground them into executable GUI actions . This setup moves beyond static, information-provided evaluations by explicitly assessing an agent’s end-to-end capability to autonomously search relevant web documentation and ground the acquired procedural documents into precise GUI actions. To systematically evaluate current GUI agents under this paradigm, we conduct extensive experiments on DocOS. Specifically, we design two distinct task settings to decouple and assess the core capabilities Proactively Document Search , which evaluates the agent’s ability to locate documents via web browsing, and Document-Grounded Execution , which tests its capacity to implement instructions given the documentation. Results reveal that GUI agents are strictly constrained by dual bottlenecks: agents struggle to reliably locate relevant information during the proactive search phase, and even when provided with oracel document in the execution phase, they frequently fail to faithfully ground retrieved instructions into precise actions. These findings highlight critical gaps in both document retrieval and document-conditioned execution, limiting end-to-end reliability in real-world deployment. Overall, our contributions are summarized as follows: • To the best of our knowledge, we are the first to propose Proactive Document-Guided Action for GUI agents, a novel paradigm that mirrors human problem-solving by enabling agents to autonomously search the open web for documents and apply them to solve long-tailed tasks. • Guided by this paradigm, we introduce DocOS , a benchmark designed to evaluate GUI agents to autonomously navigate a web browser, locate relevant online documentation, comprehend procedural instructions, and accurately translate them into actions. • We perform a systematic evaluation under two complementary settings: Proactive Document Search and Document-Grounded Execution . Our results show that the evaluated agents are fundamentally limited by dual bottlenecks : inaccurate information localization during document search and unfaithful grounding of instructions into actions, underscoring a critical gap toward robust, self-guided deployment. 2 Related Work Table 1 : Comparison of virtual agent benchmarks. The columns indicate whether a benchmark supports proactive document search, document-grounded execution, interaction with dynamic web environments (i.e., non-static web pages), complex multi-step tasks, and fine-grained subtask-level evaluation. Proactively Document Search Document-Grounded Execution Dynamic Web Environment Complex Task Subtask Evaluation OmniACT ( Kapoor et al. 2024 ) ✗ ✗ ✗ ✗ ✗ ScreenSpot-Pro ( Li et al. 2025 ) ✗ ✗ ✗ ✗ ✗ WebArena ( Zhou et al. 2024 ) ✗ ✗ ✗ ✗ ✗ VisualWebArena ( Koh et al. 2024 ) ✗ ✗ ✗ ✗ ✗ OSWorld ( Xie et al. 2024 ) ✗ ✗ ✗ ✗ ✗ CRAB ( Xu et al. 2025b ) ✗ ✗ ✗ ✗ ✓ VideoGUI ( Lin et al. 2024 ) ✗ ✗ ✗ ✓ ✓ GUI-World ( Chen et al. 2024 ) ✗ ✗ ✗ ✓ ✗ AitW ( Rawles et al. 2023 ) ✗ ✗ ✗ ✓ ✗ Mind2Web ( Deng et al. 2023 ) ✗ ✗ ✗ ✓ ✗ Spider2-V ( Cao et al. 2024 ) ✗ ✗ ✓ ✓

Read the complete text