UI-KOBE: Knowledge-Oriented Behavior Exploration for Lightweight Graph-Guided GUI Agents

Yuxiang Chai, Han Xiao, Xinyu Fu, Jinpeng Chen, Rui Liu, Hongsheng Li · 2026

Read UI-KOBE: Knowledge-Oriented Behavior Exploration for Lightweight Graph-Guided GUI Agents by Yuxiang Chai, Han Xiao, Xinyu Fu, Jinpeng Chen, Rui Liu, Hongsheng Li (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2605-29534.txt.

Introduction

Small mobile interface agents, often unreliable at planning from screenshots alone, gain reusable knowledge about each app in UI-KOBE, a system from Yuxiang Chai and five colleagues. Large vision-language models handle phone tasks well but are too heavy for on-device use. UI-KOBE first explores an app alone and maps its screens as graph nodes joined by executable transitions. At run time the small agent locates its current node and chooses among staying put, a neighboring transition, finishing, or a free fallback action.

Catalog record

CREATORYuxiang Chai, Han Xiao, Xinyu Fu, Jinpeng Chen, Rui Liu, Hongsheng Li
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE38204
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2605.29534

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

UI-KOBE: Knowledge-Oriented Behavior Exploration for Lightweight Graph-Guided GUI Agents Yuxiang Chai, Han Xiao, Xinyu Fu, Jinpeng Chen, Rui Liu, Hongsheng Li Abstract Recent advances in mobile GUI agents have shown strong potential for automating mobile tasks, but most effective systems still depend on large vision-language models for screenshot understanding and long-horizon planning. Small GUI agents that can be deployed directly on mobile devices are more attractive for practical use, offering lower inference cost and better protection of sensitive on-device information. However, due to limited model capacity, such lightweight agents remain unreliable when planning and executing GUI tasks end-to-end from screenshots alone. We propose Knowledge-Oriented Behavior Exploration ( UI-KOBE ), a framework that improves lightweight mobile GUI agents with reusable app-specific graph knowledge. UI-KOBE first autonomously explores a mobile application and constructs an app knowledge graph, where nodes represent distinct UI states and edges represent executable transitions. At runtime, a lightweight GUI agent uses the graph as external guidance: given a user task and the current screenshot, it identifies the current graph node and selects among self-loop actions, neighboring transitions, task completion, or fallback free actions associated with that node. By supporting runtime decisions with app-specific graph guidance, UI-KOBE reduces the burden of end-to-end GUI planning and helps lightweight models perform mobile GUI tasks more effectively, offering a practical step toward efficient, interpretable, and privacy-conscious on-device GUI agents. 1 Introduction Graphical User Interface (GUI) agents have recently shown strong potential for automating mobile and desktop tasks, driven by advances in vision-language models (VLMs) that can interpret screenshots and generate actions. Typically GUI interaction is an end-to-end formulation: given a task and the current screen, the model directly plans and executes a sequence of actions. While effective with large-scale proprietary or open-source models, this paradigm introduces two practical challenges. First, large open-source models require substantial computational resources, making deployment on-device difficult, while proprietary models requires high API costs. Second, smaller models that are suitable for on-device deployment, such as 4B-scale models, often struggle with long-horizon reasoning and planning, leading to unreliable task execution. Despite the limitation, lightweight GUI agents are highly desirable. They offer lower inference cost and better alignment with real-world deployment scenarios where sensitive user data can remain local. However, enabling small models to perform complex GUI tasks remains an open challenge. In particular, asking a small model to reason over the entire task at each step places a heavy burden on its limited capacity. Figure 1: Overview of our framework. UI-KOBE first explores a target app and constructs an app knowledge graph, where nodes represent UI states and edges represent executable transitions. At runtime, a lightweight GUI agent uses the graph to identify the current node, retrieve local transition knowledge, select the next action, and execute the task step by step. In this work, we argue that mobile GUI task execution should not rely solely on end-to-end reasoning at runtime. Instead, we propose to decouple app knowledge acquisition from task-time execution . We introduce Knowledge-Oriented Behavior Exploration ( UI-KOBE ), a framework that builds a reusable knowledge graph of an application through autonomous exploration, and this graph can be used to guide a runtime agent during task execution. Figure 1 illustrates the overall pipeline, including app exploration, graph construction, and graph-guided runtime usage. UI-KOBE represents an application as a directed graph, where nodes correspond to distinct UI states and edges correspond to transitions between states. The graph is constructed by an exploration agent that iteratively observes screens, executes actions, and records transitions. Each node captures semantic and structural information about a UI state, while each edge encodes both low-level actions and higher-level interaction patterns. Importantly, this graph represents general app behavior rather than specific task data, making it reusable across different tasks and users. At runtime, the graph serves as an external knowledge source for a lightweight GUI agent. Instead of performing open-ended planning, the agent first identifies the current UI state within the graph and then selects the next action from a constrained set of graph-supported options, including self-loop operations and transitions to neighboring states. This formulation reduces GUI task execution to a sequence of guided local decisions, significantly lowering the reasoning burden on small models. When graph guidance is unavailable, the system falls back to a naive planner, ensuring robustness without reverting to full end-to-end reasoning. By leveraging pre-built app knowledge, UI-KOBE enables small models to perform GUI tasks more reliably. It also improves interpretability by grounding each action in explicit graph structures and supports reuse across tasks without repeated exploration. This paper makes the following contributions: • We propose a paradigm that decouples app knowledge acquisition from task-time execution, enabling graph-guided mobile GUI agents for lightweight models. • We introduce UI-KOBE, a method for constructing a reusable app knowledge graph through autonomous exploration, including principled definitions of UI states (nodes) and executable transitions (edges). • We design a graph-guided runtime agent that leverages local graph context to replace end-to-end planning with guided decision making, substantially improving the capability and reliability of small GUI agents. 2 Related Work 2.1 End-to-End GUI Agents GUI agents aim to complete user tasks by perceiving graphical interfaces, reasoning over instructions, and executing actions such as clicking, typing, and swiping; recent surveys and benchmarks provide comprehensive overviews and evaluation resources for this rapidly growing area ( 20 ; 11 ; 9 ; 4 ; 5 ; 17 ) . Most recent systems formulate GUI control as an end-to-end problem, where the model predicts actions directly from screenshots or UI representations and task instructions. Representative examples include UI-TARS 15 , Mobile-Agent-V3/V3.5 27 ; 24 , UI-Genie 22 , UI-Venus 19 , and MAI-UI 30 , which improve GUI grounding, planning, and execution through stronger foundation models, trajectory data, reinforcement learning, and model merging. Several works further study smaller GUI models, such as InfiGUI-R1 12 , UI-R1 14 , Ferret-UI-Lite 26 , and small variants of MAI-UI/UI-Venus, showing the promise of lightweight agents for efficient deployment. Different from these end-to-end approaches, our work reduces the runtime reasoning burden of lightweight agents by first constructing reusable app-specific graph knowledge and then using it to guide step-by-step decisions. 2.2 Exploration-Based GUI Agents Recent work has also explored using app-specific knowledge, memory, or trajectory history to improve GUI task execution. AppAgent 29 builds an app-level knowledge base from autonomous exploration and demonstrations, enabling agents to reuse prior interaction experience. AutoDroid 21 constructs UI transition graphs (UTGs) through app exploration and uses them as structured app memory for mobile task automation. UI-Mem 23 introduces a memory mechanism that stores and reuses historical GUI interaction experience to improve long-horizon task execution and reduce repeated errors. KG-RAG 8 transforms fragmented UTGs into a vectorized knowledge database of intent-trajectory pairs, allowing agents to retrieve relevant navigation paths during online execution. GraphPilot 28 constructs app-specific knowledge graphs of page functions, element functions, and transition rules, and uses them to generate nearly complete action sequences with fewer LLM queries. Different from these methods, UI-KOBE focuses on building a semantic state-transition graph as a reusable behavioral abstraction, and uses it as a local decision scaffold for lightweight GUI agents: instead of retrieving an entire trajectory or generating a full action sequence, the runtime agent

Read the complete text