Executable Agentic Memory for GUI Agent

Zerui Qin, Sheng Yue, Xingyuan Hua, Yongjian Fu, Ju Ren · 2026

Read Executable Agentic Memory for GUI Agent by Zerui Qin, Sheng Yue, Xingyuan Hua, Yongjian Fu, Ju Ren (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2605-12294.txt.

Introduction

Qin and four colleagues replace step-by-step guesswork in screen agents with a stored, runnable memory. Their Executable Agentic Memory organises past routines as a knowledge graph, built by state-aware search and mining of repeated action groups. Planning then runs as value-guided search over that graph. The approach beats strong baselines on AndroidWorld by up to about a fifth, cuts token costs sharply against a large commercial model, and answers in under three seconds on average.

Catalog record

CREATORZerui Qin, Sheng Yue, Xingyuan Hua, Yongjian Fu, Ju Ren
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE77786
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2605.12294

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Executable Agentic Memory for GUI Agent Zerui Qin, Sheng Yue, Xingyuan Hua, Yongjian Fu, Ju Ren Abstract Modern GUI agents typically rely on a model-centric and step-wise interaction paradigm, where LLMs must re-interpret the UI and re-decide actions at every screen, which is fragile in long-horizon tasks. In this paper, we propose Executable Agentic Memory (EAM), a structured Knowledge Graph (KG) that shifts GUI planning from free-form generation to a robust retrieval-and-execution process. Our approach includes a sample-efficient memory construction pipeline using state-aware DFS and action-group mining to compress multi-step routines. To ensure efficient planning, we introduce a value-guided graph search where a lightweight Q-function model steers Monte Carlo Tree Search (MCTS) over the KG. We theoretically establish bias-consistency for the Q-model and derive sample complexity bounds for path recovery. Empirically, EAM outperforms state-of-the-art baselines like UI-TARS-7B by up to 19.6 % 19.6\% on AndroidWorld, while reducing token costs 6 × 6\times relative to GPT-4o. With a 2.8 2.8 s average latency, EAM enables reliable, quick, and long-horizon GUI automation. 1 Introduction Modern Graphical User Interface (GUI) agents powered by (multimodal) LLMs can operate real-world apps by “seeing” screens and generating actions ( 26 ; 23 ; 31 ) . However, the dominant interaction paradigm remains model-centric and step-wise : at every screen, an LLM must re-interpret the UI, re-decide the next action, and implicitly maintain task progress in its context window. This makes long-horizon automation fragile: small perceptual or reasoning errors would compound, easily producing hallucinated actions and incorrect detours, especially in heterogeneous app environments where training coverage is limited ( 18 ; 16 ; 28 ; 5 ) . For instance, UI-TARS-7B ( 18 ) , considered a SOTA GUI agent model, achieves only 33% success rate on the long-horizon AndroidWorld benchmark ( 19 ) , while M3A, an agentic framework powered by GPT-4o, attains merely 40.5% ( 19 ) . To improve robustness, a natural direction is to equip agents with external knowledge and memory. Some efforts ( 25 ; 24 ; 2 ; 21 ) maintain textual memory of historical interactions, such as workflow patterns and decision heuristics, and inject them into the LLM’s context to guide task planning. Others construct external knowledge bases by extracting action-level knowledge (e.g., element functionality or successful trajectories) from exploration, storing them as vector databases or knowledge graphs, and retrieving relevant knowledge at inference time to augment decision-making ( 29 ; 10 ; 7 ; 14 ) . However, such in-context knowledge injection remains unreliable due to model-centric generation and ignorance of inherent structured information in historical trajectories, making it difficult to reliably reproduce executable paths from historical knowledge. Moreover, repeated retrieval and step-wise generation introduce substantial cost and latency, hindering real-time deployment. In this paper, we investigate Executable Agentic Memory (EAM), which can serve as a persistent, structured representation of the environment interaction logic, learned from historical interactions, and can be queried at test time so that planning can be augmented by retrieval and verification rather than free-form generation. Specifically, EAM enables the agent to (1) remember the GUl as a state machine (what states exist, which actions are available, and where they lead), and (2) reason over this memory to extract an executable path that is guaranteed to stay on valid transitions. To this end, we first propose a sample-efficient memory construction pipeline: a state-aware DFS exploration strategy that systematically covers task-relevant transitions with minimal redundant interactions, coupled with state deduplication and action-group mining to compress frequent multi-step routines into reusable high-level actions, yielding a compact yet executable GUI logic knowledge graph. We then propose a compute-efficient retrieval mechanism: a value-guided graph search procedure in which a lightweight Q-function model steers MCTS over the constrained KG action space to rapidly select faithful high-reward paths from noisy experience; when needed, the agent can make only a single cloud call to summarize and validate the retrieved path into a grounded plan. Theoretically, we establish a bias-consistency guarantee for the learned Q-model on the critical set and derive a finite-sample complexity bound under which the value-guided MCTS recovers the optimal execution path with high probability. We evaluate our method on the AndroidWorld, MobileMiniWob++, and DroidTask benchmarks. Results show that our framework consistently outperforms the existing baselines, surpassing the state-of-the-art UI-TARS-7B by up to 19.6\% while reducing token costs 6\times relative to GPT-4o. Our Q-guided MCTS and iterative self-training pipeline bridge the reasoning gap for small models through fine-grained credit assignment, while the action group mechanism minimizes search complexity to reach a 2.8 s average latency. These findings demonstrate that grounding decision-making in structured knowledge graphs enables reliable, high-speed, and long-horizon planning for GUI agents. 2 Related Work GUI Agents. Early efforts adapted foundation models (GPT-4, GPT-4o) to GUI tasks ( 26 ; 22 ) , with Zheng et al . ( 34 ) demonstrating that GPT-4V outperforms text-based models in web scenarios. Zhang et al . ( 31 ) augment GPT-4V with a memory module for historical actions. Subsequent work explored modular frameworks: Wang et al . ( 23 ) integrate planning, decision, and reflection modules; Zhang et al . ( 33 ) propose multi-agent collaboration; and Zhu et al . ( 36 ) design a hierarchical planner-executor architecture. However, these cloud-based frameworks incur high API costs and latency, and suffer from hallucinations due to limited GUI domain knowledge. More recent work pursues end-to-end GUI agents via parameter training. Cheng et al . ( 1 ) train a dedicated GUI grounding model with cross-platform data, while UI-TARS ( 18 ) introduces a comprehensive pre-training to fine-tuning pipeline. To improve generalization, Luo et al . ( 16 ) and Lu et al . ( 15 ) apply rule-based RL algorithms such as GRPO ( 20 ) . AutoDroid-V2 ( 27 ) fine-tunes a lightweight model to generate executable scripts in one shot. Despite these advances, on-device models ( \leq 3B) remain limited in reasoning, struggling with complex multi-step tasks. Knowledge-aware GUI Agents. To mitigate hallucinations and improve adaptability, some works utilize historical memory to guide task planning. Wang et al . ( 25 ) propose a workflow memory extracting reusable patterns from past experiences. Mobile-Agent-E ( 24 ) introduces a self-evolving framework accumulating general guidance over time. MAGNET ( 21 ) constructs dual-level memory for element grounding and workflow retrieval to handle UI drift. However, these methods rely solely on LLMs’ contextual understanding without accounting for dynamic environment interactions. Another line of work focuses on reliable action generation. AutoDroid ( 26 ) collects transition knowledge via random exploration. GUI-explorer ( 29 ) mines element functionality by analyzing GUI state changes. KG-RAG ( 7 ) transforms UI Transition Graphs into vector databases and distills reusable actions based on intent. While these approaches improve action accuracy, path generation still relies on LLM reasoning over retrieved context rather than direct extraction from an executable state machine. Moreover, massive API calls for step-wise decision-making incur substantial costs and latency. LLM-based Monte Carlo Tree Search. Inspired by AlphaGo, recent work explores guiding LLM inference with tree search to improve reasoning on structured tasks. Zhou et al . ( 35 ) propose an LLM-MCTS framework leveraging environment feedback for decision-making, while Xie et al . ( 30 )

Read the complete text