How Smart Is Your GUI Agent? A Framework for the Future of Software Interaction DOI: XXXXXXX.XXXXXXX Journal: JACM Volume: 37 4 111 8

Sidong Feng, Chunyang Chen · 2026

Read How Smart Is Your GUI Agent? A Framework for the Future of Software Interaction DOI: XXXXXXX.XXXXXXX Journal: JACM Volume: 37 4 111 8 by Sidong Feng, Chunyang Chen (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2602-11514.txt.

Introduction

GUI agents are called agents at wildly other levels of autonomy. Sidong Feng and Chunyang Chen call for conceptual clarity with GUI Agent Autonomy Levels, a six-level frame that makes autonomy explicit. People now navigate web, desktop, and mobile through agents rather than clicking step by step.

The frame aims to clarify capability, responsibility, and risk, and to help benchmark progress toward trustworthy software contact. The paper is a position and frame piece, short and programmatic, arguing that without shared autonomy levels, comparisons between agents obscure more than they reveal.

Catalog record

CREATORSidong Feng, Chunyang Chen
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE15 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2602-11514

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

How Smart Is Your GUI Agent? A Framework for the Future of Software Interaction DOI: XXXXXXX.XXXXXXX Journal: JACM Volume: 37 4 111 8 Sidong Feng, Chunyang Chen Abstract . GUI agents are rapidly becoming a new interaction to software, allowing people to navigate web, desktop and mobile rather than execute them click by click. Yet “agent” is described with radically different degrees of autonomy, obscuring capability, responsibility and risk. We call for conceptual clarity through GUI Agent Autonomy Levels (GAL), a six-level framework that makes autonomy explicit and helps benchmark progress toward trustworthy software interaction. Figure 1. Users rely on a GUI agent in scenarios for the web (left), desktop (middle), and mobile (right). 1. Introduction The Graphical User Interfaces (GUIs) have become a global front door for modern software, mediating nearly every digital task, from writing documents and managing files to navigating enterprise systems and configuring applications. Their visual, flexible nature has made them indispensable for human users, but these same qualities make them difficult for machines to interact with. Unlike APIs, GUIs were never designed for autonomous interaction: their layouts shift across devices, elements move or re-render dynamically, and meaningful actions often depend on visual interpretation rather than stable programmatic execution. As software ecosystems grow more complex and fragmented, users increasingly face repetitive, multi-step workflows that remain manual despite major advances in digital automation. This gap has motivated the emergence of GUI agents, that is, an entity designed to understand user objectives, perceive interface elements, and act directly within GUIs on behalf of users. Early generations of GUI agents emerged as extensions of software automation efforts, providing foundational capabilities for interacting with interfaces programmatically, like Selenium ( 12 ) and AppleScript ( 2 ) , relied on predefined workflows and rigid selectors. Although effective for scripted testing or routine task replication, they offered little adaptability to dynamic layouts or ambiguous user goals. The recent GUI agents, powered by large multimodal models, can intelligently interpret interface layouts and autonomously execute multi-step actions across diverse software environments. Research prototypes such as WebAgent ( 5 ) and UI-TARS ( 9 ) demonstrate that GUI agents can jointly ground language, vision, and action. Complementing these advances, open-source benchmarks like OSWorld ( 14 ) and AndroidWorld ( 10 ) provide extensive evaluation environments that enable systematic measurement of agent capabilities and track improvements in complex GUIs. Meanwhile, industry has begun to deploy GUI agents at scale, delivering tangible benefits to everyday digital tasks and real-life workflow, beyond research settings. Emerging AI browsers, most notably OpenAI’s ChatGPT Atlas ( 7 ) and Perplexity Comet ( 8 ) , demonstrate how agents can autonomously navigate the unprecedented scale, semantic heterogeneity, and dynamic web content to assist with user tasks, e.g., e-commerce product comparison and checkout. Anthropic’s Claude Computer Use ( 1 ) further illustrates the maturation of these agents by operating full desktop environments, where it can compose documents, manipulate files, analyze data, and coordinate actions across applications. Beyond desktop and browser contexts, AI-integrated devices such as the Doubao AI Phone embed agentic capabilities directly into the operating system, enabling GUI agents to exercise end-to-end control within mobile interfaces, manage apps, and support on-device personal workflows. The expanding set of research prototypes, open-source benchmarks, and industrial-grade GUI agents has accelerated widespread exploration of this emerging field, accompanied by a number of surveys, technical reports, and blogs attempting to map the fast-moving landscape. Yet, the field still lacks a shared vocabulary for describing how autonomous a GUI agent truly is. Inspired by the autonomy levels like SAE levels ( 11 ) , this paper introduces the G UI A gent Autonomy L evels (GAL) .These levels provide a conceptual scaffold for researchers, practitioners, and users to benchmark agent capabilities, track progress over time, and identify the scientific challenges that must be addressed to advance the field. Figure 2. Levels of GUI Agent Autonomy 2. GUI Agent Autonomy Levels 2.1. Level 0 (No Automation) Example : “Set up a date event”: The user opens a calendar app, scrolls to a specific date, types in an event, and manually clicks “Save”, one step at a time. At this level, the user performs all actions manually. Every click, scroll, and keyboard input must be initiated by the user, with no automation, assistance, or system memory to support the task. The GUI responds only to direct human input, typically through a mouse, keyboard, or touch interface. 2.2. Level 1 (Minimal Assistance) Example : “Reply to an email”: As the user type, the agent suggests auto-completions based on prior messages, flags cases where an attachment is mentioned but missing, and highlights the “Send” button once required fields are complete. At Level 1, the agent begins to provide minimal assistance, but it does not execute any actions on the user’s behalf. Instead, it observes the user’s interactions and offers lightweight guidance, such as contextual cues, tooltips, or hints. Basic intelligence emerges at this stage: the agent can analyze GUI elements and provide supportive information, yet it remains strictly advisory rather than operational. Example tools include smart compose in Gmail . Such writing assistants analyze user input in real time, suggesting grammar corrections and alternative phrasing within the GUI. They guide the user through underlines and pop-up prompts, but do not make changes unless explicitly approved. Similarly, In-App Hints and Onboarding Assistants provide contextual tooltips and walkthroughs that help users navigate unfamiliar or complex interfaces, but they never interact with the interface directly. 2.3. Level 2 (Basic Automation) Example : “Open Instagram” and “Click Login button”: The agent executes each command immediately, without clarification or follow-up queries. At Level 2, the agent can now perform basic, single-step actions, but only when explicitly instructed by the user. The agent does not make decisions or initiate actions on its own, rather, it listens to user commands and executes them precisely. This marks a transition from passive guidance to active operation. Although the user must still decompose tasks into individual steps, the agent can execute these direct operations quickly and reliably, without requiring the user to physically interact with the GUI. Example tools include Selenium ( 12 ) , UI Automator ( 4 ) , and AppleScript ( 2 ) . These tools are widely used to automate user interactions in web, mobile, and desktop. They are script-driven and excel at executing predefined actions in a step-by-step manner (such as opening windows and moving files), but offer limited flexibility when interfaces change or workflows deviate from expected patterns. 2.4. Level 3 (Conditional Automation) Example : “Set an alarm for 7:30 AM every weekday”: The agent opens the clock application, navigates to the alarm tab, checks whether a recurring alarm already exists, creates one if necessary, and sets the appropriate time and days. If it encounters an unexpected situation, such as requesting additional permissions, it alerts the user and waits for confirmation before proceeding. At Level 3, the agent moves beyond responding to isolated commands and begins functioning as a true GUI agent, capable of perceiving interface elements, interpreting user intent, and autonomously executing multi-step workflows. Such agents can navigate sequences of screens and complete tasks end-to-end. However, their autonomy remains bounded by predefined rules and human-approved

Read the complete text