Instruction Agent: Enhancing Agent with Expert Demonstration

Yinheng Li, Hailey Hultquist, Justin Wagle, Kazuhito Koishida · 2025

Read Instruction Agent: Enhancing Agent with Expert Demonstration by Yinheng Li, Hailey Hultquist, Justin Wagle, Kazuhito Koishida (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2509-07098.txt.

Introduction

Computer-using agents that copy a human demonstration instead of improvising are built by Yinheng Li and three co-authors in a 2025 artificial intelligence paper. Graphical interface agents have advanced quickly but still fail on complex tasks with novel interface elements, long action sequences, and personal workflows. One expert demonstration is the input here.

From that single demonstration, the Instruction Agent builds a step-by-step instruction list, then follows the user's intended path closely as it acts, avoiding improvisation errors. A verifier module judges the outcome of each action, and a backtracker handles surprises such as pop-up windows. On a set of OSWorld tasks that all top-ranked agents had failed, the system achieved a sixty percent success rate. The authors offer the design as a practical, extensible bridge toward reliable real-world interface automation.

Catalog record

CREATORYinheng Li, Hailey Hultquist, Justin Wagle, Kazuhito Koishida
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE34 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2509-07098

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Instruction Agent: Enhancing Agent with Expert Demonstration Yinheng Li, Hailey Hultquist, Justin Wagle, Kazuhito Koishida Abstract Graphical user interface (GUI) agents have advanced rapidly but still struggle with complex tasks involving novel UI elements, long-horizon actions, and personalized trajectories. In this work, we introduce Instruction Agent , a GUI agent that leverages expert demonstrations to solve such tasks, enabling completion of otherwise difficult workflows. Given a single demonstration, the agent extracts step-by-step instructions and executes them by strictly following the trajectory intended by the user, which avoids making mistakes during execution. The agent leverages the verifier and backtracker modules further to improve robustness. Both modules are critical to understand the current outcome from each action and handle unexpected interruptions(such as pop-up windows) during execution. Our experiments show that Instruction Agent achieves a 60% success rate on a set of tasks in OSWorld that all top-ranked agents failed to complete. The Instruction Agent offers a practical and extensible framework, bridging the gap between current GUI agents and reliable real-world GUI task automation. 1 Introduction Graphical User Interface (GUI) agents leverage UI elements and input methods such as keyboard and mouse to interact with digital devices similarly to humans. GUI agents are predominantly powered by Multimodal Large Language Models (MLLMs). Recent research on GUI agents Koh et al. (2024) ; Cheng et al. (2024) ; Zhang et al. (2025) ; OpenAI (2025) ; Anthropic PBC (2024) ; Qin et al. (2025) has demonstrated significant potential in automating user interactions and facilitating diverse tasks across digital platforms, including desktops and mobile devices. Since the introduction of evaluation benchmarks such as OSWorld Xie et al. (2024) and Windows Agent Arena Bonatti et al. (2024) , rapid progress has been made in the performance capabilities of GUI agents. For instance, in the OSWorld benchmark Xie et al. (2024) , the task success rate has increased from 5.8% in April 2024 (achieved by Gemini Vision Pro) to 42.9% by OpenAI’s CUA O3 model. Nevertheless, a substantial performance gap substantial gap remains compared to human-level performance (72.36%). Despite the rapid development of GUI-based agents, some tasks remain difficult or infeasible for current agents due to complex UI elements, highly idiosyncratic procedures, or long-horizon workflows spanning many steps. However, automating these tasks is still highly desirable. Figure 1: Instructor–Actor Agent In this paper, we present a method that automates many tasks previously unsolved by the SOTA agents using a single human demonstration. Our agent is a test-time–only, training-free system built on existing LLM APIs and open-source models. From the demonstration, it derives a high-quality task plan as well as UI grounding hints for the grounding model. To ensure the correctness and reliability of each action execution, we added verification and backtracking modules. Specifically, our framework comprises the following components (Figure 1 ): 1. An Instructor Model that extracts precise action plans from user demonstrations. 2. An Actor Model that follows these demonstration plans step-by-step, with built-in error tolerance and uncertainty handling modules. Although our framework relies on expert demonstrations, these demonstrations can be easily captured through quick, ad-hoc user recordings. Our agent framework is particularly beneficial in the following scenarios: 1. Novel or unintuitive UI elements : Applications frequently introduce novel UI elements that even humans find challenging to understand. Figure 2 provides an example such of UI elements. Humans typically learn from interacting with these elements, and this knowledge can be effectively transferred to agents via expert demonstrations at a minimal cost. Figure 2: An example of a non-intuitive UI AxureBoutique (2023) . 2. Long-horizon tasks requiring high accuracy at each step : For tasks involving many sequential steps, assuming the success of each step is independent: The probability of overall success ( P_{\text{success}} ) decreases exponentially with each individual step’s success probability ( p_{i} ) : P_{\text{success}}=\prod_{i=1}^{n}p_{i} Therefore, ensuring high precision in each step is critical. Our agent’s strictly following the steps from the demonstration significantly enhances the likelihood of success for individual steps and for the entire long-horizon task. 3. Tasks requiring highly specific steps : While many computer tasks are, in principle, solvable in multiple ways, user-specific configurations can cause different approaches to yield different outcomes. A personalized solution can be sometimes optimal for agent execution. For example, a user may prefer Firefox because it stores their credentials and browsing history; in such a setup, successfully placing an Amazon order may only be possible using Firefox with the default profile. Understanding these requirements beforehand are challenging for current agents, which tend to select the most common actions by default. Our demonstration-based approach enables the agent to exactly follow the user’s personalized trajectory and avoids the bias toward "common" approaches. 4. Tasks that are hard to express concisely. User often specify one’s task with a few sentence for GUI agent to execute. However, some tasks are difficult to describe in a short passage—for example, those with personalized, step-dependent details or long-horizon procedures. Although one could enumerate all particulars in a longer description, the agent can still miss details during execution. In such cases, a short demonstration recording is often the most reliable way to convey exactly what the user needs. Finally, the instruction agent frame work we propose can have broader use cases. Expert demonstrations can potentially be augmented and generalized to support similar tasks. Furthermore, demonstrations and actions can be encapsulated as reusable tools or APIs, thereby enabling broader GUI task automation. 2 Related Work Script-Based Automation Before the emergence of LLM-powered GUI agents, script-based automation methods were widely used to perform automatic computer operations Tupsakhare (2019) ; Oksanen (2023) . These scripts, however, rely heavily on static environments and tend to fail in dynamic contexts commonly encountered in modern apps and webpages. For example, script-based solutions break easily if UI elements move due to software updates, window resizing, or unexpected pop-up windows. GUI Agents Significant progress has been made on GUI agents OpenAI (2025) ; Agashe et al. (2025) ; Anthropic PBC (2024) ; Bonatti et al. (2024) . Early agent systems typically relied on structured representations (e.g., accessibility trees or HTML), whereas more recent work operates directly on visual screenshots. Architectures range from end-to-end models to modular pipelines with separate planning, grounding, and execution components. Despite these advances, substantial gaps remain relative to human-level performance, especially on complex or novel tasks with challenging interfaces or long-horizon trajectories. Typical GUI-agent pipelines comprise four primary modules Wang et al. (2025) : (1) Planning, which produces a step-by-step solution; (2) GUI Understanding (grounding), which identifies UI elements and their interactive affordances; (3) Action Decision, which selects concrete actions based on the plan; and (4) Execution, which interacts with the operating system via APIs. Among these, planning and grounding are reported to be the predominant sources of error: according to Agent-S2 Agashe et al. (2025) , planning accounts for approximately 41% and grounding for about 20.5% of failures on OSWorld. In this paper, we directly address these two bottlenecks. The human demonstration provides high-quality plan for the agent, and instruction generation provides rich and detailed hints to improve grounding accuracy. GUI Grounding GUI grounding is a critical component in GUI-based agents, as it connects the UI environment to the agent’s internal knowledge. Numerous studies have advanced

Read the complete text