Beyond Chat and Clicks: GUI Agents for In-Situ Assistance via Live Interface Transformation CCS: Human-centered computing Graphical user interfaces CCS: Human-centered computing Web-based interaction CCS: Human-centered computing Interactive systems and tools

Pan Hao, Rishi Selvakumaran, Jacob Sun, Qianwen Wang · 2026

Read Beyond Chat and Clicks: GUI Agents for In-Situ Assistance via Live Interface Transformation CCS: Human-centered computing Graphical user interfaces CCS: Human-centered computing Web-based interaction CCS: Human-centered computing Interactive systems and tools by Pan Hao, Rishi Selvakumaran, Jacob Sun, Qianwen Wang (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2604-14668.txt.

Introduction

Hao, Selvakumaran, Sun, and Wang propose in-situ assistance inside live web pages. It works through lightweight DOM changes, without rebuilding the app or its logic. Their design space and flow show agents adding, changing, or redoing items to ease grasp, use, and moving. DOMSteer, a Chrome add-on, reads help requests and live context, grounding them to items. It reversibly tweaks page items: tooltips, marks, and layout reorder. Tests on two complex pages show reliable, quick help, and a user study against ChatGPTAtlas confirms both.

Catalog record

CREATORPan Hao, Rishi Selvakumaran, Jacob Sun, Qianwen Wang
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE60812
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2604.14668

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Beyond Chat and Clicks: GUI Agents for In-Situ Assistance via Live Interface Transformation CCS: Human-centered computing Graphical user interfaces CCS: Human-centered computing Web-based interaction CCS: Human-centered computing Interactive systems and tools Pan Hao, Rishi Selvakumaran, Jacob Sun, Qianwen Wang Abstract . Complex visual interfaces are powerful yet have a steep learning curve, as users must navigate feature-rich visual interfaces while reasoning about domain-specific operations. Existing approaches either deliver assistance through a separate chat-based interaction, or require substantial application-specific engineering to build support natively into each interface. To address the gaps, we propose in-situ assistance : a mode of support delivered directly within any live web interface through lightweight, browser-level interventions on the Document Object Model (DOM), without rebuilding the application or modifying its underlying logic. We contribute a design space and a computational pipeline for DOM-mediated in-situ assistance, characterizing how GUI agents can insert, mutate, or recompose web elements to make the interface easier for users to understand, use, and navigate. We instantiate in-situ assistance in DOMSteer , a Chrome extension that interprets a user’s help request and live interface context, grounds it to relevant UI elements, and executes reversible DOM manipulations directly on the live page to deliver assistance, including contextual tooltips, control highlighting, layout reorganization. Quantitative evaluations on two complex visual interfaces show that DOMSteer delivers reliable and efficient in-situ assistance. Use cases and a comparative user study with ChatGPTAtlas demonstrate the usability and effectiveness of DOMSteer . Altogether, these findings point to a broader role for GUI agents: not just assisting from the sidelines, but actively reconfiguring live interfaces to support users in the moment. Figure 1. DOMSteer delivers assistance directly within the live interface through three DOM operations: (A) Mutate that adapts existing elements, (B) Insert that adds new contextual content, and (C) Recompose that reorganizes interface structure. 1. Introduction Graphical user interfaces (GUIs) have long been the dominant medium for understanding and interacting with data, ranging from chart creation ( 49 ) , visual analytics ( 5 ) , to communicative data dashboard ( 46 ) . Many such tools have become de facto standards within their respective domains, deeply embedded in professional workflows, such as Autodesk in engineering design, Tableau in business analytics, and Epic in clinical healthcare. However, the complexity of such interfaces is inherently coupled to the scale and complexity of the data they are designed to support. As data grows in scale and complexity, these interfaces tend to expose users to dense, hierarchically organized visual components and interactions, resulting in steep learning curves that can overwhelm novices and reduce working efficiency. Much effort has been devoted to mitigating these challenges, spanning interface tutorials and onboarding experiences ( 46 ; 39 ) , and adaptive interfaces that adjust layouts, widget types, and toolbars based on user tasks, abilities, and usage patterns such as recency and frequency ( 12 ; 44 ; 11 ; 15 ; 14 ; 48 ) . While these approaches offer valuable improvements, they are largely grounded in design heuristics or limited to predefined adaptation rules, constraining their ability to generalize to the diverse, context-dependent needs of real-world users at scale. Recent advances in large language models (LLMs) have created new opportunities to address these challenges. One line of work augments GUIs with a separate conversational panel that provides on-demand guidance ( 47 ; 23 ; 6 ) . However, users often struggle to articulate the full interface context in language ( 53 ) , and must further translate the system’s response back into the spatial and interactive structure of the GUI. Recent agentic systems, such as Gemini in Chrome ( 16 ) and ChatGPT Atlas ( 32 ) , can respond to user requests through low-level interface actions such as clicking, typing, and scrolling ( 21 ) . However, their performance degrades on complex professional environments involving careful decision-making (lower than 20% accuracy for domain specific workflows ( 41 ) ). Another line of work uses LLMs to dynamically generate task-specific interfaces tailored to user needs. For example, DynaVis ( 45 ) generates interactive widgets for visualization editing based on natural language. BISCUIT ( 7 ) creates ephemeral UI in computational notebook to understand and steer codes. However, these generative interfaces often require constructing new interactive systems from scratch ( e.g. , rebuilding the connections to underlying data and reimplementing interaction logic). This imposes substantial engineering overhead, underutilizes the mature interfaces already widely adopted in practice, and often assumes access to underlying data that may not always be available. To address the gaps, we propose in-situ assistance that helps users navigate complex visual interfaces. By in-situ assistance , we refer to assistance delivered directly within the existing web-based interface that the user is actively working with, rather than through a separate interface or by rebuilding the application. We enable this capability through browser-level interventions on the Document Object Model (DOM) ( 50 ) , allowing a client-side layer to add, modify, and interact with existing interface components without access or changing server-side logic. To operationalize this idea, we propose a design space and a computational pipeline for in-situ assistance and then develop DOMSteer , a Chrome browser extension that delivers such assistance directly within live interfaces. We evaluate DOMSteer on 101 annotated challenges across two real-world interfaces, DataVoyager and TensorFlow Playground. Our handbook-based retrieval achieves comparable assistance quality to direct LLM generation while reducing latency by 11.8 \times . A within-subject user study ( N{=}12 ) further shows that DOMSteer reduces task completion time by 25% and improves task accuracy to 100%, compared with 91.7% for a chat-based baseline. Main contributions of this paper include 1 1 1 Project website and artifacts are available at https://visual-intelligence-umn.github.io/DOMSteer/ . : • A design space for in-situ assistance in web-based GUIs via direct DOM manipulation. • A computational pipeline that efficiently and accurately translates user needs into live interface transformation. • DOMSteer , a Chrome Browser extension that operationalizes this pipeline to deliver in-situ assistance for any web-based interfaces without needing access to their source code. • A mixed-method evaluation of the proposed approach, , combining quantitative experiments with user studies to examine the effectiveness and usability of DOMSteer . 2. Related Work 2.1. Adaptive and Dynamic User Interfaces Adaptive user interfaces have long been a long-pursuit goal in human–computer interaction, aiming to improve usability and task performance by tailoring interaction to users, tasks, and context ( 29 ; 18 ; 22 ) . Prior work has explored various ways to design adaptive interfaces, such as adaptive layouts ( e.g. , grouping, ordering) and widget types to match users’ abilities or interaction histories ( 12 ; 44 ; 48 ) , adaptive menus and toolbars that promote, reorder, or highlight options based on usage patterns ( e.g. , recency, frequency) ( 11 ; 15 ) and dynamic control size and placement to reduce motor effort ( 13 ) . More recently, LLM-powered systems have extended adaptive UI beyond rearranging existing controls, synthesizing task-specific, ephemeral elements from natural-language requests and interface context on demand ( 45 ; 7 ) , such as DynaVis ( 45 ) adds interactive widgets for users to refine a visualization, BISCUIT ( 7 ) creates ephemeral UI

Read the complete text