Explorer : Robust Collection of Interactable GUI Elements
Iason Chaimalas, Arnas Vyšniauskas, Gabriel Brostow · 2025
Read Explorer : Robust Collection of Interactable GUI Elements by Iason Chaimalas, Arnas Vyšniauskas, Gabriel Brostow (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2504-09352.txt.
Introduction
Chaimalas, Vyšniauskas, and Brostow present Explorer, a tool that maps the clickable parts of an app's screens, in this two thousand twenty-five paper. Automating software interfaces is hard partly because gathering clean training data about each app's buttons and text fields is itself a chore.
Explorer trains on a live copy of a target app, detecting buttons and entry fields with high precision for that specific app rather than aiming at all apps at once. It also records user sessions and builds a map of screens and how they connect, which supports route planning through the interface. The team shows voice-guided navigation as one use and releases the code openly. The focus on one known app suits accessibility tools tailored to a person.
Catalog record
| CREATOR | Iason Chaimalas, Arnas Vyšniauskas, Gabriel Brostow |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 70 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2504-09352
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Explorer : Robust Collection of Interactable GUI Elements Iason Chaimalas, Arnas Vyšniauskas, Gabriel Brostow Abstract . Automation of existing Graphical User Interfaces (GUIs) is important but hard to achieve. Upstream of making the GUI user-accessible or somehow scriptable, even the data-collection to understand the original interface poses significant challenges. For example, large quantities of general UI data seem helpful for training general machine learning (ML) models, but accessibility for each person can hinge on the ML’s precision on a specific app. We therefore take the perspective that a given user needs confidence, that the relevant UI elements are being detected correctly throughout one app or digital environment. We mostly assume that the target application is known in advance, so that data collection and ML-training can be personalized for the test-time target domain. The proposed Explorer system focuses on detecting on-screen buttons and text-entry fields, i.e. interactables, where the training process has access to a live version of the application. The live application can run on almost any popular platform except iOS phones, and the collection is especially streamlined for Android phones or for desktop Chrome browsers. Explorer also enables the recording of interactive user sessions, and subsequent mapping of how such sessions overlap and sometimes loop back to similar states. We show how having such a map enables a kind of path planning through the GUI, letting a user issue audio commands to get to their destination. Critically, we are releasing our code for Explorer openly at https://github.com/varnelis/Explorer . 1. Introduction We focus on the process of collecting relevant data for AI automation of graphic user interfaces (GUIs). There is now significant appetite to build computer “bots” that can use, or at least understand, the GUIs of existing software. Getting a computer to use computers is far from just a philosophical exercise. Improved screen-readers ( Zhang et al., 2021 ) are one successful commercialized example, where blind and low-vision users benefit from an AI-driven bot as the intermediary to their laptop or phone. Additionally, there is increasing public awareness and demand for personalized GUI automators, as exemplified by the record sales of the Rabbit R1 gadget, despite its flaws ( Hollister, 2024 ) . Programming-by-Demonstration ( Cypher et al., 1993 ; wis, 2001 ) (PbD) systems seek to sufficiently understand a GUI’s widgets to be able to repeat chains of events, such as needed for automatic software testing on device farms ( Su et al., 2021 ) . We are especially motivated by downstream use-cases where someone is building or benchmarking a personalized AI assistant, or adapting existing GUIs for accessibility. There is a small but growing community of researchers and developers in this space ( Jiang et al., 2023 ; Jiang et al., 2022 ) . They train Machine Learning (ML) models on supervised data to make software capable of some form of GUI understanding. Rico ( Deka et al., 2017 ) and Wu et al.’s Never-ending UI Learner ( Wu et al., 2023a ) are exemplary proofs of concept, because they show how much can be achieved with, respectively, hand-validated or auto-crawled datasets. But they also demonstrate the acute need for the proposed Explorer , our open codebase of data-collection, self-labeling and GUI automation tools for both desktop and mobile (Android) GUIs. RICO ( Deka et al., 2017 ) is a fixed-size snapshot of GUIs from 2017. It would cost $20k to recreate and more to grow it now. Indeed, NeverEnding ( Wu et al., 2023a ) demonstrates how effective auto-collection (“crawling”) can be in the mobile iOS ecosystem. They collect labeled pairs from 6,461 distinct apps for 5000 hours. But neither the code nor the data from NeverEnding is available now. The same is true for data described by Spotlight ( Li and Li, 2023 ) . There could easily be commercial impediments keeping Apple and Google from sharing the code and data in these publications. We do not have such limitations, so our contributions are: • A suite of data-collection tools for automatically finding and labeling “clickable” elements, effective, with some differences, on Windows, Mac, and Linux desktops, and Android phones, • A model for recording, mapping, and then using application-specific user sessions. Both contributions are proposed as complementary to existing works, and our code and non-personal data are being released on GitHub and Hugging Face, on acceptance. Our code for scalable data collection builds on the Kotlin ( Kotlin, 2023 ) and Selenium ( Selenium Team, [n. d.] ) APIs. The second contribution focuses on letting users enroll and then automatically access one application deeply and reliably, with obvious trade-offs compared to automatically detecting interactables fairly well across numerous apps. We model the different states when using a GUI-based application as an interconnected graph of unique states, where user actions (e.g. text input, mouse clicks, phone taps) induce state transitions. Our Explorer ’s architecture involves ML object detection and screen similarity, to understand and actuate GUIs with no reliance on platform-specific APIs, although those were helpful for prior PbD works ( Li et al., 2017 ; Li et al., 2019 ; Riva and Kace, 2021 ; Sereshkeh et al., 2020 ; Li et al., 2010 ) . As a specific proof of concept, our Explorer enables a user to navigate a GUI with hands-free voice commands, and to replicate an end-to-end user-demonstrated GUI traversal. We evaluated the Explorer in a variety of ways, and on multiple popular computer websites, namely KhanAcademy, Wolfram Alpha, Wikipedia, and phone applications, i.e. Spotify and the Android Telephone app. Perhaps surprisingly, our experiments indicate that higher accuracy hinges on more than general-purpose large-scale data collection. 2. Related Work In the field of GUI Automation, emphasis must be placed on ease of use for the end-user, and applicability to a wide task set. Generally, while application-specific internal tools like SDKs and APIs enable automation on a per-application basis, they require considerable programming by experienced users. The same limitation applies to early systems Sikuli ( Yeh et al., 2009 ) and Prefab ( Dixon and Fogarty, 2010 ) , which detected and reverse-engineered GUI widgets (e.g. buttons, toolbars, etc.) based on exact pixel-appearance. Concurrently, Koala ( Little et al., 2007 ) and CoScripter ( Leshed et al., 2008 ) automated Firefox-based tasks executed by a user, using a browser-specific plug-in. These methods enabled a more abstracted control of the GUI environment, but still required end-user programming. Consequently, low-code ( Hirzel, 2023 ) and no-code ( Lab, 2013 ) automation solutions have arisen within the PbD paradigm, and the concept of a ‘‘demonstration’’ has advanced to a continuous video recording of the GUI screen as the user executes a specific action sequence 1 1 1 This involves keyboard or mouse events in computers, and finger-taps in phones & tablets ( Lab, 2013 ) . To detect user-actuated interactables on Android devices, VASTA ( Sereshkeh et al., 2020 ) and V2S ( Bernal-Cárdenas et al., 2020 ) used Computer Vision object detectors ( Lin et al., 2017b ; Redmon et al., 2016 ; Ren et al., 2015 ; Tian et al., 2019 ) , which is more platform-independent than early systems using pixel-detection ( Dixon and Fogarty, 2010 ; Yeh et al., 2009 )