UISim: An Interactive Image-Based UI Simulator for Dynamic Mobile Environments
Jiannan Xiang, Yun Zhu, Lei Shu, Maria Wang, Lijun Yu, Gabriel Barcik, James Lyon, Srinivas Sunkara, Jindong Chen · 2025
Read UISim: An Interactive Image-Based UI Simulator for Dynamic Mobile Environments by Jiannan Xiang, Yun Zhu, Lei Shu, Maria Wang, Lijun Yu, Gabriel Barcik, James Lyon, Srinivas Sunkara, Jindong Chen (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2509-21733.txt.
Introduction
Testing mobile interfaces on physical devices is cumbersome, and static screenshot analysis cannot capture dynamics. Jiannan Xiang and 8 co-authors introduce an image based UI simulator in a 2025 paper. UISim explores mobile phone environments purely from screen images. Given an initial screen and a user action, it predicts what comes next. A two stage method drives the simulation.
First, the abstract layout of the next UI state is predicted. Then a new, visually consistent image is synthesized from that layout. Realistic UI transitions follow without a device farm. Immediate benefits include UI testing, rapid prototyping, and synthetic data generation. Training AI agents to interact with interfaces is a further aim. Dynamic, diverse mobile environments motivate the design.
Catalog record
| CREATOR | Jiannan Xiang, Yun Zhu, Lei Shu, Maria Wang, Lijun Yu, Gabriel Barcik, James Lyon, Srinivas Sunkara, Jindong Chen |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 26 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2509-21733
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
UISim: An Interactive Image-Based UI Simulator for Dynamic Mobile Environments Jiannan Xiang, Yun Zhu, Lei Shu, Maria Wang, Lijun Yu, Gabriel Barcik, James Lyon, Srinivas Sunkara, Jindong Chen Abstract Developing and testing user interfaces (UIs) and training AI agents to interact with them are challenging due to the dynamic and diverse nature of real-world mobile environments. Existing methods often rely on cumbersome physical devices or limited static analysis of screenshots, which hinders scalable testing and the development of intelligent UI agents. We introduce UISim, a novel image-based UI simulator that offers a dynamic and interactive platform for exploring mobile phone environments purely from screen images. Our system employs a two-stage method: given an initial phone screen image and a user action, it first predicts the abstract layout of the next UI state, then synthesizes a new, visually consistent image based on this predicted layout. This approach enables the realistic simulation of UI transitions. UISim provides immediate practical benefits for UI testing, rapid prototyping, and synthetic data generation. Furthermore, its interactive capabilities pave the way for advanced applications, such as UI navigation task planning for AI agents. Our experimental results show that UISim outperforms end-to-end UI generation baselines in generating realistic and coherent subsequent UI states, highlighting its fidelity and potential to streamline UI development and enhance AI agent training. 2 2 footnotetext: A demonstration video is available at https://youtu.be/QZOhjFQRZxs . 1 Introduction Developing and testing user interfaces (UIs), particularly for mobile devices, is a complex and often resource-intensive process. Traditional methods rely heavily on manual screen captures and scripted interactions performed on physical devices or emulators ( Stone et al. 2005 ; Sridevi 2014 ; Thimbleby 1990 ; Carroll et al. 1988 ) . These methods are time-consuming, inflexible and do not scale well for comprehensive testing or agent training. Real-world UI environments are inherently dynamic and diverse, presenting significant challenges for automated analysis, rapid prototyping, and the development of robust UI navigation agents. As such, the ability to efficiently simulate these environments, allowing for interactive exploration and generation of plausible next states, is crucial for advancing both UI development and AI-driven human-computer interaction. Table 1: A comparative analysis of UI simulation and generation paradigms. This table evaluates various methods against critical features including dynamic state simulation, the ability to exert fine-grained structural control, and overall scalability. UISim (Ours) is demonstrated to address the combined challenges that other approaches face, offering a comprehensive solution for interactive mobile UI environments. Methods Dynamic State Simulation Fine-Grained Structural Control Scability Physical Emulator ✓ ✓ ✗ High-Level UI Spec Generation ✗ ✓ ✓ Static Screenshot Analysis ✗ ✗ ✓ End-to-End Image Generation ✓ ✗ ✓ UISim ( Ours ) ✓ ✓ ✓ Existing approaches to UI simulation and automation often fall short in critical areas, as shown in Table 1 . Some methods focus on generating UIs from high-level specifications ( Usman et al. 2014 ; Usman et al. 2020 ; Son et al. 2013 ; Li et al. 2022 ; Chen et al. 2024 ) , which limits their ability to accurately replicate the visual nuances of real-world applications. Others rely on static screen captures ( Wu et al. 2021 ; Beltramelli 2018 ) , preventing dynamic interaction and exploration of subsequent states. While recent advances in image generation ( Podell et al. 2023 ; Betker et al. 2023 ) offer promising directions, end-to-end approaches often struggle with the precise control needed for coherent UI state transitions and often produce less realistic visual output. An interactive, image-based simulator that can dynamically respond to user actions and generate high-fidelity, plausible next-UI screens remains a significant challenge, especially for complex real-world UIs. Such a system is crucial for enabling rapid prototyping, scalable testing of UI designs, and critically, for fostering the development of intelligent agents that can learn to navigate complex digital environments through rich visual feedback. To address these pressing limitations, we introduce UISim, a novel image-based UI simulator that enables dynamic and interactive simulation of mobile phone environments purely from screen images. Our system uniquely provides a flexible and scalable platform for exploring UI states and actions without requiring direct access to a physical device or a complex rendering engine. Given an initial phone screen image and a user action (e.g., open the email app), UISim employs a robust two-stage method to generate the subsequent UI state: it first predicts abstract layout information describing the possible next screen based on the input action, then synthesizes a new, visually consistent UI image based on this predicted layout. The decoupled design breaks down the complex problem of image-to-image UI transformation into more manageable sub-problems, offering fine-grained control over the content and structure of the simulated UI, inherently leading to higher fidelity and more diverse generation capabilities compared to end-to-end image generation methods. Our experimental results show that UISim outperforms baselines by 36.73 on Fréchet Inception Distance, demonstrating its superiority in generating realistic and coherent subsequent UI states. UISim delivers immediate and substantial practical benefits across multiple domains. For developers and designers, it enables rapid prototyping and UI testing, allowing quick iteration on designs, visualization of user flows, and observation of application behavior under various simulated user interactions, drastically reducing the need for cumbersome manual device testing. For researchers, it facilitates the generation of vast amounts of synthetic UI data, crucial for training data-hungry machine learning models for UI analysis and automation, and evaluating UI automation in a reproducible environment. Furthermore, the simulator’s inherent capability to generate plausible next UI states based on arbitrary user actions is important for advanced applications such as UI navigation task planning. By providing a ”look-ahead” mechanism for AI agents to explore potential interaction sequences and their visual outcomes, UISim can facilitate the development of more intelligent and adaptable agents capable of achieving complex, high-level goals like ”send an email” or ”book a flight” across diverse and unfamiliar applications. We believe this interactive, image-based simulator represents a crucial step towards more efficient and scalable development of both UI-centric applications and general-purpose AI agents interacting with human-designed interfaces. 2 Related Work UI Simulation and Automation User interface (UI) development and testing have traditionally relied on either manual procedures or tooling that requires direct access to application source code or runtime environments. Tools like Appium ( Appium Contributors 2024 ) , UIAutomator ( Android Open Source Project 2024 ) , and the Android Studio Emulator ( Android Developers 2024 ) support scripted interaction and visual inspection but are limited by their dependency on physical or emulated devices and their inability to simulate arbitrary UI states at scale. To address some of these limitations, model-driven UI generation approaches such as Model-Based Testing of GUI Applications and Automated UI Generation from Task Models ( Usman et al. 2014 ; Usman et al. 2020 ; Son et al. 2013 ; Li et al. 2022 ; Chen et al. 2024 ) use high-level specifications to produce static UI layouts. However, these methods focus on synthesis rather than simulation and lack support for visually grounded state transitions or dynamic user interaction. Other recent works aim to understand or reconstruct UIs from screen images. For example, Beltramelli 2018 and Wu et al. 2021 attempt to infer UI structures from