VLM-Fuzz : Vision Language Model Assisted Recursive Depth-first Search Exploration for Effective UI Testing of Android Apps

Biniam Fisseha Demissie, Yan Naing Tun, Lwin Khin Shar, Mariano Ceccato · 2025

Read VLM-Fuzz : Vision Language Model Assisted Recursive Depth-first Search Exploration for Effective UI Testing of Android Apps by Biniam Fisseha Demissie, Yan Naing Tun, Lwin Khin Shar, Mariano Ceccato (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2504-11675.txt.

Introduction

Biniam Fisseha Demissie and co-authors present a 2025 computing paper on automated testing of Android apps. Good testing must explore many app states by simulating taps and system events. Reaching high code coverage is hard because screens are complex, widgets need exact inputs, and testers get stuck looping on one screen.

Their tool, VLM-Fuzz, explores an app with a depth-first search guided by heuristics. Static analysis first pulls the app's components and interactive widgets from its files. A vision language model is then called on demand to read tricky layouts and pop-ups. Hints from both sources, plus the current screen state, steer the next move. The design aims to break out of loops and cover more code than prior fuzzers.

Catalog record

CREATORBiniam Fisseha Demissie, Yan Naing Tun, Lwin Khin Shar, Mariano Ceccato
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE73 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2504-11675

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

VLM-Fuzz : Vision Language Model Assisted Recursive Depth-first Search Exploration for Effective UI Testing of Android Apps Biniam Fisseha Demissie, Yan Naing Tun, Lwin Khin Shar, Mariano Ceccato Abstract Testing Android apps effectively requires a systematic exploration of the app’s possible states by simulating user interactions and system events. While existing approaches have proposed several fuzzing techniques to generate various text inputs and trigger user and system events for UI state exploration, achieving high code coverage remains a significant challenge in Android app testing. The main challenges are (1) reasoning about the complex and dynamic layout of UI screens; (2) generating required inputs/events to deal with certain widgets like pop-ups; and (3) coordination between current test inputs and previous inputs to avoid getting stuck in the same UI screen without improving test coverage. To address these problems, we propose a novel, automated fuzzing approach called VLM-Fuzz for effective UI testing of Android apps. We present a novel heuristic-based depth-first search (DFS) exploration algorithm, assisted with a vision language model (VLM), to effectively explore the UI states of the app. We use static analysis to analyze the Android Manifest file and the runtime UI hierarchy XML to extract the list of components, intent-filters and interactive UI widgets. VLM is used to reason about complex UI layout and widgets on an on-demand basis. Based on the inputs from static analysis, VLM, and the current UI state, we use some heuristics to deal with the above-mentioned challenges. We evaluated VLM-Fuzz based on a benchmark containing 59 apps obtained from a recent work and compared it against two state-of-the-art approaches: APE and DeepGUI . VLM-Fuzz outperforms the best baseline by 9.0%, 3.7%, and 2.1% in terms of class coverage, method coverage, and line coverage, respectively. We also ran VLM-Fuzz on 80 recent Google Play apps (i.e., updated in 2024). VLM-Fuzz detected 208 unique crashes in 24 apps, which have been reported to respective developers. 1 Introduction Mobile apps have become an integral part of our daily lives. While developers must ensure app quality, they often work under time-to-market pressure, primarily focusing their testing on core functionalities and common usage scenarios. This often leads to released apps containing bugs or vulnerabilities which may severely impact user experiences and thus, the app’s popularity. To deal with inadequate testing, automatic Android fuzzing has been an active area of research in software engineering community for the past decade. Existing GUI testing approaches can be categorized based on their exploration strategies such as random Android (2024a) , search-based Amalfitano et al. (2012) ; Amalfitano et al. (2014) ; Azim and Neamtiu (2013) ; Mao et al. (2016) , model-based Su et al. (2017) ; Dong et al. (2020) , or program analysis based Anand et al. (2012) ; Moran et al. (2017) approaches. But these approaches still have limitations in terms of (1) dealing with the evolving nature of Android platform; (2) scalability of the test generators when dealing with a large state space; and (3) low code coverage when dealing with complex interactions with the user, the system, and other apps because test inputs generated by these approaches do not resemble real users’ actions Peng et al. (2022) . To deal with these challenges, some approaches have incorporated machine learning techniques such as deep learning YazdaniBanafsheDaragh and Malek (2021) and reinforcement learning Pan et al. (2020) ; Romdhana et al. (2022) . However, deep learning or reinforcement learning approaches generally require large amount of labeled training data and it may not be scalable to label a representative list of GUI items. Large Language Models (LLMs) such as GPT and Gemini have emerged as a powerful tool for natural language understanding and image recognition. Recent advances in LLM have triggered various studies examining the use of these models for software engineering tasks such as code completion Du et al. (2024) and test generation Ryan et al. (2024) . As such, recent approaches have applied large language models for GUI testing of Android apps Feng and Chen (2024) ; Liu et al. (2024a) . These approaches essentially formulate the GUI testing problem as a questions & answering task, i.e., asking the LLM to play the role of a human tester to test the target app, with various text prompting techniques. While these approaches may be effective, by leveraging only text-based prompting techniques, they have not explored the full capability of LLMs where they can be leveraged to reason with images such as screenshots of activity component screens. They have also not explored the synergistic combination of search/model-based state exploration strategies with LLMs. While LLMs may be good at generating test inputs for a given GUI, a good state space exploration strategy is still needed to keep track of explored and unexplored states and to identify promising paths that could lead to better code coverage. Our goal here is to address the low code coverage limitation highlighted in Peng et al. (2022) . The main challenges we aim to address in this paper are 1 reasoning about complex and dynamic layout of UI screens; 2 generating required inputs/events to deal with certain widgets like pop-ups; and 3 coordination between current test inputs and previous inputs to avoid getting stuck in the same UI screen without improving test coverage. We propose VLM-Fuzz , a novel automated fuzzing approach for effective UI testing of Android apps. The approach first analyzes the AndroidManifest file to identify app components and relevant system events for each component, and it examines the dynamic UI hierarchy XML to capture interactive UI widgets. To optimize testing efficiency, VLM-Fuzz evaluates component complexity based on the number of interactive UI elements and allocates testing time budget accordingly. It then generates appropriate inputs/events to different widgets leveraging both heuristics and Vision Language Model (VLM). It adopts a heuristic-based recursive depth-first search (DFS) exploration strategy to explore different UI states of the app. The contributions of this paper are A. Novel Android app fuzzing algorithm : We propose an algorithm that combines a heuristic-based recursive depth-first search exploration strategy with Vision Language Model, which addresses the challenges involved in reasoning with complex, dynamic UI layouts and exploring the state space in an efficient manner. B. Comparative study with state of the arts (SOTAs) : we compare our approach against APE Gu et al. (2019) and DeepGUI YazdaniBanafsheDaragh and Malek (2021) — based on a benchmark consisting of 59 apps. Our approach achieves a high code coverage (68.5 % class coverage, 53.2 % method coverage, and 46.5 % line coverage on average), outperforming the best baseline by 9.0%, 3.7%, and 2.1% in terms of class coverage, method coverage, and line coverage, respectively. We also compare our approach against GPTDroid Liu et al. (2024a) , a recent and closely related tool, analytically (as its tool was unavailable despite our requests). We were able to replicate it following the steps given in the article, which are essentially iterative promptings of GUI information. C. Ablation study on the usefulness of vision LM : We investigate the VLM’s ability to reason with GUI objects and how useful it is in improving the test coverage. D. Bug detection capability on real world apps : We further evaluate our approach

Read the complete text