From Natural Language to Executable Properties for Property-based Testing of Mobile Apps

Yiheng Xiong, Ting Su, Jingling Sun, Jue Wang, Qin Li, Geguang Pu, Zhendong Su · 2026

Read From Natural Language to Executable Properties for Property-based Testing of Mobile Apps by Yiheng Xiong, Ting Su, Jingling Sun, Jue Wang, Qin Li, Geguang Pu, Zhendong Su (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2603-21263.txt.

Introduction

Xiong and co-authors turn plain language property descriptions into executable properties for testing mobile apps. Property-based testing works well, but writing the properties takes manual effort and skill. Their iPBT tool first builds an enriched widget context with multimodal models, which ties visual elements to their functions. A language model then writes framework specific executable properties. On one hundred twenty four descriptions, the tool reaches about ninety five percent accuracy. Users in a study wrote properties in about half the time.

Catalog record

CREATORYiheng Xiong, Ting Su, Jingling Sun, Jue Wang, Qin Li, Geguang Pu, Zhendong Su
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE62889
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2603.21263

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

From Natural Language to Executable Properties for Property-based Testing of Mobile Apps Yiheng Xiong, Ting Su, Jingling Sun, Jue Wang, Qin Li, Geguang Pu, Zhendong Su Abstract . Property-based testing (PBT) is a popular software testing methodology and is effective in validating the functionality of mobile applications (apps for short). However, its adoption in practice remains limited, largely due to the manual effort and technical expertise required to specify executable properties. In this paper, we propose a novel structured property synthesis approach that automatically translates property descriptions in natural language into executable properties, and implement it in a tool named iPBT . Our approach decomposes the problem into UI semantic grounding and executable property synthesis. It first builds an enriched widget context via multimodal LLMs to align visual elements with their functional semantics, and then uses an LLM with in-context learning to generate framework-specific executable properties. We evaluate iPBT with a closed-source LLM (GPT-4o) and an open-source LLM (DeepSeek-V3) on 124 diverse property descriptions derived from an existing benchmark dataset. iPBT achieves 95.2% (118/124) accuracy on both LLMs. Notably, an ablation study reveals that the enriched widget context contributes to an absolute improvement of up to 20.2% (from 75.0% to 95.2%). A user study with 10 participants demonstrates that iPBT reduces the time required to write executable properties by 56%, suggesting substantially lower manual effort. Furthermore, evaluations on 1,180 linguistically diverse variations demonstrate iPBT ’s robustness (87.6% accuracy), indicating its capability to handle varied expressions. 1. Introduction Property-based testing (PBT) has emerged as a powerful testing methodology that validates software program correctness by checking properties. Unlike example-based testing ( 11 ) which relies on specific input-output pairs to determine test outcomes, PBT systematically generates a large number of inputs to verify whether the system under test satisfies the defined properties. The pioneering PBT framework QuickCheck ( 10 ) has inspired many other PBT frameworks that successfully uncover bugs difficult to detect with traditional testing techniques across various software domains. ( 5 ; 27 ; 25 ; 24 ; 46 ; 55 ; 75 ) . Recently, several research efforts have applied PBT in testing mobile apps ( 75 ; 61 ; 62 ; 29 ) to address the oracle problem, determining whether an app’s behavior aligns with expected outcomes. In these work, users specify expected behaviors as executable properties, and then the PBT framework automatically generates GUI event sequences to validate them. For example, consider Amaze ( 1 ) , a popular file management app. A typical property is that when a user clicks on a directory ( e.g. , "Download"), the app should open that directory and display its contents, rather than triggering a file-opening dialog (which is only expected for files). Fig. 1 (a) illustrates the expected app behavior. Fig. 1 (b) shows the corresponding executable property written in Kea ( 75 ) , a recent effective PBT framework for finding functional bugs in mobile apps. The property consists of a precondition (lines 1-4) to check the presence of the file and search button; an interaction scenario (lines 7-12) simulates a click action to open the directory; and a postcondition (line 13) checks whether the path contains the directory name. Then, the PBT framework can generate a large number of GUI events to verify this property and report bugs when the property is violated. Figure 1. An example of the property in app Amaze. However, these PBT frameworks see limited adoption because testers must translate high-level intents into framework-constrained, executable properties, which demands substantial manual effort and expertise. Testers must learn framework-specific DSLs, including specialized syntax, APIs, and conventions. This imposes a steep learning curve, particularly for those without a strong programming background. Moreover, specifying executable properties requires the tedious manual inspection of the app’s view hierarchy to locate low-level UI widget identifiers ( e.g. , id="Search" ). It also requires correctly implementing complete executable properties under the constraints of the testing framework, both of which are non-trivial and error-prone in practice. Together, these barriers create a significant gap that restricts the widespread application of PBT. To bridge this gap, we enable testers to specify properties in natural language, which is more intuitive and lightweight. To reduce ambiguity while preserving accessibility, we structure property descriptions in a widely-used Hoare logic format, like Given-When-Then used in Gherkin ( 20 ) for Behavior-Driven Development ( 59 ) . Fig. 1 (c) illustrates such a structured description. While translating natural language into executable code has been studied, classic rule-based approaches ( 63 ; 48 ; 12 ) are insufficient for mobile PBT due to two main limitations. First, rule-based approaches lack flexibility. Enumerating rules to cover diverse paraphrases of user intent ( e.g. , “open the folder” vs. “navigate into the directory”) does not scale and itself incurs substantial manual effort. Second, these approaches lack semantic grounding. They struggle to accurately map high-level widget descriptions ( e.g. , “search bar”) to the low-level widget identifiers. More fundamentally, these limitations expose a deeper challenge: grounding the informal test intent into concrete executable properties. To address this challenge, we propose a structured property synthesis approach that automatically translates natural-language property descriptions into executable properties. We decompose this problem into two distinct phases: UI semantic grounding and executable property synthesis. We first tackle UI semantic grounding by extracting comprehensive GUI information ( e.g. , view hierarchies and screenshots) from the app and constructing enriched widget contexts for each UI widget. We employ Multimodal Large Language Models (MLLMs) to generate semantic annotations for each widget by jointly reasoning over visual appearance and structural information. These annotations serve as grounding signals, enabling accurate mapping between user-described widgets ( e.g. , "item name") and concrete widget identifiers ( e.g. , "line"), even when the identifiers themselves are not descriptive. Building on this grounded widget context, we then perform executable property synthesis. We leverage LLMs as inference engines to synthesize executable properties that integrate (i) the property description, (ii) enriched widget context, (iii) framework APIs, and (iv) few-shot demonstrations ( 51 ) . This design eliminates the need for handcrafted rules while enabling the model to adapt to the specific constraints of property generation. We implemented our approach as a tool named iPBT . To evaluate its effectiveness, we conduct experiments with a closed-source LLM (GPT-4o ( 44 ) ) and an open-source LLM (DeepSeek-V3 ( 13 ) ) on 124 diverse properties from the Kea benchmark ( 75 ) . iPBT correctly synthesizes executable properties for 118 out of 124 cases (95.2%) with both models. Notably, our ablation study reveals that the enriched widget context contributed to a 20.2% absolute increase in accuracy, highlighting its critical role in grounding UI semantics. Beyond accuracy, a user study demonstrated the practical utility of iPBT , reducing property authoring time by 56% compared to manual composition. We further evaluate robustness by using another LLM, Llama-3.1 ( 39 ) , to generate 1,180 diverse paraphrased variants of the original property descriptions. Under these variations, iPBT achieves 87.6% accuracy (1,034/1,180) with GPT-4o and 87.5% (1,032/1,180) with DeepSeek-V3. These results indicate that iPBT is robust to the variability of natural language property descriptions. In summary, this paper has made the following contributions: •

Read the complete text