LLMAID: Identifying AI Capabilities in Android Apps with LLMs
Pei Liu*, Terry Yue Zhuo, Jiawei Deng, James Thong, Shidong Pan, Sherry (Xiwei) Xu, Zhenchang Xing, Qinghua Lu, Xiaoning Du, Hongyu Zhang · 2025
Read LLMAID: Identifying AI Capabilities in Android Apps with LLMs by Pei Liu*, Terry Yue Zhuo, Jiawei Deng, James Thong, Shidong Pan, Sherry (Xiwei) Xu, Zhenchang Xing, Qinghua Lu, Xiaoning Du, Hongyu Zhang (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2511-19059.txt.
Introduction
Pei Liu and nine co-authors build LLMAID, a tool that finds and names AI features inside Android apps with a language model's help. AI now sits in much mobile software, and users and regulators ask plain questions: does this app use AI, and what kind? Past checks leaned on hand inspection and fixed rules, which cost much and age fast as methods change.
LLMAID reads app artifacts and reasons over them to flag AI capabilities and their types. The language model base lets it adapt to new AI forms that rule sets miss. Clear answers feed store labels and audits. Knowing what runs inside an app is a base step for trust.
Catalog record
| CREATOR | Pei Liu*, Terry Yue Zhuo, Jiawei Deng, James Thong, Shidong Pan, Sherry (Xiwei) Xu, Zhenchang Xing, Qinghua Lu, Xiaoning Du, Hongyu Zhang |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 51 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2511-19059
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
LLMAID: Identifying AI Capabilities in Android Apps with LLMs Pei Liu*, Terry Yue Zhuo, Jiawei Deng, James Thong, Shidong Pan, Sherry (Xiwei) Xu, Zhenchang Xing, Qinghua Lu, Xiaoning Du, Hongyu Zhang Abstract Recent advancements in artificial intelligence (AI) and its widespread integration into mobile software applications have received significant attention, highlighting the growing prominence of AI capabilities in modern software systems. However, the inherent hallucination and reliability issues of AI continue to raise persistent concerns. Consequently, application users and regulators increasingly ask critical questions such as: “Does the application incorporate AI capabilities?” and “What specific types of AI functionalities are embedded?” Preliminary efforts have been made to identify AI capabilities in mobile software; however, existing approaches mainly rely on manual inspection and rule-based heuristics. These methods are not only costly and time-consuming but also struggle to adapt advanced AI techniques. Moreover, they lack the granularity needed to accurately characterize the diverse range of AI functionalities. Specifically, existing methods often focus on specific types of information, such as AI packages and on-device models, while overlooking other potential AI factors, i.e. , APIs and HTTPS requests. As a result, these methods fail to provide a comprehensive AI service summarization of software applications. To address the limitations of existing methods, we propose LLMAID ( Large Language Model for AI Discovery ), a semantics-driven framework that combines program analysis techniques with large language models (LLMs) to identify and analyze AI services in mobile applications. LLMAID includes four main tasks: (1) candidate extraction, (2) knowledge base interaction, (3) AI capability analysis and detection, and (4) AI service summarization. We apply LLMAID to a dataset of 4,201 Android applications and demonstrate that it identifies 242% more real-world AI apps than state-of-the-art rule-based approaches. Our experiments show that LLMAID achieves high precision and recall, both exceeding 90%, in detecting AI-related components. Additionally, a user study indicates that developers find the AI service summaries generated by LLMAID to be more informative and preferable to the original app descriptions. Finally, we leverage LLMAID to perform an empirical analysis of AI capabilities across Android apps. The results reveal a strong concentration of AI functionality in computer vision (54.80%), with object detection emerging as the most common task (25.19%). ∎ 1 Introduction Advancements in artificial intelligence (AI) and their successful integration into mobile applications (apps) have enhanced the capabilities of Android apps in daily activities, such as optical character recognition, autonomous driving, object recognition and shopping recommendations. To support developers’ integration of AI capabilities, several vendors have introduced mobile AI frameworks, such as TensorFlow Lite 2 , Caffe2 6 , Core ML 32 , and ncnn 33 . These frameworks enable efficient on-device model building and inference, offering better privacy protection and reducing bandwidth usage compared to cloud-based models 18 . In recent years, there has been a surge in the development of large AI models, such as stable diffusion 10 , ChatGPT 39 , LLaMA 45 , and DeepSeek 29 . With their advanced abilities to understand and generate content, emerging app developers increasingly resort to these models via the cloud to enhance their businesses 49 ; 21 . With the widespread adoption of AI and the growing awareness of its inherent limitations, issues of AI discoverability and transparency have attracted attention from both application consumers and various regulatory bodies. To have a comprehensive understanding of AI services in software artifacts for these stakeholders, this study first tries to uncover the current status quo of AI services in real-world Android applications. In this paper, we define this process as AI Discovery , which encompasses determining whether AI capabilities are integrated or not and further elucidating the functional characteristics of the incorporated AI services. Several studies have been conducted to investigate AI techniques in smartphone apps 49 ; 52 ; 26 ; 42 ; 9 . However, these studies exhibit several notable constraints: 1) lacking attention to other potential AI components, 1 1 1 Component in the paper may refer to either high-level modules or low-level elements such as APIs and libraries, depending on context. such as integrated development packages, internal assets, and HTTP requests and API invocations in detailed implementations and integrated libraries. Ignoring these components would lead to an incomplete and insufficient analysis; 2) relying on manual inspection or pre-defined keyword searches, which require up-to-date human expertise, making them expensive and difficult to scale; 3) depending on human-defined rule-based methods, which are constrained by domain knowledge and thus ineffective for identifying AI components in diverse real-world applications. To address these aforementioned challenges, we argue that automating the human-like semantic-based AI Discovery is essential, enabling the characterization of real-world AI components by interpreting their meaningful names. Inspired by recent findings 19 ; 36 ; 27 ; 40 that large language models (LLMs) are capable of exhibiting human-like reasoning and decision-making, we explore how these models can enhance the AI Discovery process by interpreting semantically meaningful AI component names. Moreover, recent studies have shown that these LLMs can automate various software tasks, including automatic program repair 51 ; 24 ; 48 ; 20 , code generation 35 ; 7 ; 15 ; 28 , and improving software analysis efficiency and effectiveness 31 ; 25 . These findings provide evidence that LLMs may have substantial knowledge in software practice, motivating us to explore their full potential in AI Discovery within mobile applications. To this end, we propose LLMAID ( Large Language Model for AI Discovery ), the first framework that leverages large language models (LLMs) to identify AI components extracted through static program analysis and interpret their functional capabilities. The LLMAID framework consists of four key stages: (1) candidate extraction, (2) knowledge base interaction, (3) AI capability analysis and detection, and (4) AI service summarization. This structured approach allows us to perform AI discovery in an automated, step-by-step manner, significantly reducing the need for manual intervention. We implemented LLMAID using the widely adopted GPT-3.5-turbo model, which is much more cost-effective than the advanced GPT-4 model 38 . To evaluate the effectiveness of LLMAID, we conducted validation on a sample of 382 AI applications selected from a total of 56,682 already identified AI apps 26 . Manual evaluation of the results confirms that LLMAID achieves high precision and recall (exceeding 90%) in detecting AI components. Furthermore, we applied LLMAID to a dataset of 3,819 real-world Android apps published on Google Play between 2019 and 2023. Compared to the state-of-the-art (SOTA) detection approach, AI Discriminator 26 , LLMAID identifies more than twice as many AI Android apps, showcasing its superior detection capability. Our analysis reveals that the majority of detected AI components pertain to the domains of computer vision (54.80%) and data analysis (26.85%), with object detection (25.19%) and data processing (23.17%) being the most prevalent tasks. At the application level, we find that 47.44% (643/1366) of AI-enabled apps offer services related to data analysis, while 46.41% (634/1366) incorporate computer vision functionalities. We further demonstrate the practical utility of LLMAID by evidencing that developers perceive its AI service report as both informative and preferable. Our main contributions are as follows: • We present LLMAID, a prototype that automatically identifies AI components in Android apps and generates corresponding AI functionality summarizations. • We curate datasets