Evaluating Large Language Models in detecting Secrets in Android Apps
Marco Alecci, Jordan Samhi, Tegawendé F. Bissyandé, and Jacques Klein · 2025
Read Evaluating Large Language Models in detecting Secrets in Android Apps by Marco Alecci, Jordan Samhi, Tegawendé F. Bissyandé, and Jacques Klein (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2510-18601.txt.
Introduction
Marco Alecci, with Jordan Samhi, Tegawendé Bissyandé, and Jacques Klein, tackles a common mobile security flaw: secret keys baked directly into Android apps. Developers hardcode passwords, tokens, and client identifiers to reach cloud services, and attackers can pull them out by reverse engineering the app. Older detectors rely on patterns, code analysis, or trained models, so each needs advance knowledge of what a secret looks like.
Their tool, SecretLoc, uses a large language model that reads context and structure to spot secrets without preset patterns or labeled training data. On a standard benchmark it found secrets every older method missed, including types never catalogued before, such as keys for AI services, access tokens, and private encryption keys. In total it uncovered 4,828 previously missed secrets and more than ten new types, and the team responsibly reported further finds in fresh apps from the Google Play store.
Catalog record
| CREATOR | Marco Alecci, Jordan Samhi, Tegawendé F. Bissyandé, and Jacques Klein |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 58 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2510-18601
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Evaluating Large Language Models in detecting Secrets in Android Apps Marco Alecci, Jordan Samhi, Tegawendé F. Bissyandé, and Jacques Klein Abstract Mobile apps often embed authentication secrets, such as API keys, tokens, and client IDs, to integrate with cloud services. However, developers often hardcode these credentials into Android apps, exposing them to extraction through reverse engineering. Once compromised, adversaries can exploit secrets to access sensitive data, manipulate resources, or abuse APIs, resulting in significant security and financial risks. Existing detection approaches, such as regex-based analysis, static analysis, and machine learning, are effective for identifying known patterns but are fundamentally limited: they require prior knowledge of credential structures, API signatures, or training data. In this paper, we propose SecretLoc, an LLM-based approach for detecting hardcoded secrets in Android apps. SecretLoc goes beyond pattern matching; it leverages contextual and structural cues to identify secrets without relying on predefined patterns or labeled training sets. Using a benchmark dataset from the literature that covers over 5000 5000 Android apps, we demonstrate that SecretLoc detects secrets missed by regex-, static-, and ML-based methods, including previously unseen types of secrets. In total, we discovered 4828 secrets that were undetected by existing approaches, discovering more than 10 ”new” types of secrets, such as OpenAI API keys, GitHub Personal Access Tokens, RSA private keys, and JWT tokens, and more. We further extend our analysis to newly crawled apps from Google Play, where we uncovered and responsibly disclosed additional hardcoded secrets. Across a set of 5000 apps, we detected secrets in 2124 apps (42.5%), several of which were confirmed and remediated by developers after we contacted them. Our results reveal a dual-use risk: if analysts can uncover these secrets with LLMs, so can attackers. This underscores the urgent need for proactive secret management and stronger mitigation practices across the mobile ecosystem. I Introduction Mobile applications (apps) increasingly depend on cloud services for features such as data storage, authentication, and many others, making their integration nearly ubiquitous. To enable such integrations, developers frequently rely on authentication secrets (e.g., API keys, access tokens, client IDs) embedded within their apps. Nevertheless, a recurring problem is that developers often hardcode these secrets directly into Android apps, leaving them vulnerable to extraction through reverse engineering, as confirmed by existing studies [ 1 , 2 , 3 ] . Once exposed, adversaries can exploit these secrets to access sensitive data, manipulate or corrupt resources, incur financial costs through unintended API usage, or cause denial of service (DoS) by exhausting invocation limits [ 1 ] , therefore representing a known vulnerability: “CWE-798: Use of hard-coded credentials” [ 4 ] . While most existing tools focus broadly on code and code-sharing platforms, Li et al. [ 1 ] recently conducted a systematic categorization of secret-detection approaches applicable to Android apps, covering more than ten existing tools [ 5 , 6 , 7 , 8 , 9 , 10 , 11 , 2 , 12 , 13 , 14 , 15 , 16 ] . The approaches were classified into three main families of techniques: ① intrinsic-value-based analysis , which relies on properties such as entropy and regex matching; ② static analysis , which traces data flows through APIs; and ③ machine-learning-based methods , which classify candidate strings based on contextual features. They then tested one approach from each category (specifically, Meli et al. [ 5 ] for intrinsic-value analysis, LeakScope [ 2 ] for static analysis, and PassFinder [ 12 ] for machine-learning-based analysis) on a dataset of 5135 Android apps, discovering 2142 secrets affecting 2115 different apps [ 1 ] . While each family of detection techniques has its own advantages, they all share a critical limitation: they require knowing exactly what to look for in advance . Regex-based detectors [ 5 ] rely on carefully crafted patterns (e.g., Google API keys with fixed structure AIza[0-9A-Za-z_]35 ) static analysis tools [ 2 ] depend on predefined API signatures, and machine-learning methods [ 12 ] require training data that may not capture novel or obfuscated secrets. As a result, all three approaches can fail to detect previously unknown or cleverly hidden authentication credentials. This observation raises a question: To what extent can large language models (LLMs) help overcome this limitation? LLMs are trained to capture both the intrinsic structure of strings and their surrounding semantic context, which may allow them to infer when a value is a secret even without prior knowledge of what to look for. In this paper, we explore this possibility by proposing SecretLoc : a novel LLM-based approach for detecting hardcoded secrets in Android apps. On the same dataset of apps analyzed by Li et al. [ 1 ] , our approach detected secrets that were previously undetected by the approaches tested in their study. We report two interesting and motivating examples: ❶ SecretLoc identified new types of secrets (such as OpenAI API keys, GitHub tokens, and RSA private keys). For example, an OpenAI API key had previously gone undetected.. This occurred for two main reasons: (i) the regex-based detection method tested in [ 5 ] did not account for the specific structure of OpenAI keys, and (ii) the OpenAI API methods were not incorporated into LeakScope [ 2 ] . This illustrates how LLM-based approaches can uncover previously unseen secrets without requiring frequent updates to detection tools or predefined key formats. ❷ Existing regex-based tools are generally effective at detecting Google API keys due to their well-defined structure (i.e., AIza[0-9A-Za-z-_]35 , as shown by Li et al. [ 1 ] ). However, these tools can fail when keys are slightly modified or obfuscated—for example, by simple Base64 encoding. In our experiments, we encountered a Base64 string ”QUl6YV*******…” (which decodes to ”AIza**…”) in an app that was completely missed by the regex-based system. In contrast, our LLM-based approach successfully identified the key, demonstrating its ability to detect API keys even when they deviate from known formats. On the benchmark of Li et al. [ 1 ] , we evaluated SecretLoc to determine to what extent our approach rediscovers previously confirmed secrets and identifies additional ones missed by existing tools. SecretLoc successfully detected 93% of the secrets reported in the benchmark and uncovered 3671 additional ones, many of which were later validated through regex checks and manual inspection. Beyond this retrospective evaluation, we applied our approach to newly crawled apps from Google Play and identified clear-text secrets in the wild. Across a set of 5000 apps, we detected secrets in 2124 apps (42.5%), several of which were confirmed and remediated by developers after we contacted them. We are continuing to monitor developer responses and provide assistance for proper remediation where needed. Our insights show that LLMs can detect previously unseen or obfuscated secrets by leveraging both string structures and surrounding context, without relying on predefined patterns or training data. This capability, combined with the widespread presence of hardcoded secrets in mobile apps, underscores their potential for defensive use while highlighting the urgent need for proper secret management to prevent real-world attacks. Contributions. This paper makes the following contributions: • We propose SecretLoc , an LLM-based approach for detecting secrets in Android apps, capable of identifying both known and previously unseen secrets. • We perform