Scaling Up: Revisiting Mining Android Sandboxes at Scale for Malware Classification
Francisco Handrick, Ismael Medeiros, Leandro Oliveira, João Calássio, Rodrigo Bonifácio, Krishna Narasimhan, Mira Mezini, Márcio Ribeiro · 2025
Read Scaling Up: Revisiting Mining Android Sandboxes at Scale for Malware Classification by Francisco Handrick, Ismael Medeiros, Leandro Oliveira, João Calássio, Rodrigo Bonifácio, Krishna Narasimhan, Mira Mezini, Márcio Ribeiro (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2505-09501.txt.
Introduction
Handrick and co-authors repeat a malware-hunting study at ten times its original scale, in this 2025 paper. The method under test runs Android apps in a sandbox to expose hidden bad behavior in repackaged apps. The first study used only about a hundred app pairs, which left its generality in doubt.
The new data set holds more than four thousand pairs of original and repackaged apps across many malware families. The team rerun the sandbox method and grade how well it sorts bad apps from good ones. A replication at this scale shows whether early promise was luck. Such checks matter because phone malware threatens privacy for billions of users.
Catalog record
| CREATOR | Francisco Handrick, Ismael Medeiros, Leandro Oliveira, João Calássio, Rodrigo Bonifácio, Krishna Narasimhan, Mira Mezini, Márcio Ribeiro |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 65 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2505-09501
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Scaling Up: Revisiting Mining Android Sandboxes at Scale for Malware Classification Francisco Handrick, Ismael Medeiros, Leandro Oliveira, João Calássio, Rodrigo Bonifácio, Krishna Narasimhan, Mira Mezini, Márcio Ribeiro Abstract The widespread use of smartphones in daily life has raised concerns about privacy and security among researchers and practitioners. Privacy issues are generally highly prevalent in mobile applications, particularly targeting the Android platform—the most popular mobile operating system. For this reason, several techniques have been proposed to identify malicious behavior in Android applications, including the Mining Android Sandbox approach (MAS approach), which aims to identify malicious behavior in repackaged Android applications (apps). However, previous empirical studies evaluated the MAS approach using a small dataset consisting of only 102 pairs of original and repackaged apps. This limitation raises questions about the external validity of their findings and whether the MAS approach can be generalized to larger datasets. To address these concerns, this paper presents the results of a replication study focused on evaluating the performance of the MAS approach regarding its capabilities of correctly classifying malware from different families. Unlike previous studies, our research employs a dataset that is an order of magnitude larger, comprising 4,076 pairs of apps covering a more diverse range of Android malware families. Surprisingly, our findings indicate a poor performance of the MAS approach for identifying malware, with the F1-score decreasing from 0.89 for the small dataset used in the previous studies to 0.54 in our more extensive dataset. Upon closer examination, we discovered that certain malware families partially account for the low accuracy of the MAS approach, which fails to classify a repackaged version of an app as malware correctly. Our findings highlight the limitations of the MAS approach, particularly when scaled, and underscore the importance of complementing it with other techniques to detect a broader range of malware effectively. This opens avenues for further discussion on addressing the blind spots that affect the accuracy of the MAS approach. I Introduction Mobile technologies, such as smartphones and tablets, have become fundamental to how we function as a society. Almost two-thirds of the world population uses mobile technologies [ 1 , 2 ] , with the Android Platform dominating this market and accounting for more than 70% of the mobile market share , with almost 2.5 million Android applications 1 1 1 In this paper, we will use the terms Android Applications, Android Apps, and Apps interchangeably, to refer to Android software applications (apps) available on the Google Play Store, in June 2023 [ 3 ] . As popularity rises, so does the risk of potential attacks, prompting collaborative efforts from both academia and industry to design and develop new techniques for identifying malicious behavior or vulnerable code in Android apps [ 4 ] . One popular class of Android malware is based on repackaging [ 5 , 6 ] , where a benign version of an app is infected with malicious code, e.g., to broadcast sensitive information to a private server [ 7 ] , and subsequently shared with users using even official app stores. The Mining Android Sandbox approach (MAS approach) was initially designed to construct sandboxes based on exploratory calls to sensitive APIs [ 8 ] . The MAS approach operates in two distinct phases. In the first phase (exploratory phase), automated test case generation tools are utilized to abstract the behavior of an app, focusing on recording calls to sensitive APIs (Application Programming Interfaces). Subsequently, during normal app execution (exploitation phase), the generated sandbox blocks any calls to sensitive APIs that were not observed during the exploratory phase. Prior studies [ 5 , 9 ] have investigated the effectiveness of the MAS approach in detecting potential malicious behavior in repackaged apps. These studies have also conducted comparisons of the approach’s performance by employing different test case generation tools during the exploratory phase, including Monkey [ 10 ] , DroidBot [ 11 ] , and Droidmate [ 12 ] —bringing evidence that DroidBot outperforms other test generation tools, uncovering many potential malicious behaviors. Nonetheless, these previous studies have two main limitations. First, they use a small dataset of malware comprising only 102 pairs of original/repackaged versions of an app---which might compromise external validity. Second, their assessments do not investigate the impact of relevant features of the repackaged apps on the accuracy of the MAS approach for malware classification, including (a) whether or not the repackaged version is a malware, (b) the similarity between the original and the repackaged versions of an app, and (c) the malware family 2 2 2 Malware families (such as gappusin , kuguo , dowgin , etc.) are often used to classify malware in groups that share similar codebases, attack methods, and objectives [ 13 ] . when the repackaged version of an app is a malware. These limitations compromise a broader understanding of the MAS approach performance. We present more details about the MAS approach and related work in Section II . To better understand the impact of these issues on previously published results, this paper presents a replication of the study conducted by Bao et al. [ 5 ] . We aim to verify the original study’s findings by executing the test case generation tool DroidBot [ 11 ] in the same settings as the original research. Unlike the original study, here we use a curated dataset of app pairs (original/repackaged versions) significantly larger than the dataset used in Bao et al.’s study. Our new dataset contains 4,076 pairs of original and repackaged apps. We present more details about the datasets, data collection, and analysis procedures in Section III . Negative result. Our study reveals a significantly lower accuracy (F1-score of 0.54 ) of the MAS approach in comparison to what the MAS approach approach performs in the small dataset (F1-score of 0.89). Since an accuracy of 0.54 is unsatisfactory for a trustworthy malware classification technique, we conduct a set of experiments to understand the reasons for the lower accuracy in our dataset. Our further assessments reveal that the MAS approach fails to correctly classify most samples from a specific set of malware families, particularly those from the gappusin family (a particular adware class frequently appearing in repackaged apps). Out of the total of 1,337 samples within this family in our large dataset, the MAS approach failed to classify 1,170 samples as malware correctly. Accordingly, these families are responsible for substantially reducing the recall of the MAS approach. We detail the results of our experiments in Section IV . We also discuss the implications and possible threats to the validity of our study in Section V and present some final remarks in Section VI . The main artifacts we produced during this research are available in the paper repository. https://github.com/droidxp/paper-ecoop-results II Background and Related Work There are many tools available that help developers reverse engineer Android bytecode [ 14 ] . For this reason, software developers can easily decompile trustworthy apps, modify their contents by inserting malicious code, repackage them with malicious payloads, and re-publish them in app stores, including official ones like the Google Play Store. It is well-known that repackaged Android apps can leverage the popularity of real apps to increase their propagation and spread malware [