Malware Detection based on API Calls: A Reproducibility Study

Juhani Merilehto · 2026

Read Malware Detection based on API Calls: A Reproducibility Study by Juhani Merilehto (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2601-08725.txt.

Introduction

Juhani Merilehto wrote a study that tries to repeat an older study on how to spot bad software. The older work was by a team led by Fellicious. It counts how often a program calls on the system, and it does not care about the order of those calls. A random forest model then sorts programs into safe or bad.

Merilehto uses the same public data set as the first team. The set holds a large group of training samples and a large group of test samples. He builds four forms of the model. They use single calls, pairs of calls, triples of calls, and all three mixed. A reader learns how well a known result holds up when a new person runs it again.

Catalog record

CREATORJuhani Merilehto
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE33 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2601-08725

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Malware Detection based on API Calls: A Reproducibility Study Juhani Merilehto Abstract This study independently reproduces the malware detection methodology presented by Fellicious et al. [ 7 ] , which employs order-invariant API call frequency analysis using Random Forest classification. We utilized the original public dataset (250,533 training samples, 83,511 test samples) and replicated four model variants: Unigram, Bigram, Trigram, and Combined n-gram approaches. Our reproduction successfully validated all key findings, achieving F1-scores that exceeded the original results by 0.99% to 2.57% across all models at the optimal API call length of 2,500. The Unigram model achieved F1=0.8717 (original: 0.8631), confirming its effectiveness as a lightweight malware detector. Across three independent experimental runs with different random seeds, we observed remarkably consistent results with standard deviations below 0.5%, demonstrating high reproducibility. This study validates the robustness and scientific rigor of the original methodology while confirming the practical viability of frequency-based API call analysis for malware detection. 1 Introduction The reproducibility crisis in computational science has highlighted the critical need for independent validation of research findings [ 3 ] . In machine learning and cybersecurity, where models often influence real-world security decisions, reproducibility becomes not merely an academic concern but a practical necessity. This study addresses this need by conducting an independent reproduction of the malware detection methodology presented by Fellicious et al. [ 7 ] . The original work introduced a novel approach to malware detection based on order-invariant analysis of Windows API calls. Unlike sequence-dependent methods that analyze the temporal ordering of API function invocations, their approach focuses exclusively on the frequency distribution of API calls and n-gram patterns. This design choice offers potential advantages in evasion resistance, as attackers cannot trivially bypass detection by reordering API calls while maintaining malicious functionality. The authors made their research reproducible by publicly releasing both their dataset (330,000 malware samples and 10,000 benign applications, totaling 572GB of uncompressed API call traces) on Zenodo [ 6 ] and their implementation code on GitHub. This exemplary commitment to open science enabled our independent validation. 1.1 Reproduction Goals Our reproduction study aims to: 1. Validate the core finding that order-invariant API call frequency analysis achieves high malware detection accuracy (F1 > 0.85 for optimal models) 2. Confirm the optimal configuration identified in the original work: 2,500 API calls as the threshold balancing detection accuracy and early detection capability 3. Verify the performance hierarchy among n-gram models (Unigram, Bigram, Trigram, Combined) 4. Assess the reproducibility of results across multiple independent experimental runs 5. Provide an independent baseline for future research building upon this methodology 1.2 Contributions This reproduction study makes the following contributions: • Independent validation : We successfully reproduced all key findings from the original paper, achieving performance improvements of 1–2.5% across all evaluation metrics • Reproducibility assessment : Through three independent experimental runs with different random seeds, we demonstrate that the methodology yields consistent results with standard deviation below 0.5% • Scientific confirmation : Our results validate the robustness of order-invariant, frequency-based malware detection and confirm its practical applicability • Transparent reporting : We document our complete methodology, deviations from the original approach, and detailed comparison of results to facilitate future reproductions 1.3 Paper Organization The remainder of this paper is organized as follows. Section 2 reviews related work in malware detection and reproducibility studies. Section 3 describes our reproduction methodology, including dataset preparation, feature engineering, and model training. Section 4 presents our experimental results and compares them with the original paper. Section 5 analyzes the findings, discusses reasons for performance improvements, and assesses reproducibility. Section 6 concludes with recommendations for practitioners and researchers. 2 Related Work Malware detection has evolved from signature-based approaches to sophisticated machine learning methods that analyze behavioral patterns. This section reviews relevant work in API-based malware detection, deep learning approaches, and reproducibility studies in cybersecurity. 2.1 Behavioral Malware Detection Traditional signature-based detection methods, which match known malware patterns, are ineffective against polymorphic and zero-day malware [ 17 ] . Behavioral analysis, particularly through API call monitoring, has emerged as a more robust alternative. API calls provide a rich source of behavioral information, as they represent the interface between user-space applications and the operating system kernel [ 16 ] . 2.2 API Call Sequence Analysis Several approaches have explored sequential patterns in API calls. Xu et al. [ 16 ] applied Recurrent Neural Networks (RNNs) to API call sequences, achieving promising results but requiring substantial computational resources. Pascanu et al. [ 14 ] demonstrated that LSTM networks could capture long-term dependencies in malware behavior. However, sequence-based methods are potentially vulnerable to evasion through API call reordering attacks. Graph-based methods represent another approach to capturing API call relationships. Hu et al. [ 10 ] constructed control flow graphs from API sequences, while Kirat et al. [ 13 ] developed behavior-based malware detection using graph mining. These methods offer richer structural representation but at increased computational cost. 2.3 Frequency-Based and Statistical Approaches As an alternative to sequence analysis, frequency-based methods analyze the distribution of API calls without considering temporal ordering. This approach trades some behavioral information for increased robustness against evasion. The original work we reproduce [ 7 ] follows this philosophy, hypothesizing that the presence and frequency of API calls are more fundamental indicators of malicious intent than their precise ordering. Parameter-augmented methods enhance API call analysis by incorporating function arguments and return values [ 5 ] , providing finer-grained behavioral characterization at the cost of increased feature dimensionality and potential overfitting. 2.4 Public Datasets and Benchmarks A critical challenge in malware detection research is the availability of large-scale, high-quality datasets. Several efforts have produced public benchmarks: the Drebin dataset [ 2 ] for Android malware, the EMBER dataset [ 1 ] for Windows PE files, and VirusTotal submissions [ 11 ] . However, datasets containing raw API call traces at the scale provided by Fellicious et al. [ 6 ] (330,000 malware samples with complete API sequences) remain rare, making their contribution particularly valuable for reproducibility. 2.5 Reproducibility in Machine Learning The reproducibility crisis has received increasing attention in machine learning research [ 8 , 12 ] . Factors affecting reproducibility include insufficient methodological detail, unreported hyperparameters, unavailable datasets, and stochastic training procedures [ 9 ] . Best practices include releasing code and data [ 15 ] , documenting random seeds, and reporting results across multiple runs [ 4 ] . In cybersecurity specifically, reproducibility is complicated by the sensitive nature of malware samples, legal restrictions on malware distribution, and the adversarial environment where disclosure of detection methods may enable evasion. The original paper we reproduce addresses these challenges by releasing sanitized API call traces (not raw malware binaries) on a persistent repository (Zenodo), enabling independent validation while mitigating security and legal concerns. 2.6 Positioning This Work Our reproduction study contributes to the growing body of work validating published machine learning research in cybersecurity. By independently implementing the methodology of Fellicious et al. [ 7 ] and comparing results quantitatively, we provide empirical evidence for the robustness of order-invariant, frequency-based malware detection. This validation is essential for building confidence in the approach before deployment in production security systems. 3 Reproduction Methodology This section describes our independent implementation of the methodology presented

Read the complete text