Machine Learning Transferability for Malware Detection

César Vieira, João Vitorino, Eva Maia, Isabel Praça · 2026

Read Machine Learning Transferability for Malware Detection by César Vieira, João Vitorino, Eva Maia, Isabel Praça (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2603-26632.txt.

Introduction

Vieira, Vitorino, Maia, and Praça test how well malware detectors transfer across datasets with mismatched features. Public malware datasets rarely share the same feature sets, which blocks reuse of trained models. Their pipeline unifies EMBERv2 features and trains paired models on two data mixes. The models are then scored on TRITIUM, INFERNO, and the large SOREL-20M set. The study weighs which preprocessing choices keep detection strong when the data shifts, a key need as obfuscation keeps evolving.

Catalog record

CREATORCésar Vieira, João Vitorino, Eva Maia, Isabel Praça
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE28057
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2603.26632

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Machine Learning Transferability for Malware Detection César Vieira, João Vitorino, Eva Maia, Isabel Praça Abstract Malware continues to be a predominant operational risk for organizations, especially when obfuscation techniques are used to evade detection. Despite the ongoing efforts in the development of Machine Learning (ML) detection approaches, there is still a lack of feature compatibility in public datasets. This limits generalization when facing distribution shifts, as well as transferability to different datasets. This study evaluates the suitability of different data preprocessing approaches for the detection of Portable Executable (PE) files with ML models. The preprocessing pipeline unifies EMBERv2 (2,381-dim) features datasets, trains paired models under two training setups: EMBER + BODMAS (EB) and EMBER + BODMAS + ERMDS (EBR) . Regarding model evaluation, both EB and EBR models are tested against TRITIUM, INFERNO and SOREL-20M. ERMDS is also used for testing for the EB setup. The findings indicate that compact boosting static detectors are applicable for on-host use, but require a careful analysis of how PE obfuscation techniques affect the feature distributions of training datasets and during model inference. 1 Introduction Malware is defined as software that intentionally compromises the confidentiality, integrity, or availability of information systems, thereby enabling attackers to engage in extortion, disruption and espionage activities [ 19 ] . From 2023 to 2025, the European Union Agency for Cybersecurity (ENISA) identified the Public Administration as the most frequently targeted by malware within the European Union (EU) , with ransomware and data-related intrusions as the main attack vectors [ 6 ] , [ 7 ] . These intrusions are indicative of trends that have been observed in the context of specialized attack ecosystems, which include Ransomware-as-a-Service (RaaS) and access-brokered access [ 8 ] . In order to prevent the execution of malicious programs, organizations can make use of both signature and anomaly-based approaches. Signature-based detection approaches are capable of achieving high operational speeds and low False Positive Rate (FPR) by comparing the hash of a given binary with a database of known malware sample hashes or matching file content against a set of known byte or string patterns, such as YARA rules [ 13 ] . On the other hand, anomaly-based detection approaches increasingly rely on Machine Learning (ML) to learn a profile of benign host behavior or a discriminative boundary from static features, which requires hyperparameter optimization. Methods include supervised, semi-supervised and unsupervised families, ranging from gradient-boosted trees and feed-forward networks to one-class and reconstruction-based approaches [ 9 ] . However, despite the growing usage of ML in anomaly-based approaches, their application to real-world Portable Executable (PE) files still faces challenges. The limited compatibility and standardization of the utilized features across public datasets hinders the reproducibility and transferability of detection pipelines [ 15 ] . Moreover, models trained on one dataset may generalize poorly to others due to distributional mismatch and evolving attacker tooling, which may cause concept drift over time, requiring periodic model retraining and recalibration of feature distributions and data preprocessing approaches [ 4 ] . This study evaluates different data preprocessing approaches for the detection of PE files with ML models. A static detection pipeline is built on EMBER -v2 features with two training setups: EMBER + BODMAS and EMBER + BODMAS + ERMDS . Principal Component Analysis (PCA) or XGBoost Feature Selection (XGBFS) are then applied to the unified datasets, producing feature vectors of 128/256/384 dimensions where FLAML tuned model pairs ( LightGBM , XGBoost , Extra Trees , Random Forest ) are then trained against the aforementioned reduced vector dimensions. Evaluation covers in-distribution test splits and cross-dataset inference on SOREL-20M , TRITIUM, INFERNO (and ERMDS when withheld), reporting F1, AUC and TPR @1%/0.1% FPR to assess generalization, obfuscation robustness and drift sensitivity. This paper is divided into multiple sections. Section 2 reviews existing work regarding malware detection. Section 3 describes the workflow of the study. Section 4 presents and discusses the obtained results. Finally, section 5 summarizes the main findings and outlines future work on generalization and concept drift. 2 Related Work To choose the most relevant datasets and data preprocessing approaches, it is important to analyze recent literature regarding static Windows malware detection. Table 1 provides a summary of relevant scientific papers, covering the key milestones and more recent developments. Historically, signature-based detection on Windows has relied on deterministic content rules that require minimal computation. However, such systems are vulnerable to semantics-preserving changes, packing, polymorphism and other forms of obfuscation, which limit the coverage of novel and zero-day malware. Motivated by these limitations, Schultz et al. [ 22 ] and Kolter & Maloof [ 12 ] proposed approaches that relied on the extraction of features such as imported functions, strings and byte n-grams, which were then processed by ML models like Naïve Bayes (NB) and boosted decision trees. Subsequently, Saxe & Berlin [ 21 ] replaced manual feature selection with automatic representations derived from two-dimensional byte-entropy histograms of entire binaries. This approach captured spatial correlations and entropy variations within executable content, improving scalability, robustness to obfuscation and higher classification accuracy compared to manually engineered n-gram features, marking an early transition toward deep-learning static malware analysis. Table 1: Overview of static malware detection approaches. Year Paper Data Approach Metrics 2001 Schultz et al. [ 22 ] Proprietary PE Imports/strings/byte n-grams \to NB, boosted trees TPR 0.97; FPR 0.04; Acc 97.1% 2004 Kolter & Maloof [ 12 ] Proprietary PE Byte n-grams \to Boosted J48 AUC 0.9958 2015 Wang et al. [ 24 ] BIG 2015 Ensemble (RF & GBM) on byte/ASM features 0.0028 Log Loss 2015 Saxe & Berlin [ 21 ] Proprietary PE CNN on 2D byte–entropy images \uparrow vs. engineered n-grams 2018 Raff et al. [ 16 ] Windows PE corpora 1D CNN on raw bytes Acc 0.940 & AUC 0.981 2018 Anderson & Roth [ 2 ] EMBER 2017/2018 LGBM AUC \approx 0.999; 92.9% TPR @0.1% FPR 2019 Rudd et al. [ 20 ] Proprietary PE FFNN with auxiliary losses \Delta FNR -42.6\% @ 10^{-3} ; -53.8\% @ 10^{-5} 2020 Harang & Rudd [ 10 ] SOREL-20M LightGBM ; multi-task FFNN (ALOHA-style) AUC \approx 0.998; TPR > 90%@0.1% FPR 2021 Rahali et al. [ 17 ] EMBER Cost-sensitive LGBM (custom logistic loss) \downarrow error vs. LGBM 2023 Austin et al. [ 3 ] EMBER , SOREL-20M AutoML-tuned FFNN (static & online settings) 0.956 TPR @0.1% ( EMBER ); 0.965 ( SOREL-20M ) 2023 Manikandaraja et al. [ 14 ] TRITIUM, INFERNO EMBER , BODMAS & SOREL classifiers TRITIUM F1/ AUC : 0.904/0.913; INFERNO: 0.959/0.961 2025 Choudhary et al. [ 5 ] EMBER , BODMAS, ERMDS Dim. reduction ( PCA / XGBoost ) + two-instance soft-vote ( RF / ET / LightGBM / XGBoost ) F1 \approx 0.9773; AUC \approx 0.9958 The earliest major benchmark for static malware classification was Microsoft BIG 2015 [ 18 ] , which provided over 500 GB of labeled Windows Portable Executable binaries grouped into nine malware families. The competition instigated the first comparative studies between traditional feature-engineered models and emerging deep-learning methods for family classification. The solution that was found to be most effective by Wang et al. [ 24 ] used an ensemble of Random Forest (RF) and Gradient Boosting on engineered byte

Read the complete text