Demystifying the Role of Rule-based Detection in AI Systems for Windows Malware Detection
Andrea Ponte 1 , Luca Demetrio 1 , Luca Oneto 1 , Ivan Tesfai Ogbu 3 , Battista Biggio 2 , Fabio Roli 1 2 · 2025
Read Demystifying the Role of Rule-based Detection in AI Systems for Windows Malware Detection by Andrea Ponte 1 , Luca Demetrio 1 , Luca Oneto 1 , Ivan Tesfai Ogbu 3 , Battista Biggio 2 , Fabio Roli 1 2 (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2508-09652.txt.
Introduction
Ponte and colleagues wrote in twenty twenty-five about how rule-based checks shape AI systems that hunt Windows malware. Many systems pair fixed signatures with machine learning models. The two parts are usually built apart and only joined at the end. The team instead trained the learning part only on files that signatures had missed. They asked whether that change makes the whole system stronger or weaker.
Models trained this way resisted crafted files meant to fool them. They also held up better as malware changed over time. The cost was a fixed floor of false alarms from poorly chosen rules. The authors discuss those limits frankly. They suggest adding behavior-based analysis next to harden the system further.
Catalog record
| CREATOR | Andrea Ponte 1 , Luca Demetrio 1 , Luca Oneto 1 , Ivan Tesfai Ogbu 3 , Battista Biggio 2 , Fabio Roli 1 2 |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 28 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2508-09652
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Demystifying the Role of Rule-based Detection in AI Systems for Windows Malware Detection Andrea Ponte 1 , Luca Demetrio 1 , Luca Oneto 1 , Ivan Tesfai Ogbu 3 , Battista Biggio 2 , Fabio Roli 1 2 Abstract Malware detection increasingly relies on AI systems that integrate signature-based detection with machine learning. However, these components are typically developed and combined in isolation, missing opportunities to reduce data complexity and strengthen defenses against adversarial EXEmples, carefully crafted programs designed to evade detection. Hence, in this work we investigate the influence that signature-based detection exerts on model training, when they are included inside the training pipeline. Specifically, we compare models trained on a comprehensive dataset with an AI system whose machine learning component is trained solely on samples not already flagged by signatures. Our results demonstrate improved robustness to both adversarial EXEmples and temporal data drift, although this comes at the cost of a fixed lower bound on false positives, driven by suboptimal rule selection. We conclude by discussing these limitations and outlining how future research could extend AI-based malware detection to include dynamic analysis, thereby further enhancing system resilience. I Introduction To increase the likelihood of detecting the always-evolving variants of malware, malware detectors are enriched with several layers of detection, both relying on pattern matching and machine learning (ML) components. We name such combination as AI systems , aligned with the latest directives of the European Union through the EU AI Act aiact . This is consistently reported as the standard by companies that sell antivirus (AV) programs eset ; avira ; windefender ; macos , while academia is recently starting to investigate the problem of composing sequential layers of detection modules ponte2025slifer . Regardless of their provenience, either from industry or academia, the first layer of detection is achieved through pattern-matching with YARA rules. 1 1 1 https://github.com/VirusTotal/yara Manually crafted by domain experts, YARA rules contain both specific patterns of byte associated with malicious samples, and conditions that must be met to trigger detection. However, even if many samples are already stopped by available rules, malware detectors are trained on all available data, and deployed alongside YARA rules. While intuitive, this methodology might not be optimal, since each layer of detection alters the distribution of data fed in input to the subsequent detection module. As a result, models are trained on samples they will never see at test time, complicating not only the training process due to the massive amount of samples used to create these detectors, but also potentially including spurious correlations exploited by attackers with adversarial EXEmples – carefully-manipulated Windows malware that try to evade ML detection demetrio2019explaining ; demetrio2021adversarial . Fig. 1 : AI System for malware detection, which combines signature-based detection to ML, both during (1) training, by removing samples that trigger YARA rules, and (2) test time, by using ML only on unknown samples. Hence, in this work we perform preliminary steps towards an empirical understanding of the role of YARA detection layer at training and test time, as shown in Fig. 1 . As far as we know, we are the first to propose such an analysis, where we exclude from training data all those samples that are either matched by at least one YARA rule, or are contained inside an allowlist composed utilities harvested from fresh installation of Windows. We empirically quantify how such pre-filtering influences the learning process of the AI system we build, showing that this methodology matches and exceeds the performances of models trained on all the available samples at low FPR (1%), also when exposed to never-seen future data possibly affected by concept drift. Also, thanks to the presence of YARA rules, the AI system we build exhibits higher robustness to adversarial EXEmples rather then regularly-trained models, since the decision function is harder to explore by the attack. Similar to previous work ponte2025slifer , we observe that the computation of adversarial EXEmples can introduce unforeseeable artifacts that are detected by rules when injected by the optimization process. However, these benefits are balanced by a fixed amount of false positives induced by rules themselves. Even if some of them are very effective in stopping hundreds of malware samples, they can not be tuned to adapt to the training data and reduce their response to false alarms, differently from machine learning models. We conclude our work by remarking that these are preliminary results on the performance of AI systems for malware detection, but already highlighting potential improvement in the field in terms of efficiency and robustness. II Background and Related Work Before describing our methodology, we introduce the key concepts needed to fully understand our manuscript, comprising the types of data we are dealing with, how to detect malicious samples among them, and how attackers can evolve their technique to evade complex ML models used for malware detection. Fig. 2 : Depiction of the Windows PE File Format. Windows PE File Format. Each Windows program is stored as a file, whose structure is defined by the Portable Executable (PE) format 2 2 2 https://learn.microsoft.com/en-us/windows/win32/debug/pe-format . As depicted Fig. 2 , the format provides precise information on how to load programs in memory, and it is divided into metadata and code. The latter, alongside other relevant information such as initialized data, are stored in multiple sections (E in Fig. 2 ), which constitute the majority of each program. Detection with YARA Rules. YARA is a pattern-matching tool used to detect known malware based on signatures. These signatures are textual descriptions containing binary patterns that help identify malicious programs. Each YARA rule consists of metadata that describes its purpose, followed by strings or patterns—such as hexadecimal sequences or regular expressions—that the tool searches for within files. Finally, each rule includes a firing condition, which determines how malicious activity is identified. This condition is structured as an if-then-else logic block that defines the detection algorithm. This mechanism is not only helpful in detecting malware samples, since conditions can be designed to identify well-known unharmful executables, such as operating systems applications and certified software installers. Static Malware Detection with Machine Learning. This type of analysis discriminates legitimate and malicious programs by analyzing the sole structure of executables without running them, and train machine learning models on top of these activities. The latter is achieved by training models either on features, which are measurements computed on data through dedicated algorithms that leverage domain knowledge anderson2018ember ; saxe2015deep ; or on raw bytes raff2018malware often treated as gray-scale images nataraj2011malware ; or on a combination of the two mentioned approaches liu2017automatic . While promising, static analysis might be circumvented by obfuscation since malware samples can manifest malicious behaviors at runtime that cannot be easily inferred without execution. Also, complex feature extraction algorithms might crash on specific samples, requiring flexibility in the analysis process. To overcome these limitations, static malware detectors can be improved by either applying redundant and multiple controls ponte2025slifer also involving signature-based detection, or merging it with dynamic analysis trizna2022quo , which also captures the runtime behavior of samples. However, as far as we know, none of these work on either single- or multi-layer detectors considered the influence that each layer imposes on