Enhancing Network Intrusion Detection Systems: A Multi-Layer Ensemble Approach to Mitigate Adversarial Attacks
Nasim Soltani 1,3 , Shayan Nejadshamsi 4 , Zakaria Abou El Houda 1,3 , Raphaël Khoury 2,3 , Kelton A. P. Costa 5 , Tiago H. Falk 1,3 , Anderson R. Avila 1,3 1 Institut national de la recherche scientifique (INRS-EMT), Université du Québec, Montréal, Québec, Canada 2 Department of Computer Science and Engineering, Université du Québec en Outaouais, Gatineau, Québec, Canada 3 INRS-UQO Mixed Research Unit on Cybersecurity, Gatineau, Québec, Canada 4 Concordia Institute for Information Systems Engineering (CIISE), Concordia University, Montréal, Québec, Canada 5 Department of Computing, São Paulo State University, São Paulo, Brazil · 2026
Read Enhancing Network Intrusion Detection Systems: A Multi-Layer Ensemble Approach to Mitigate Adversarial Attacks by Nasim Soltani 1,3 , Shayan Nejadshamsi 4 , Zakaria Abou El Houda 1,3 , Raphaël Khoury 2,3 , Kelton A. P. Costa 5 , Tiago H. Falk 1,3 , Anderson R. Avila 1,3 1 Institut national de la recherche scientifique (INRS-EMT), Université du Québec, Montréal, Québec, Canada 2 Department of Computer Science and Engineering, Université du Québec en Outaouais, Gatineau, Québec, Canada 3 INRS-UQO Mixed Research Unit on Cybersecurity, Gatineau, Québec, Canada 4 Concordia Institute for Information Systems Engineering (CIISE), Concordia University, Montréal, Québec, Canada 5 Department of Computing, São Paulo State University, São Paulo, Brazil (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2603-10413.txt.
Introduction
Nasim Soltani and co-authors build a multilayer defense to keep machine learning based network intrusion detection systems safe from adversarial attacks. Such attacks can fool detectors with crafted traffic, so the team generated hostile examples with a GAN and with the fast gradient sign method. Their defense stacks classifiers in a first layer, then checks benign calls with an autoencoder in a second layer, and adds adversarial training. On two standard datasets, the approach raised resilience against these attacks.
Catalog record
| CREATOR | Nasim Soltani 1,3 , Shayan Nejadshamsi 4 , Zakaria Abou El Houda 1,3 , Raphaël Khoury 2,3 , Kelton A. P. Costa 5 , Tiago H. Falk 1,3 , Anderson R. Avila 1,3 1 Institut national de la recherche scientifique (INRS-EMT), Université du Québec, Montréal, Québec, Canada 2 Department of Computer Science and Engineering, Université du Québec en Outaouais, Gatineau, Québec, Canada 3 INRS-UQO Mixed Research Unit on Cybersecurity, Gatineau, Québec, Canada 4 Concordia Institute for Information Systems Engineering (CIISE), Concordia University, Montréal, Québec, Canada 5 Department of Computing, São Paulo State University, São Paulo, Brazil |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 25627 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2603.10413
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Enhancing Network Intrusion Detection Systems: A Multi-Layer Ensemble Approach to Mitigate Adversarial Attacks Nasim Soltani 1,3 , Shayan Nejadshamsi 4 , Zakaria Abou El Houda 1,3 , Raphaël Khoury 2,3 , Kelton A. P. Costa 5 , Tiago H. Falk 1,3 , Anderson R. Avila 1,3 1 Institut national de la recherche scientifique (INRS-EMT), Université du Québec, Montréal, Québec, Canada 2 Department of Computer Science and Engineering, Université du Québec en Outaouais, Gatineau, Québec, Canada 3 INRS-UQO Mixed Research Unit on Cybersecurity, Gatineau, Québec, Canada 4 Concordia Institute for Information Systems Engineering (CIISE), Concordia University, Montréal, Québec, Canada 5 Department of Computing, São Paulo State University, São Paulo, Brazil Abstract Adversarial examples can represent a serious threat to machine learning (ML) algorithms. If used to manipulate the behaviour of ML-based Network Intrusion Detection Systems (NIDS), they can jeopardize network security. In this work, we aim to mitigate such risks by increasing the robustness of NIDS towards adversarial attacks. To that end, we explore two adversarial methods for generating malicious network traffic. The first method is based on Generative Adversarial Networks (GAN) and the second one is the Fast Gradient Sign Method (FGSM). The adversarial examples generated by these methods are then used to evaluate a novel multilayer defense mechanism, specifically designed to mitigate the vulnerability of ML-based NIDS. Our solution consists of one layer of stacking classifiers and a second layer based on an autoencoder. If the incoming network data are classified as benign by the first layer, the second layer is activated to ensure that the decision made by the stacking classifier is correct. We also incorporated adversarial training to further improve the robustness of our solution. Experiments on two datasets, namely UNSW-NB15 and NSL-KDD, demonstrate that the proposed approach increases resilience to adversarial attacks. Keywords: Adversarial Machine Learning, Intrusion Detection, Generative Models, Generative Adversarial Network (GAN), Fast Gradient Sign Method (FGSM). I Introduction A Machine Learning (ML) model might incorrectly classify an input if it has been carefully crafted for this purpose as part of an adversarial attack [ 17 ] . This is typically achieved by adding a small perturbation \epsilon to the input. This perturbation, often imperceptible, is enough to alter the model’s behaviour, such that f(x)\neq f(x+\epsilon) . As shown in Figure. 1 , the adversarial example is designed in such a way as to make the ML model map the input to the wrong class. The consequences of successful attacks can be far-reaching, including confidentiality and privacy breaches, or violations of integrity and availability, impacting the reliability of ML systems in critical applications such as medical diagnosis [ 10 ] and Network Intrusion Detection Systems (NIDS) [ 6 ] . To address this class of vulnerabilities, Adversarial Machine Learning (AML) emerges as a field that evaluates ML models against malicious attacks and proposes countermeasures to mitigate their detrimental effects [ 14 ] . Fig. 1 : Under a small perturbation, \epsilon , input x becomes an adversarial example that is misclassified into f(x+\epsilon) , which falls on the wrong side of the decision boundary. NIDS monitors network traffic to detect malicious behavior [ 5 ] . As a defense mechanism, its resilience to adversarial attacks is crucial, and this problem has been addressed in the recent literature. For example, Alhajjar et al. [ 7 ] , focus on measuring the vulnerability of NIDS to adversarial attacks capable of evading ML-based models. The adversarial inputs are generated using three techniques: particle swarm optimization (PSO), Genetic Algorithm (GA), and Generative Adversarial Network (GAN). The authors were able to increase the misclassification rates across several ML models, highlighting the need for more resilient NIDS. Similarly, Han et al. [ 12 ] , evaluate the robustness of ML-based NIDS against traffic-space adversarial attacks. The generated examples were based on gray- and black-box methods and achieved over 97% evasion. The authors also proposed a defense mechanism that reduced evasion to 50%. The proposed attacks and the robustness of the defense method were evaluated on the state-of-the-art NIDS Kitsune and on several ML-based NIDS. Abou Khamis et al. [ 3 ] , propose an intrusion detection system based on a min-max approach. Adversarial attack samples were generated using the UNSW-NB15 network dataset. Their defense mechanism relied on optimizing a model for intrusion detection, and the authors report that their approach was able to increase the robustness of NIDS. To remain effective, NIDS must withstand adversarial attacks while continuously integrating new anti-manipulation strategies [ 4 ] . Hence, the aim of this work is twofold. First, we seek to create adversarial attacks to assess the vulnerabilities of traditional ML classifiers. Two network communication datasets, namely UNSW-NB15 [ 2 ] and NSL-KDD [ 1 ] , are used to generate network adversarial examples. We also propose a multi-layer solution that integrates a stacking classifier and an autoencoder to detect malicious network traffic. The contributions of this paper are as follows: • We evaluate the impact of two adversarial methods to generate malicious network traffic on the effectiveness of ML-based NIDS. • We propose a defense mechanism based on a multi-layer approach and adversarial training, which successfully enhances the robustness of NIDS in two benchmarks. The remainder of this paper is structured as follows: Section II presents the mechanism used to generate adversarial attacks. Section III describes the proposed defense mechanism. Section IV details the experimental setup and Section V reports our results. Section VI concludes the paper. II Attack Methodology Fig. 2 : The Generative Adversarial Network against NIDS is composed of a generator G , a discriminator D and a voting classifier. While G is trained to produce mutable adversarial vectors, D is optimized to identify malicious generated network traffic in collaboration with the voting classifier. In this section, we describe the two network adversarial attack strategies used in this study: Generative Adversarial Networks (GAN) and the Fast Gradient Sign Method (FGSM), where FGSM serves as a baseline for comparison due to its simplicity and efficiency. II-A Generative Adversarial Network Inspired previous research [ 7 , 9 ] , we propose the GAN architecture depicted in Figure 2. Note that after vectorizing the network traffic into mutable features M , and immutable features I , the data is separated into malicious and benign traffic. For the datasets adopted in this work, we considered the same immutable and mutable features defined by Alhajjar et al. [ 7 ] . The generator G , receives malicious input z , which is mutable features perturbed by noise, and referred to as M^{\prime} . Note that the immutable features are not used as input in this case. The generator outputs M^{*} , which is concatenated with I , producing new adversarial examples. The discriminator, D , receives as input real benign samples from the original network traffic as well as crafted malicious (i.e., adversarial examples) ones from the generator G . This approach ensures that malicious and crafted examples are produced by the generator while real and benign ones come from the original datasets. Similarly to the min-max approach presented by Kamis et al. [ 3 ] , the generator is trained to deceive the discriminator with malicious examples and the discriminator is optimized to