Self-Supervised Transformer-based Contrastive Learning for Intrusion Detection Systems

Ippokratis Koukoulis, Ilias Syrigos, and Thanasis Korakis · 2025

Read Self-Supervised Transformer-based Contrastive Learning for Intrusion Detection Systems by Ippokratis Koukoulis, Ilias Syrigos, and Thanasis Korakis (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2505-08816.txt.

Introduction

Ippokratis Koukoulis, Ilias Syrigos, and Thanasis Korakis propose a new way to spot network attacks in a 2025 paper. Attacks that have never been seen before are hard to catch, and systems that learn from labeled data struggle when traffic patterns change. The team aims for a method that learns from raw traffic with no labels.

Their method trains a transformer model by contrasting altered copies of packet sequences, so it learns the shape of normal and odd flows on its own. It beat older self-taught methods based on flow records, with gains of up to twenty points in tests across different data sets. With a little labeled data added, it also gave a strong base for standard detection tasks.

Catalog record

CREATORIppokratis Koukoulis, Ilias Syrigos, and Thanasis Korakis
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE38 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2505-08816

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Self-Supervised Transformer-based Contrastive Learning for Intrusion Detection Systems Thanks: This work has received co-funding from the Smart Networks and Services Joint Undertaking (SNS JU) under the European Union’s Horizon Europe research and innovation programme, in the frame of the NATWORK project (Net-Zero self-adaptive activation of distributed self-resilient augmented services) under Grant Agreement No 101139285. Ippokratis Koukoulis, Ilias Syrigos, and Thanasis Korakis Abstract As the digital landscape becomes more interconnected, the frequency and severity of zero-day attacks, have significantly increased, leading to an urgent need for innovative Intrusion Detection Systems (IDS). Machine Learning-based IDS that learn from the network traffic characteristics and can discern attack patterns from benign traffic offer an advanced solution to traditional signature-based IDS. However, they heavily rely on labeled datasets, and their ability to generalize when encountering unseen traffic patterns remains a challenge. This paper proposes a novel self-supervised contrastive learning approach based on transformer encoders, specifically tailored for generalizable intrusion detection on raw packet sequences. Our proposed learning scheme employs a packet-level data augmentation strategy combined with a transformer-based architecture to extract and generate meaningful representations of traffic flows. Unlike traditional methods reliant on handcrafted statistical features (NetFlow), our approach automatically learns comprehensive packet sequence representations, significantly enhancing performance in anomaly identification tasks and supervised learning for intrusion detection. Our transformer-based framework exhibits better performance in comparison to existing NetFlow self-supervised methods. Specifically, we achieve up to a 3% higher AUC in anomaly detection for intra-dataset evaluation and up to 20% higher AUC scores in inter-dataset evaluation. Moreover, our model provides a strong baseline for supervised intrusion detection with limited labeled data, exhibiting an improvement over self-supervised NetFlow models of up to 1.5% AUC when pretrained and evaluated on the same dataset. Additionally, we show the adaptability of our pretrained model when fine-tuned across different datasets, demonstrating strong performance even when lacking benign data from the target domain. I Introduction Digital connectivity is significantly expanding day by day and we now live in a highly interconnected world. At the same time, well-known cyber threats are still being encountered worldwide, there are new emerging ones that are often unknown to existing defense mechanisms and cybersecurity experts, consequently going undetected to form zero-day attacks. According to Rapid7’s Attack Intelligence Report [ 1 ] , 2023 was the year that showed a higher percentage of zero-day vulnerabilities than any previous year, with every such breach resulting in significant operational and financial costs, as highlighted by IBM 2024 Data Breach Report [ 2 ] , which indicates that the average cost of a data breach stands at 4.88 million dollars. Therefore, there is an urgent need for intelligent and innovative approaches that extend the traditional threat detection mechanisms to offer accurate and effective zero-day threat detection. Traditional intrusion detection systems are unable to identify new threats if there does not already exist a known signature for them, while anomaly detection systems often fail to distinguish between malicious and legitimate anomalies, which leads to a lot of false positives [ 3 ] . To overcome these challenges, many approaches have been developed that employ Machine Learning and Deep Learning methods to recognize abnormal traffic patterns, thereby enabling them identify known and zero-day attacks more effectively. Although these supervised methods are accurate, they are limited because they require a large and carefully labeled training dataset. This limits their capacity to generalize to various types of traffic and attacks and turns them inefficient, as it renders them overly reliant on the manual labor required to label the traffic flows. Self-supervised learning (SSL) techniques, such as transformer-based architectures and large language models (LLMs), offer great potential for overcoming the limitations and shortcomings of supervised learning, as highlighted by recent advancements in Deep Learning, especially in the domains of computer vision and natural language processing (NLP). SSL techniques are able to learn useful representations from unlabeled data that can be leveraged for a variety of downstream tasks, such as classification. In addition, contrastive learning, a process that has been employed to extract representations from images [ 4 ] as well as from text [ 5 ] , is superior at learning representations by maximizing the similarity between related data points and minimizing dissimilar ones. Consequently, contrastive learning models are significantly more generalizable and robust, when encountering new and previously unseen traffic patterns. This paradigm has also been recently adapted in the domain of tabular data with conventional neural network architectures [ 6 ] and transformer architectures [ 7 ] . Related works that are based on self-supervised contrastive learning [ 8 ] derive insights from flow-level statistical features such as traffic volume and packet frequency. These features are capable of significant predictive capabilities. Nevertheless, they are unable to encapsulate packet-based specific information and frequently require manual selection based on domain expertise and traffic characteristics, resulting in a reliance on handcrafted features that significantly limits their adaptability to a variety of network environments. Additionally, the ML models that are trained on a fixed statistical distribution may be unable to detect novel attack strategies as network architectures and cyber threats continue to evolve. In order to address these limitations, it is essential to implement a more advanced methodology that is capable of automatically extracting meaningful representations from network traffic sequences. In this paper, we propose a novel transformer-based framework that can extract a flow representation from a sequence of packets. To achieve this, we train our model using a contrastive learning process, where the model learns to minimize the distance between an original flow packet sequence and an augmented one, while maximizing the distance with other samples. To create augmented packet sequences, we employ a process that allows us to mix packets of the original flow with that of another one, to create a new sequence that is similar to the original. The source code for the model implementation is provided here: https://github.com/koukipp/contrastive_transformers_ids . The main contributions of our paper are the following: • The development of a novel transformer-based framework pretrained on unlabeled data traffic that can extract useful flow representations directly from raw packet sequences, reducing the need for manual extraction of statistical features from flows. • The proposal of a simple augmentation process that enables effective contrastive learning on sequences of packets. • The evaluation of the extracted representations in an unsupervised setting to detect anomalous flows and demonstrate our framework’s ability to recognize novel attacks across multiple datasets. • The demonstration of the effectiveness of our pretraining procedure to enhance the generalization of intrusion detection in supervised settings, where the model is finetuned with a small amount of samples. II Related Work II-A Transformer-based models for network traffic In the context of sequential modeling for network traffic [ 9 ] , transformer [ 10 ] architectures have gained traction. This is due to the fact that general models can be fine-tuned on a wide range of downstream tasks using labeled data, including intrusion detection, after being pretrained on unlabeled network traffic. In [ 11 ] , the authors propose a foundation model that facilitates pretraining on raw packet data and fine-tuning on specific security-related tasks. In this work, the packet trace of each flow is tokenized by splitting the header and a small

Read the complete text