Malicious DNS detection by combining improved transformer and CNN Authors: Heyu Li, Zhangmeizhi Li, Shuyan Zhang, Xiao Pu Published: 2024 ABSTRACT AbstractWith the widespread application of the Internet, network security issues have become increasingly prominent. As an important infrastructure of the Internet, the domain name server has been attacked in various forms. Traditional methods for detecting malicious domain servers are usually based on rules or feature engineering, requiring a large amount of manual participation and rule library updates. These methods cannot adapt to the constantly changing threat environment. In response to these issues, this study first improves the Transformer by adjusting its attention head and encoding method. Then, the model is combined with convolutional neural networks. Finally, a block-based ensemble classifier is used for classification detection. The relevant outcomes showed that the average accuracy score of the proposed method was as high as 95.8 points, the average detection time score was 96.8 points, the average feature extraction ability score of the model was 96.3 points, and the overall performance score was 97.6 points. This method has significant advantages over traditional methods in terms of accuracy and detection time, providing a new tool for detecting malicious domain servers.Keywords: Transformer, CNN, Malicious DNS detection, Network security, Multiple attention mechanismSubject terms: Engineering, Mathematics and computing Introduction With the popularity of the Internet, network security has gradually become a global focus. Malicious Domain Name Naming System (DNS) activities, including domain hijacking, DNS-based attacks, and domain secret communication, pose a serious threat to network security, potentially leading to data breaches, malware propagation, and network service interruptions. The increasing harmfulness of these activities makes malicious DNS detection a critical task in protecting network security. Traditional DNS data analysis relies on sequence data such as domain name strings or IP address sequences. In recent years, deep learning techniques have made significant progress in processing these data, especially in malicious DNS detection1. However, despite the new solutions brought by deep learning technology for network security, there are still some challenges and limitations in existing research2. Firstly, the data dependency in many studies limits the generalization ability of existing models. The model may only perform well on specific types of data and perform poorly in different environments or new types of attacks. Secondly, the complexity and computational cost of the model cannot be ignored3. Although complex deep learning architectures improve detection accuracy, they also increase the demand for computing resources and processing time, which is a major challenge for security systems that require rapid response4. In addition, false positives and false negatives still exist. High accuracy claims may mask these flaws, leading to resource waste or security risks. Furthermore, insufficient dynamic adaptability means that existing models may not be able to respond promptly to emerging threats5. Finally, deep learning models have poor interpretability and are considered a “black box”, with a lack of transparency in the decision-making process, which is a major obstacle for security applications that require high levels of trust. Especially, although Transformer models perform well in sequence modeling and semantic understanding, they have high computational complexity and low processing efficiency when dealing with long sequences. Moreover, its ability to capture local features is relatively weak. Therefore, this study proposes an improved method combining Transformer and Convolutional Neural Network (CNN) for malicious DNS detection6. This method aims to overcome the limitations of existing technologies and improve the accuracy, efficiency, and adaptability of malicious DNS detection. It is expected to provide a more comprehensive and efficient network security solution by integrating the long sequence processing capability of Transformer and the local feature capture capability of CNN. The main contributions of this study are as follows. A new malicious DNS detection framework is proposed to improve detection performance by improving Transformer and CNN. This method reduces the model’s dependence on specific datasets and enhances its generalization ability. Meanwhile, it reduces computational costs and improves processing speed. By integrating classifiers, the interpretability of the model is improved. These improvements will make a significant contribution to network security. The main contribution of this study is the proposed malicious DNS detection model that combines improved Transformer and CNN, aiming to overcome the limitations of existing technologies and improve detection performance. By improving the attention head and encoding method of Transformer, combined with the local feature capture ability of CNN, this model enhances its generalization ability and interpretability while reducing computational costs and improving processing speed. The significance of this study in practical applications is significant. This provides a more comprehensive and effective solution for network security protection. This not only helps to protect individuals and organizations from threats such as data breaches, malware propagation, and network service interruptions, but also promotes technological progress in the field of network security, with important practical application value and social significance. This paper mainly includes four parts. The second chapter summarizes the research status of transformer technology and CNN in network security. The third chapter builds the malicious DNS detection technology based on the ITransformer-CNN algorithm. Chapter 4 provides a comparative experiment on the algorithm performance and an application analysis. Chapter 5 summarizes the full text and future prospects. Related works With the popularization of the Internet, network security issues are receiving increasing attention from experts and scholars. Alieyan et al.7 proposed a DNS rule-based Botnet detection method (DNS-BD) to address the negative economic impact of Botnet on organizations and individuals. This method utilized DNS query and response behavior. The relevant outcomes showed that this method could detect Botnet, with an accuracy of 99.35% and a false alarm rate of 0.25%. Ravi et al.8 proposed a novel deep learning technology that did not require disruptive engineering checks. This addressed the labor-intensive, time-consuming, error-prone, and significant limitations of existing detection Domain Name Generation Algorithms (DGA). The evaluation results indicated that this method effectively identified DNS homonym attacks and DGA, with an accuracy of 0.99. Lyu et al.9 proposed a method for detecting distributed DNS attacks to address the non-robustness of Distributed Denial of Service (DDoS) on the ground of DNS. This method was combined with machine learning to identify abnormal behavior at various levels of the hierarchical structure. The results showed that the accuracy of the model in identifying DNS attack categories exceeded 99%, with good robustness. Altuncu et al.10 proposed a real-time deep network system to address the threat of DNS channel attacks. This system made decisions on threats in real-time systems and trained through various deep network topologies. The test results showed that this method successfully prevented 99.91% of attacks. Singh et al.11 proposed a machine learning algorithm that combined decision trees, logistic regression, K-nearest neighbors (KNN), and random forests to detect and filter DoH traffic to address the security issues of DNS over HTTPS (DoH). This algorithm divided it into benign DoH traffic and malicious DoH traffic. The relevant outcomes showed that this method had an accuracy of over 99%. Liang et al.12 proposed a hybrid Transformer CNN based on CNN and Transformer branches to address the high complexity and low accuracy in remote sensing change detection. This network used a Transformer-based progressive sampling mechanism to focus the model’s attention on the objects of interest. This model outperformed other advanced methods in terms of performance, achieving an accuracy rate of 98%. Rai et al.13 proposed a new method for detecting and classifying High Impedance Faults (HIFs) in traditional over-current relays. The method was based on a deep learning model that used Transformer networks and CNNs. The outcomes showed that the proposed protection method could reliably detect and classify HIF, and had robust to noise. Bagal et al.14 trained the Mol centroid model based on the Transformer decoder. A masked self-attention mechanism was used to generate molecules with drug features for the next label prediction task, thereby solving the high complexity of reverse molecular design in drug design. The experiments showed that this model could generate expected skeleton and molecular properties based on the required skeleton SMILES strings and attribute values. Chen et al.15 proposed an innovative real-time multi-satellite scheduling method to improve the efficiency and speed of Earth Observation Satellite (EOS) scheduling. This method consisted of a hierarchical prediction model on the basis of machine learning and a heuristic local search algorithm. Compared with existing methods, the proposed method had a shorter calculation time and a 21.3% higher planning benefit. Beijing et al.16 designed an improved quaternion dual stream R-CNN model to address the limitations of existing methods in color image stitching and localization. This model located the concatenated region at the pixel level by adding a mask branch that combined a fully convolutional network and a conditional random field. The relevant outcomes showed that the proposed method was more excellent than existing methods on localization ability, generalization ability, and robustness. A list of relevant research is shown in Table 1. Table 1List of related research.ResearcherMethod descriptionAdvantagesLimitationsAlieyan et al.7A DNS rule-based Botnet detection method (DNS-BD) that utilizes DNS query and response behaviorAccuracy of 99.35%, false alarm rate of 0.25%May depend on specific datasets, and limited generalization abilityRavi et al.8A novel deep learning technology that does not require disruptive engineering checks, addressing the labor-intensive, time-consuming, error-prone, and significant limitations of existing DGA detection methodsEffectively identify DNS homonym attacks and DGA, with an accuracy of 0.99High computational complexity, and may not be suitable for real-time applicationsLyu et al.9A distributed DNS attack detection method combined with machine learning to identify abnormal behavior at various levels of the hierarchical structureWith an accuracy of 99%, good robustnessRequire large amounts of labeled data, and high training costsAltuncu et al.10A real-time deep network-based DNS channel attack detection system trained through various deep network topologiesSuccessfully prevented 99.91% of attacksHigh real-time processing requirements, and high demand for computational resourcesSingh et al.11A machine learning algorithm combining decision trees, logistic regression, KNN, and random forests for detecting and filtering DoH trafficAccuracy over 99%, able to distinguishing between benign and malicious DoH trafficHigh model complexity, and poor interpretabilityLiang et al.12A hybrid Transformer CNN network based on CNN and Transformer branches, using a Transformer-based progressive sampling mechanismAccuracy of 98%, superior to other advanced methodsHigh computational complexity when handling long sequences, and weaker ability to capture local featuresRai et al.13A deep learning model based on Transformer networks and CNNs for detecting and classifying high impedance faults (HIF)Reliably detect and classify HIF, robust to noiseRequires high-quality labeled data, and longer training timeBagal et al.14A Mol centroid model based on a Transformer decoder, using a masked self-attention mechanism for the next label prediction taskGenerate molecules with drug characteristics, and solve high complexity problemsDifficult to train, and high data quality requirementsChen et al.15An innovative real-time multi-satellite scheduling method combining a hierarchical prediction model based on machine learning and a heuristic local search algorithmShorter computation time, and 21.3% improvement in planning efficiencySuitable for specific scenarios, and limited generalization abilityBeijing et al.16An improved quaternion dual-stream R-CNN model for color image stitching and localization, combining a fully convolutional network and a conditional random fieldSuperior in localization ability, generalization ability, and robustness compared with existing methodsHigh model complexity, and high demand for computational resources List of related research. Although the above research has achieved significant results in DNS security, there are some common limitations. Although the DNS-BD method proposed by Alieyan et al. has high accuracy, a false positive rate of 0.25% may cause interference to users. The method proposed by Ravi et al. reduces the need for engineering inspections, but deep learning model training is still time-consuming and computationally intensive. Although the method designed by Lyu et al. improves the detection accuracy of DDoS attacks, it may lack defense capabilities against other types of DNS attacks. The system proposed by Altuncu et al. can detect threats in real-time, but its generalization ability to unknown attacks needs to be verified. To address these issues, the research innovatively combines the advantages of One Hot Encoding (OHE) and CNN to improve the accuracy of malicious DNS traffic detection. By fine feature extraction and ensemble learning, false positives and false negatives have been reduced. The improved transformer model has increased computational efficiency and reduced the time required for model training and detection. The block-based ensemble classifiers enhances the model’s generalization ability to unknown attack types. Model optimization enables it to quickly respond to DNS queries in real-time systems and provide timely security protection. Through these improvements, the ITransformer-CNN model aims to provide a more accurate, faster, and robust malicious DNS detection solution to address constantly changing network security threats. A malicious DNS detection scheme based on ITransformer-CNN This chapter mainly includes three sections. The first section first introduces the relevant concepts and hazards of malicious DNS, and then proposes the overall scheme of the ITransformer-CNN malicious DNS detection model based on the characteristics of DNS. The second section mainly focuses on improving the attention header and encoding method of Transformer. The third section designs the ITransformer-CNN algorithm based on the improved Transformer. Overall design of transformer-CNN malicious DNS detection scheme Malicious DNS refers to a domain name used for malicious activities or for malicious purposes. DNS is a system used in the Internet to convert domain names into corresponding IP addresses, serving as a bridge to map human-readable domain names to computer-recognizable IP addresses17,18. Malicious DNS includes various forms of malicious domain names, malicious hosts, and malicious IP addresses. These malicious elements may be related to network attacks, fraudulent behavior, malware distribution, phishing attacks, etc. A malicious domain name can mimic the domain name of a legitimate website and be used for phishing attacks, guiding users to input sensitive information19,20. Malicious hosts or IP addresses may be used to host command and control servers for malicious software or to conduct DDoS attacks. The malicious DNS is to conceal the true identity and malicious activities of attackers, making them more difficult to detect and prevent. The user device sends a Domain Name Resolution Request (DNRR) to the DNS server through the DNS client. If the request is hijacked to a malicious DNS server, the malicious DNS server will return a false IP address to the DNS client. When the DNS client receives a fake IP address, the user’s device will connect to the malicious destination, leading to various security issues. The schematic diagram of this process is shown in Fig. 1. The interaction process between user devices and malicious DNS servers. The malicious DNS diagram describes the interaction process between user devices and malicious DNS servers, including eight parts. The user’s device represents the computer, phone, or other device used by the user. The user enters a domain name in the browser and initiates a DNRR. The DNS client is a software located on the user’s device, which is responsible for sending the user’s DNRR to the DNS server and receiving the returned IP address. A malicious domain name refers to a domain name used for malicious activities, such as the domain name of a phishing website or the domain name of a malware command and control server. The user-entered domain name may be malicious. DNS servers are managed by Internet Service Providers (ISP) or other trusted institutions. When the DNS client sends a DNRR, the DNS server is responsible for resolving the domain name to the corresponding IP address and returning the result to the DNS client. Malicious DNS servers are controlled by attackers. It can hijack users’ DNRR and return false IP addresses. In this way, user devices will be directed to malicious websites or services. A false IP address is an IP address returned by a malicious DNS server that points to a malicious website or service. When the DNS client receives a false IP address, the user’s device will connect to the malicious destination. Malicious destination refers to a malicious website or service pointed to by false IP addresses returned by malicious DNS servers. It may be a phishing website, a malware distribution site, or a server used for other malicious activities. As a result of the attack, when a user’s device is connected to a malicious destination, it may suffer from security threats, such as personal information leakage, malware infection, or being induced to commit fraud. In response to above characteristics of malicious DNS, an improved Transformer is combined with CNN to fully utilize the advantages of both. Among them, Transformers can effectively capture and process sequence information, such as domain names. CNN can effectively extract non-sequential information, such as IP packet header information. By combining these two methods, malicious features in DNS traffic can be extracted more comprehensively and efficiently. The overall design structure of the ITransformer-CNN malicious DNS detection scheme is shown in Fig. 2. Schematic diagram of the overall design structure of ITransformer-CNN malicious DNS detection scheme. Figure 2 shows the overall design structure of the ITransformer-CNN malicious DNS detection scheme. This diagram includes three modules: flow analysis, data processing, and ITransformer-CNN model. In the data processing module, DNS data are stored in text format. Each record contains information such as the requester’s IP, request domain name, response IP, query type (such as A record, MX record, etc.), query timestamp, etc. For the convenience of subsequent processing, these data usually are converted into JSON structured format. In the data preprocessing stage, it is necessary to clean the original data. This includes removing duplicate records, handling missing values, correcting erroneous data, etc. In addition, it is necessary to standardize the domain name, such as converting it to lowercase, removing trailing points, etc., to ensure that subsequent feature extraction and model training can be based on a unified data format. Based on the Zeek, the traffic resolution module converts the original DNS traffic data into a form that is easy to understand and process, including domain names and other traffic information. The data processing module is mainly responsible for processing the data obtained from the traffic analysis module. Specifically, this module uniformly converts domain name data into domain name sequences through sequence length, while performing data cleaning and normalization on other traffic information. This processing method helps to eliminate differences and noise between data, thereby extracting more representative traffic features. The ITransformer-CNN model is the core part of the entire detection scheme, receiving processed data and performing further feature extraction and malicious DNS detection on it. Specifically, the ITransformer section is responsible for processing domain name sequences by focusing on the sequence relationships between domain name characters and utilizing the encoder structure of Transformer for feature extraction. Meanwhile, the CNN part processes other traffic information, extracts traffic features, and uses a block-based ensemble classifier to classify attack types. Improvement of transformer attention head and encoding method The Transformer model is a machine learning model utilized for Natural Language Processing (NLP) and other sequential tasks. It was proposed in 2017 to encode and decode input sequences using a self-attention mechanism21. Unlike traditional Recurrent Neural Networks (RNN), the Transformer model can process input sequences in parallel, making it more efficient in the training and inference stages22,23. Due to the fact that DNS data are usually represented in sequence form, the Transformer model can encode each element in the sequence (such as domain name, IP address, etc.) to capture contextual information and potential features in the sequence. This helps to extract effective malicious features, such as abnormal domain names and suspicious IP addresses. Secondly, since the Transformer model can capture long-distance dependencies, it can model the associations between multiple DNS queries. This is of great significance for detecting distributed malicious behavior and variations of domain name generation algorithms. Finally, the Transformer model can learn the distribution and pattern of normal DNS traffic to detect abnormal traffic that does not conform to normal behavior24,25. This helps to detect and prevent malicious attacks in a timely manner, such as domain hijacking and DGA attacks, as displayed in Fig. 3. Schematic diagram of transformer model structure. The Transformer model consists of two components: an encoder and a decoder. The encoder encodes each element in the input sequence, generating a series of vector representations that capture contextual information in the input sequence. The decoder gradually produces the target sequence Transformer model based on the output of the encoder and the previous decoding results. The core idea of the Transformer model is the self-attention mechanism. The self-attention mechanism allows the model to dynamically focus on different positions of the input sequence during the encoding and decoding process, capturing long-range dependencies. Through multi-layer stacked encoders and decoders, the Transformer model can learn more complex semantic representations and has the ability to generate sentence sequences. The position encoding in the traditional Transformer model is shown in Eq. (1). In Eq. (1), represents the element position in the sequence. represents the embedding dimension of the model. The Transformer model uses trigonometric functions for encoding because trigonometric functions have the properties, as shown in Eq. (2). In Eq. (2), the position vector represents relative position information. In the Transformer model, traditional position encoding methods such as trigonometric function encoding may not accurately capture the absolute position information of elements in the sequence. However, single hot encoding maps each position to a unique binary vector, which can more directly represent the absolute position of elements and help the model better understand and process sequence data. Therefore, the OHE is used to represent absolute position information, as shown in Eq. (3). In Eq. (3), the diagonal positions of the matrix are all 1. However, the expansion of dimensionality increases the computational cost. Therefore, for the position embedding after OHE encoding, quantization techniques are used to reduce its storage and computational costs. Firstly, the position embedding vector encoded by OHE is quantized and mapped to an integer. For example, the original 32-bit floating-point embedding vector can be mapped to an 8-bit integer, significantly reducing the storage requirements. In the quantization process, the decimal part of a floating-point number is approximated to the nearest integer, which has little impact on the model accuracy in most cases, but can significantly reduce computational resource consumption. In addition, in the forward and backward propagation of the model, the original floating-point operations (such as matrix multiplication) are replaced with integer operations, utilizing the high efficiency of integer operations to accelerate the training and inference process of the model. In traditional self-attention mechanisms, a weighted sum is generated by calculating the similarity score between the query and the key, and applying these scores to the value matrix. To reduce the dependency of the model on information, the Transformer adopts a multi-head attention mechanism. This mechanism introduces multiple sets of different linear transformations of queries, keys, and values, each of which is called an attention head. Each attention head learns different query, key, and value representations, i.e. different mapping spaces, as shown in Fig. 4. Schematic diagram of MHAM structure. It maps each input to a query vector p and a key vector U, and performs a normalized point multiplication operation on the query vector and the key vector, as shown in Eq. (5). In Eq. (5), represents the angle between two vectors. Then, it proceeds with attention calculation, as shown in Eq. (6). In Eq. (6), V represents the value vector. Then, the input are divided into headers. The dimensions of the query vector and key vector in each header are shown in Eq. (7). In Eq. (7), h represents the model divided into h attention heads. Afterwards, multiple attention results need to be connected to obtain the multi-head attention operation results, as shown in Eq. (8). In Eq. (8), represents the parameter used for linear transformation after connecting multiple attention results. represents the output result of the i-th attention head. The calculation is shown in Eq. (9). In Eq. (9), , and . Then, the data are output to the fully connected feedforward network layer, where each neuron is connected to all neurons in the previous layer. The neurons in the previous layer transmit the input signal to each neuron in the current layer, which is then weighted and summed through the weights. The fully connected layer consists of two linear transformations. The two sub-layers use layer normalization and residual connection, respectively. From this, the output results of each sub-layer is obtained, as shown in Eq. (10). In Eq. (10), represents the fully connected feedforward network and multi-head attention. represents the layer normalization operation. The relevant details of the obtained ITransformer model structure are presented in Fig. 5. Schematic diagram of ITransformer model structure. Figure 5 shows the structure of the ITransformer model. Firstly, the input domain name sequence is encoded through an OHE layer, converting each domain name into a vector representation. Then, the encoder layer receives the encoded sequence as input and generates outputs of the same dimension. Next, the data undergoes an MHAM to capture and process relevant information in the sequence. MHAM helps models focus on different functions at different locations. Each sub-layer includes residual connections and layer normalization steps to preserve and balance the information flow between input and output. To reduce overfitting, a Dropout layer is adopted, which randomly discards a portion of neurons during the training process. Finally, the global average Pooling Layer (PL) reduces the dimensionality and aggregates the features of the sequence into a fixed-length vector representation. Finally, this vector is mapped to the final sequence feature matrix through a fully connected layer. Design of Itransformer-CNN algorithm In the ITransformer-CNN model, CNNs play an important role in extracting features, reducing dimensions, and enhancing generalization ability for improved classification and prediction of domain name sequences26. The CNN in this model consists of two one-dimensional convolutional layers and two maximum PLs. The combination of these layers helps to capture local correlations and important features in the input data. Through one-dimensional convolutional layers, the model can slide and apply multiple filters in domain name sequences to detect local patterns of different sizes. This can extract representative features, including character level and word level information. Convolutional operations can also reduce the length of input sequences, thereby reducing computational complexity and model over-fitting. The flatten layer flattens the feature map output from the last PL into a one-dimensional vector, providing input for the fully connected layer. This can integrate and compress features while retaining important contextual information. The schematic diagram of the CNN model structure used in this study is shown in Fig. 6. Schematic diagram of the CNN structure used in the study. First, CNN is used to extract features of traffic information. This is mainly achieved through two-layer one-dimensional convolution and global maximum pooling. The one-dimensional convolutional window slides and extracts features on the traffic information matrix, and the calculation process is displayed in Eq. (11). In Eq. (11), serves as the output value after convolution. serves as the elements of the input sequence. serves as the weight of the convolutional kernel. The role of the PL is to downsample, decrease the quantity of parameters, and retain the main features. Secondly, CNN is also utilized for extracting high-dimensional features and converting them into one-dimensional ones. This is mainly achieved through the fully connected layer. The fully connected layer is a fully connected neural network that converts convolutional and pooling features into one-dimensional. Each neuron is connected to all neurons in the previous layer, and its output is shown in Eq. (12). In Eq. (12), represents the activation function. The activation function utilized in this study is ReLU, as shown in Eq. (13). In Eq. (13), x represents the input value. The main role of the ReLU function in neural networks is to achieve nonlinear mapping. Finally, CNN also can help reduce over-fitting and improve generalization in the model. This is mainly achieved through Dropout. Dropout is a regularization technique used to reduce overfitting. During the training process, Dropout will randomly discard a portion of neurons and set their output to 0, as shown in Eq. (14). In Eq. (14), r is a binary random vector used to control which neurons are retained (value 1) or discarded (value 0). Finally, the ITransformer algorithm is combined with CNN to obtain a malicious DNS detection model for ITransformer-CNN. In the ITransformer-CNN model, the convolution kernel size of both the first and second one-dimensional convolution layers of the CNN is set to two, with the step size being default to 1. However, the stride size and the pooling window size of the maximum pooling layer are both set to 3. To maintain consistent input sequence length and avoid information loss, a Padding-filling strategy is used. The algorithm flow is shown in Fig. 7. The algorithm flow of ITransformer-CNN malicious DNS detection model. The first step is to preprocess the domain name sequence, converting each character into a word vector, and constructing a word embedding matrix. Next, the preprocessed domain name sequence is inputted into the word embedding layer. The third step is position encoding, which generates a sparse matrix using OHE and connects it with position encoding to obtain a new embedding matrix. The fourth step is to input the encoder and input the embedding matrix into the encoder of the ITransformer model. The fifth step is the encoder layer loop, where the encoder contains multiple layers. The following steps are performed in each layer. First, the MHAM is applied to capture the correlation of input sequences, and then the output of the MHAM layer is fully connected. Finally, the output of the fully connected layer is subjected to layer normalization and residual addition operations. The sixth step is to output the ITransformer model, which passes through all layers of the encoder for obtaining the output of the ITransformer model. The seventh step is global maximum pooling, which involves performing a global maximum pooling operation on the output of the ITransformer model to reduce dimensions. The eighth step is to extract CNN features, input the processed traffic information into the CNN model, and filter the features through one-dimensional convolution. The ninth step involves performing global maximum pooling downsampling on the output of the convolutional layer. Then, the result is mapped to the sample space through the fully connected layer to reduce feature dimensionality. The tenth step is feature fusion. The features extracted from the ITransformer model are fused with the CNN model in series operation to obtain a new feature matrix. In order to solve the input feature loss of CNN, the residual connection in CNN is applied to help the model learn more complex feature representation. Finally, this study uses a block-based ensemble classifier to perform DNS classification on the features extracted from ITransformer-CNN. In the block-based ensemble classifier, multiple classifiers such as Decision Tree, Support Vector Machine (SVM), and KNN are used as the basic classifiers. Decision tree can handle category data and reduce overfitting through feature selection. SVM is adept at handling high-dimensional data, capable of non-linear classification, and robust to noise and outliers. KNN is sensitive to outliers and helps to discover new malicious behavior patterns. These classifiers have diversity, performance complementarity, and integration effects, which can complement each other and jointly improve the accuracy and robustness of malicious DNS detection. The old components are then evaluated using the mean squared error. Then, the weight of the component classifier is determined based on the test accuracy of the current data blocks. Performance analysis and application experiment of Itransformer-CNN model This chapter contains three sections. The first section mainly explores the impact of different attention heads and encoding methods on the model through the control variable method. The second section mainly tests the accuracy, recall, and other performance indicators of the ITransformer-CNN. The third section applies the ITransformer-CNN model to actual malicious DNS detection. The impact of different attention heads and encoding methods on the model To improve the accuracy and efficiency of DNS detection, a malicious DNS detection scheme combining improved Transformer and CNNs is proposed. The encoder layer of this scheme adopts a MHAM as the main working mechanism. The ITransformer includes an input layer, word embedding layer, encoder layer, full set average PL, Dropout layer, and fully connected layer. The structure of each layer of ITransformer is displayed in Table 2. Table 2Structural parameters for each layer of ITransformer.ITransformer layer nameOutput dimensions of each layerParameterInput layer(None, 40)0Word embedding layer(None, 40, 128)12,160Encoder layer(None, 40, 168)174,228Full set average pooling layer(None, 168)0Dropout Layer(3, 32)0Fully connected layer(64)10,208 Structural parameters for each layer of ITransformer. The experiment aims to determine the impact of the number of attention heads on the performance of the Transformer model. To achieve this, the control variable method is employed. First, the number of attention heads is systematically changed while keeping other parameters unchanged, and the model performance is tested under each setting. The relationship between the number of attention heads and model performance is analyzed, and the optimal setting of attention heads is found. The relevant outcomes are displayed in Fig. 8. Accuracy of model training with different number of heads. The statistical data shown in Fig. 8 show that as the quantity of attention heads grows, the training accuracy of the model rapidly enhances. When the quantity of heads reached 5, the accuracy of the model began to fluctuate and then rapidly increased again. However, when the number of heads increased to 10, the model achieved the highest accuracy, reaching 97.39%. Afterwards, the change of the model gradually flattened out. These observations suggest that increasing the number of attention heads can improve the training accuracy of the model. However, after a certain point, further increasing the number of heads has a limited effect on accuracy. The traditional transformer position encoding method is modified and the OHE method is applied. To verify the optimization effect of this encoding method on model accuracy, Binary Encoding (BE) and Mean Encoding (ME) are introduced in the experiment and compared with Trigonometric Function Encoding (TFE) and OHE. The relevant outcomes are showed in Fig. 9. Comparison of coding methods and experimental results statistics. Figure 9 shows the comparative experimental results statistics of four encoding methods. In Fig. 9a, the number of optimal memory heads varied among different encoding methods, and there were also significant differences in accuracy. The optimal number of memory heads for OHE encoding was 10, with a corresponding accuracy of 96.64%. The optimal number of memory heads for TFE encoding was 12, with a corresponding accuracy of 94.84%. The optimal number of memory heads for BE encoding was 8, and the corresponding accuracy was 94.24%. The optimal number of memory heads for ME encoding was 10, and the corresponding accuracy was 93.84%. In Fig. 9b, there was a significant difference in detection time among the four encoding methods, with the ME encoding method having the longest detection time. The OHE encoding method proposed in the study had the lowest detection time, averaging 4965s, which was 32.3% lower than traditional TFE. Performance analysis of ITransformer-CNN model The ITransformer-CNN model utilizes CNNs to extract features, reduce dimensions, and improve generalization for better classification and prediction of domain name sequences. The size of the first convolution kernel is set to 2, and the number of convolution kernels is set to 64 layers. The maximum PL step size is set to 3 and the size is 3. The size of the second layer convolution kernel is also set to 2, and the number of convolutional kernels is 32. The CNN layer parameters in the ITransformer-CNN model are showcased in Table 3. Table 3Parameters of CNN layer in ITransformer-CNN model.CNN layer namesOutput dimensions of each layerParameterInput layer(36, 1)0One-dimensional convolutional layer(36, 64)192One-dimensional maximum pooling layer(18, 64)0One-dimensional convolutional layer(18, 32)0One-dimensional maximum pooling layer(9, 32)4128Flatten layer(288)0Dropout layer(288)0Fully connected layer(64)6208 Parameters of CNN layer in ITransformer-CNN model. To verify the role of each module of the ITransformer-CNN model, the study uses the ITransformer-CNN model with traditional attention mechanism (experimental group 1), the ITransformer-CNN model with multi-head attention mechanism (experimental group 2), the ITransformer-CNN model without decision tree module (experimental group 3), the ITtransformer-CNN model without SVM module (experimental group 4), and the ITtransformer-CNN model without KNN module (experimental group 5) as comparison methods to verify the detection performance of these experimental groups in three malicious activities: DNS hijacking, DNS amplification attack, and DNS reflection attack. Accuracy, recall rate, F1 score, and MAE are used as evaluation indicators. The results are shown in Fig. 10. Results of the ITransformer-CNN model ablation experiments. In DNS hijacking attacks, experimental group 2 using multi-head attention mechanism had the highest accuracy, with accuracy rates of 97.1%, 97.3%, and 98.6% in DNS hijacking, DNS amplification attacks, and DNS reflection attacks, respectively. The average accuracy of experimental group 1 using traditional attention mechanisms was only 93.6%. The detection performance of experiment group 3, which lacked a decision tree module decreased compared with the complete model in three types of attacks. Its recall rate decreased by an average of 7% compared with the complete ITransformer-CNN model. This indicates that the decision tree module plays an important role in improving the detection accuracy and recall rate of the model, possibly because it can help the model better handle classification and decision boundaries. Experimental group 4 lacked an SVM module, and its performance also declined. In terms of F1 value, the F1 value without SVM module decreased by about 5% compared with experimental group 2. This is because the SVM module provides additional discriminative power when processing linearly inseparable data, thereby enhancing the model ability to recognize complex attack patterns. In terms of MAE, experimental group 5 lacking KNN modules also showed performance degradation. Compared with experimental group 2, the model without KNN module showed a significant increase in MAE, especially under DNS reflection attacks, with an average increase in MAE from 0.015 to 0.039. The KNN module may play a key role in local feature learning and similarity measurement, helping the model to more accurately identify attack behavior. To further verify the superiority of the ITransformer-CNN, LSTM-CNN and LSTM are introduced for comparative experiments. The DGA dataset and MDND are designed specifically for malicious DNS detection. Each dataset contains about 10,000 domain name records. These records are described in detail by features such as domain name character structures, query patterns, and associated IP addresses. About 60% of the DGA dataset are malicious domain names generated by malware of different families. MDND covers various attack types, including phishing site domain names (25%), C&C server domain names (35%), spam domain names (15%) and malicious advertising domain names (25%). The DGA dataset focuses on malicious domain names generated by different families of malware, which is essential for evaluating the ability of models to identify complex malicious behaviors. MDND covers different attack types, which helps to comprehensively verify the generalization ability and adaptability of the model. In contrast, although datasets such as CIC-Bell-DNS and Caida also contain rich DNS traffic data, they focus more on general network traffic analysis. The results are shown in Fig. 11. Accuracy performance of four algorithms in DGA dataset and MDND. Figure 11 showcases the accuracy performance of the four algorithms in the DGA dataset and MDND. In Fig. 11a, on the DGA dataset, the convergence rate of the four object detection algorithms was not significantly different, and they all tended to converge after approximately 80 iterations. However, there were significant differences in the accuracy of the four algorithms during convergence. The ITransformer-CNN model had the highest convergence accuracy, reaching 95.7%, while the LSTM algorithm had the lowest convergence accuracy, reaching only 78.4%. This indicates that although the convergence rate of the four algorithms was similar, their convergence points during the training process were significantly different. From Fig. 11b, the accuracy of the ITransformer-CNN detection algorithm in MDND was relatively stable without significant fluctuations. In contrast, the accuracy performance of the other three algorithms varied significantly, especially with significant fluctuations at certain iterations. This indicates that the robustness and performance stability of the ITransformer-CNN detection algorithm are more prominent among the four algorithms. Application experiment of Itransformer-CNN model To test the application effect of the proposed ITransformer-CNN detection algorithm in actual malicious DNS detection, the experiment applied the algorithm to real malicious DNS detection tasks. Then, the experiment uses a dataset containing known malicious and normal DNS traffic as the test set. To better evaluate the performance, the test set confusion matrix thermal diagram is used to display the prediction accuracy of multi-classification results and the confusion situation between different categories. This thermal diagram can clearly demonstrate the predictive performance of the model for different categories. In the thermal diagram, the depth of color demonstrates the prediction accuracy of the model, where dark indicates higher accuracy and light indicates lower accuracy. According to the heat map, it is possible to distinguish between categories with good prediction performance (such as spam) and categories with relatively poor prediction performance (such as malware and phishing). The relevant outcomes are displayed in Fig. 12. Model test set confusion matrix thermal diagram. Figure 12 shows the confusion matrix thermal diagram for four categories: benign samples, spam, malware, and phishing. The figure shows that the ITransformer-CNN detection algorithm performs well in terms of accuracy. Especially for the spam, the results of this model were all correct. However, in terms of detecting phishing software, the model encountered six false positives, which means that some phishing software is misjudged as belonging to other categories. In addition, for the malicious software, the model had 15 false positives, which means that some malicious software has not been correctly identified. Overall, the ITransformer-CNN detection algorithm performs well in most cases, but further improvements are necessary to improve its detection capabilities for phishing and malware. To verify the superiority of the model in the application, the Botnet detection method based on DNS rules method proposed in literature7 (DNS-BD), the CNN-based DNS detection algorithm in literature8 (CNN-DNS), and DNS detection method based on real-time deep network in literature10 (RTDN) are compared with the proposed ITransformer-CNN. The above four algorithms are trained with CIC-Bell-DNS and then applied to a network security project in Shanghai. The experimental results are shown in Table 4. Table 4Comparison experimental results of ITransformer-CNN.Experimental groupAccuracy (%)Recall (%)F1 score (%)Training time (min)ITransformer-CNN95.6783.2188.4365.32DNS-BD90.1288.3479.2250.21CNN-DNS88.5676.4577.4948.65RTDN89.8977.6778.7752.12 Comparison experimental results of ITransformer-CNN. From the table data, the ITransformer-CNN showed certain advantages in various indicators. Its accuracy was as high as 95.67%, which was about 5.55% higher than DNS-BD’s 90.12%. The recall rate was 83.21%, which was lower than the 88.34% of DNS-BD. The F1 value of ITransformer-CNN was 88.43%, which was higher than DNS-BD’s 79.22%. Compared with CNN-DNS and RTDN, the advantages were more obvious. The main reasons why ITransformer-CNN has high accuracy and F1 score are as follows. First, it adopts an improved Transformer combined with CNN structure, which can more comprehensively and efficiently extract malicious DNS features. The improved Transformer better represents absolute position information through OHE encoding and utilizes MHAM multi-attention mechanism to learn different mapping spaces, enhancing its ability to process sequence information. The CNN part extracts traffic information features through one-dimensional convolution and maximum pooling operations, reducing over-fitting and improving generalization ability. The second is to adopt a block-based ensemble classifier, where multiple classifiers complement each other to improve the detection accuracy and robustness. The training time is slightly longer but still acceptable, as its complex structure increases computational complexity while improving performance. Twenty experts in network security are recruited for the experiment to rate the system’s detection accuracy, detection time, feature extraction ability, and overall performance. The scoring results for each aspect of the system are shown in Fig. 13. ITransformer-CNN detection algorithm performance scores in all aspects. In Fig. 13, the ITransformer-CNN detection algorithm received high evaluations in all aspects. Experts were satisfied with the overall performance of the system, with an average detection accuracy score of 95.8 points, indicating that the algorithm can accurately identify most malicious DNS traffic. In addition, the average score of detection time was 96.8 points. This indicates that the execution speed is relatively fast and can process a large amount of data in a short time. Meanwhile, the average score of the feature extraction ability was 96.3 points, indicating that the algorithm can effectively extract useful feature information from DNS traffic, thereby accurately classifying different types of malicious DNS traffic. Finally, the algorithm achieved an overall performance score of 97.6 points, indicating excellent performance in all aspects. Conclusion This study improved the Transformer by adjusting its attention head and encoding method. Additionally, it introduced a malicious DNS detection model that combined the improved Transformer with CNNs. In the performance testing of the algorithm, as the number of attention heads grew, the training accuracy of the model improved rapidly. When the number of heads reached 5, the accuracy of the model began to fluctuate and then rapidly increased again. However, when the number of heads increased to 10, the model achieved the highest accuracy, reaching 97.39%. Afterwards, the change of the model gradually flattened out. In addition, the number of optimal memory heads varied among different encoding methods, and there were also significant differences in accuracy. The optimal number of memory heads for OHE encoding was 10, with a corresponding accuracy of 96.64%. The OHE encoding method proposed in the study had the lowest detection time, with an average of 4965 s, which was 32.3% less time-consuming than traditional TFE. Finally, the ITransformer-CNN model was applied for research. Relevant outcomes showed that the ITransformer-CNN detection model performed well in terms of accuracy. Especially for the spam, the results of this model were all correct. However, in terms of detecting phishing software, the model encountered six false positives, which means that some phishing software is misjudged as belonging to other categories. In addition, for the detection of malicious software, the model had 15 false positives, which means that some malicious software has not been correctly identified. In summary, the ITransformer-CNN detection algorithm performs well in most cases. However, it requires further improvements to enhance its detection capabilities for phishing and malware. Author contributions Heyu Li revised the manuscript and made a significant contribution to the revised manuscript. Zhangmeizhi Li collected the samples. Shuyan Zhang analysed the data. Xiao Pu conducted the experiments and analysed the results. All authors discussed the results and wrote the manuscript. Data availability All data generated or analyzed during this study are included in this article. Further enquiries can be directed to the corresponding author. Declarations Competing interests The authors declare no competing interests. Footnotes Publisher’s note Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations. Heyu Li and Zhangmeizhi Li contributed equally to this work. Contributor Information Heyu Li, Email: lhy18844033000@sina.com. Zhangmeizhi Li, Email: lzmzcupk@126.com. References Associated Data Data Availability Statement All data generated or analyzed during this study are included in this article. Further enquiries can be directed to the corresponding author.