Anti-Jamming Communication Using Imitation Learning Authors: Zhanyang Zhou, Yingtao Niu, Boyu Wan, Wenhao Zhou, Shu-Chuan Chu, Erdem Koyuncu Published: 2023 ABSTRACT AbstractThe communication reliability of wireless communication systems is threatened by malicious jammers. Aiming at the problem of reliable communication under malicious jamming, a large number of schemes have been proposed to mitigate the effects of malicious jamming by avoiding the blocking interference of jammers. However, the existing anti-jamming schemes, such as fixed strategy, Reinforcement learning (RL), and deep Q network (DQN) have limited use of historical data, and most of them only pay attention to the current state changes and cannot gain experience from historical samples. In view of this, this manuscript proposes anti-jamming communication using imitation learning. Specifically, this manuscript addresses the problem of anti-jamming decisions for wireless communication in scenarios with malicious jamming and proposes an algorithm that consists of three steps: First, the heuristic-based Expert Trajectory Generation Algorithm is proposed as the expert strategy, which enables us to obtain the expert trajectory from historical samples. The trajectory mentioned in this algorithm represents the sequence of actions undertaken by the expert in various situations. Then obtaining a user strategy by imitating the expert strategy using an imitation learning neural network. Finally, adopting a functional user strategy for efficient and sequential anti-jamming decisions. Simulation results indicate that the proposed method outperforms the RL-based anti-jamming method and DQN-based anti-jamming method regarding solving continuous-state spectrum anti-jamming problems without causing “curse of dimensionality” and providing greater robustness against channel fading and noise as well as when the jamming pattern changes.Keywords: anti-jamming communication, spectrum decision, imitation learning, expert strategy 1. Introduction With the large-scale application of autonomous driving [1], Intelligent Vehicles [2], drone light formation show [3], Internet-of-Things [4] and other technologies, information technology is deeply integrated into our lives. As the foundation of information technology, wireless communication is easy to be affected by user interference and malicious jamming due to the openness of its channel, which reduces the efficiency of its actual deployment [5,6,7,8]. Addressing the impact of such issues is crucial in wireless communication systems [9]. Therefore, making timely and effective spectrum decisions for different jamming patterns is practical in the anti-jamming process of wireless communication systems. Researchers have proposed various anti-jamming methods, including the extensively studied intelligent anti-jamming technology based on machine learning. Table 1 shows a summary of related research, and mainly introduces the research status of intelligent anti-jamming technology based on federated learning (FL), meta-learning (Mate-L), reinforcement learning (RL), and deep reinforcement learning (DRL). To restore normal communication quickly and stably when the wireless communication system is faced with jamming, researchers use the method based on automatic control [10,11] to prevent jamming. To address the issue of local data secrecy during multi-agent anti-jamming communication, researchers employ federated learning (FL) [12,13,14] for anti-jamming purposes. To enable agents to learn how to learn during anti-jamming communication, researchers have utilized meta-learning [15,16,17] for anti-jamming. To facilitate anti-jamming decision-making as the state changes, researchers employ reinforcement learning (RL) [18,19,20] for anti-jamming communication. To mitigate the “curse of dimensionality” problem associated with RL, researchers have extensively investigated the use of DQN [21,22,23] for anti-jamming communication. DQN has shown promise in the field of anti-jamming decision-making due to its ability to make decisions while learning and adapting to new jamming scenarios. Building upon this foundation, scholars worldwide have proposed improved methods such as multi-agent layered Q-learning (MALQL) and UCB-DQN [23]. Table 1Summary of Related Research.ReferenceYearResearch AreaTechnical SchemeMethodWeakness[10]2020LTE & LoRaAutomated LaboratoryMeasurementAutomatic ControlMethodHistorical experienceis underutilized[11]2020IoTAutomated Measurement[12]20235GFDRLFLPerformance issacrificed for security[13]2022Secrecy-driven FLFL[14]2022FANETAFRL[15]2021BeamformingMeta LearningMeta-LHigh computingpower requirements[16]2022Image ClassificationMGML[17]2020Jamming RecognitionMeta Learning[18]2021Wireless sensor networksJMAARLCurse of dimensionality; Unable to cope with continuous-state Spaces[19]2022UAV Anti-JammingCommunicationCMRL[20]2022Deceiving-basedanti-jamming methodsReinforcement Learning[21]2018Frequency Selectionof HF CommunicationDeep Reinforcement LearningDRLLow sampling efficiency;Unsteadiness[22]2023AutonomousVehicle NetworksDeep Reinforcement Learning[23]2022UAV Swarm NetworkUCB-DQNLTE: The long term evolution of Universal Mobile Telecommunications System; LoRa: Long Range Radio; FDRL: Federated Deep Reinforcement Learning; AFRL: Adaptive Federated Reinforcement Learning; MGML: Momentum Group Meta-Learning; JMAA: Joint Multi-agent Anti-jamming Algorithm; CMRL: Collaborative Multiagent Reinforcement Learning; UCB-DQN: Upper Confidence Bound Deep Q Network; FL: Federated learning; Mate-L: Mate learning; RL: Reinforcement Learning; DRL: Deep Reinforcement Learning. Summary of Related Research. LTE: The long term evolution of Universal Mobile Telecommunications System; LoRa: Long Range Radio; FDRL: Federated Deep Reinforcement Learning; AFRL: Adaptive Federated Reinforcement Learning; MGML: Momentum Group Meta-Learning; JMAA: Joint Multi-agent Anti-jamming Algorithm; CMRL: Collaborative Multiagent Reinforcement Learning; UCB-DQN: Upper Confidence Bound Deep Q Network; FL: Federated learning; Mate-L: Mate learning; RL: Reinforcement Learning; DRL: Deep Reinforcement Learning. The anti-jamming method based on automatic control can quickly and effectively deal with the current jamming and make appropriate anti-jamming actions, but this method cannot accumulate historical experience and cannot effectively predict the state change before anti-jamming. The intelligent anti-jamming communication methods are based on FL sacrifice performance for the sake of data security, and the performance is limited by the communication efficiency between agents. The intelligent anti-jamming communication method based on Meta-L requires high computing power, which is not practical for anti-jamming communication. The intelligent anti-jamming communication method based on RL fails to address the problem of continuous-state space and faces the challenge of the exponential growth of “state-action” pairs, known as the “dimension disaster”. The intelligent anti-jamming communication methods based on DQN have low sampling efficiency for state samples and limited utilization of historical state information, which may result in considerable time required for convergence even in typical or similar jamming scenarios. Furthermore, DQN imposes high requirements on the neural network, and an inappropriate neural network can lead to instability in the intelligent anti-jamming communication system based on DQN. In our previous work, we mainly studied the intelligent anti-jamming communication technology based on RL [5,24] and DQN [25], improved some technical details, and combined these with UCB technology. However, in this process, we find that both RL and DQN are faced with the dilemma of exploration and utilization, which requires a lot of time to explore the environment, and the ability to explore the environment in the utilization stage is poor. If the interference style is switched at this time, the anti-jamming ability of the system will be more significantly affected. In practice, jammers often enhance the difficulty of jamming by switching jamming patterns or dynamically adapting parameters. Consequently, efficiently utilizing experience from typical or similar scenarios to make effective anti-jamming decisions has become a challenging problem. With the advancement of research on inverse reinforcement learning, the method of imitation learning [26,27,28] can be employed to make timely and effective anti-jamming decisions in the face of typical and similar scenarios. Therefore, we can obtain an efficient spectral decision-making method by analyzing historical samples, and take it as an expert strategy(ES), to form an expert trajectory(ET). Using imitation learning neural network(ILNN), we can obtain a user strategy capable of making efficient decisions, to significantly improve the response speed and robustness of decision-making. Unlike reinforcement learning, which needs to maintain a Q-table or DQN so that each state-action has its corresponding Q-value, the imitation learning method proposed in this paper treats the decision problem as a fitting problem of state-to-action selection probability. ILNN fits the action selection probability in the next time slot according to the state of several past moments. The system selects the anti-jamming action according to the selection probability. Therefore, there is no fear of the curse of dimensionality. Based on this view, this manuscript proposes the following spectrum anti-jamming method using IL for scenarios where varied conventional jamming appears alternatively: First, a proposed Expert Trajectory Generation Algorithm (ETGA) serves as the expert strategy. Anti-jamming decision-making is carried out on historical state samples (HSS). To enable the system to stably avoid jamming signals, the expert strategy is generated accordingly. First, a proposed Expert Trajectory Generation Algorithm (ETGA) serves as the expert strategy. Anti-jamming decision-making is carried out on historical state samples (HSS). To enable the system to stably avoid jamming signals, the expert strategy is generated accordingly. Next, Imitation Learning Neural Networks are proposed. It takes historical state samples as input and the expert strategy generated by ETGA as expected output. Through network training, a user strategy (US) can be obtained, enabling it to make decisions according to the state of several previous time slots. Next, Imitation Learning Neural Networks are proposed. It takes historical state samples as input and the expert strategy generated by ETGA as expected output. Through network training, a user strategy (US) can be obtained, enabling it to make decisions according to the state of several previous time slots. Finally, the user strategy is used as a function to obtain anti-jamming decision actions efficiently and sequentially based on the states of multiple previous timeslots in the anti-jamming process. Finally, the user strategy is used as a function to obtain anti-jamming decision actions efficiently and sequentially based on the states of multiple previous timeslots in the anti-jamming process. The remainder of this manuscript is organized as follows: Section 2 introduces the system model and problem formulation. Section 3 explains the method. Section 4 presents the simulation results. Finally, Section 5 summarizes the conclusions. 2. System Model and Problem Formulation This section presents the system model and problem formulation of this manuscript. The system comprises a transmitter, receiver, agent, and jammer. The problem formulation introduces the reward function and objective function discussed in this manuscript. 2.1. System Model The system model in this manuscript, depicted in Figure 1, considers a wireless communication scenario where multiple malicious jammers are present. The transmitter T sends the signal S(t) while the receiver R is being maliciously jammed by both the determined jammer and the dynamic jammer. The agent at the receiver perceives the state s, and learns and decides on the anti-jamming action a, and transmits it to the transmitter through an independent control link to modify the communication action [5]. System model. 2.2. Problem Formulation This subsection outlines the reward function and objective function for this problem. The reward function is dependent on the current state and the chosen action. The objective function aims to minimize the disparity between the reward obtained from the user strategy and the reward achieved from the expert strategy. 2.2.1. Reward Function The communication band is divided into N non-overlapping channels, and the state s is represented by a N×1 array where each element corresponds to the strength of the noise plus jamming signal si of each channel: Similarly, the action a is defined as a N∗1 array with only one element being 1 and the others being 0. The reward r obtained by selecting action a in state s is given by the following: where ε is the energy threshold for jamming plus noise. 2.2.2. Objective Function To solve the anti-jamming problem, we use imitation learning to learn from the expert strategy and minimize the difference between the user reward expectation (URE) and the expert reward expectation (ERE) under policy π. IL aims to learn tasks from expert strategy, which can extract information about actions and the surrounding environment, and learn the relationship between states and actions. For a given expert strategy πE, the expert reward expectation can be calculated according to this strategy μE=μ(πE). Generally speaking, expert strategy πE is often manifested as expert strategy, that is, the action switching process under the expert strategy πE. For m expert strategies a0i,a1i,a2i,⋯i=1m the expert reward expectation μE is calculated as follows: At the same time, for the user who adopts user strategy π, its user reward expectation (URE) can be expressed as follows: We expect the Agent to minimize the difference between the user reward expectation μ(π) and the expert reward expectation μE under policy π: 3. Method This section proposes an anti-jamming method using imitation learning based on the above system model for the system model in Section 2. First, the process structure of the designed anti-jamming method using imitation learning is given, then the Expert Trajectory Generation Algorithm is given, and finally, the design of imitation learning neural network is given. 3.1. Process Structure As illustrated in Figure 2, the process structure of the designed anti-jamming method using imitation learning is proposed for intelligent spectrum anti-jamming in the scenario described in Section 2. Method process structure. The first step is to obtain the state sample S through the perception of the RF environment. The expert strategy AE is then generated through the expert strategy, as shown in Algorithm 1, The expert strategy is generated posteriorly, allowing the algorithm to select the action at based on either the state change before or after the timeslot, maximizing the total communication reward of this trajectory. The resulting expert strategy AE can be expressed as which is an action sequence equal to the length of the training sample S timeslot. In anti-jamming practice, the network parameters θN are adjusted through learning and adjusting to obtain the maximum reward by observing historical jamming information and obtaining the next action through the function fθN• The ILNN is trained to imitate the expert strategy to obtain the user strategy, and the function f^θN• is used as the efficient decision function during the process, obtaining the next action at+1 through the following formula: Implementing this algorithm faces two challenges, which are addressed in the next section: Obtaining the expert strategy through the Expert Trajectory Generation Algorithm; Obtaining the expert strategy through the Expert Trajectory Generation Algorithm; Training the Imitation Learning Neural Network based on the expert strategy; Training the Imitation Learning Neural Network based on the expert strategy; Algorithm 1 Expert Trajectory Generation Algorithm (ETGA)Input: Time-frequency matrix RF (N*M).Output: Expert selection sequence ESS (1*M).initialize: ESS = zeros(1,M); Consecutive unjammed channel start timeslot S1 = 1;Consecutive unjammed channel end timeslot S2 = TL.while min(ESS) == 0 do J = zeros(N,1) for ii = 1:N do if sum(RF(ii,S1:S2) > 0) == 0 then J(ii) = S2+1; else J(ii) = find(RF(ii,S1:S2) > 0,1)+S1−1 end if end for [M,N] = max(J); ESS(S1:M−1) = N*ones(M−S1,1); S1 = M;end while Input: Time-frequency matrix RF (N*M). Input: Time-frequency matrix RF (N*M). Output: Expert selection sequence ESS (1*M). Output: Expert selection sequence ESS (1*M). initialize: ESS = zeros(1,M); Consecutive unjammed channel start timeslot S1 = 1; initialize: ESS = zeros(1,M); Consecutive unjammed channel start timeslot S1 = 1; Consecutive unjammed channel end timeslot S2 = TL. Consecutive unjammed channel end timeslot S2 = TL. while min(ESS) == 0 do while min(ESS) == 0 do J = zeros(N,1) J = zeros(N,1) for ii = 1:N do for ii = 1:N do if sum(RF(ii,S1:S2) > 0) == 0 then if sum(RF(ii,S1:S2) > 0) == 0 then J(ii) = S2+1; J(ii) = S2+1; else else J(ii) = find(RF(ii,S1:S2) > 0,1)+S1−1 J(ii) = find(RF(ii,S1:S2) > 0,1)+S1−1 end if end if end for end for [M,N] = max(J); [M,N] = max(J); ESS(S1:M−1) = N*ones(M−S1,1); ESS(S1:M−1) = N*ones(M−S1,1); S1 = M; S1 = M; end while end while 3.2. Expert Trajectory Generation Algorithm (ETGA) The Expert Trajectory Generation Algorithm (ETGA) is shown in Algorithm 1. The algorithm selects the trajectory according to the time-frequency state matrix of N∗M based on the expert strategy, ensuring no collision with the jamming channel and that the communication signal stays in the same channel as long as possible for stable and sequential communication. The expert strategy corresponding to the Expert selection sequence (ESS) in Algorithm 1 is 3.3. Imitation Learning Neural Network As shown in Figure 3, it is a schematic diagram of the network structure of the trained ILNN. The trained ILNN should predict the next maximum reward action based on the past state and action of the past d timeslots: Imitation Learning Neural Network. Here, the neural network fθ can be expressed as a mapping from the input state domain S to the output action domain A, that is, fθ:S→A. Considering each hidden layer as a transformation function, the neural network can be expressed as where g is the output layer transformation function, fi is the ith hidden layer, and L is the number of hidden layers, the hidden layer can be expressed as follows: where fi∘fi−1=fifi−1• and f1 to fL are the transformation functions of the hidden layer. The transformation function of the hidden layer and the output layer can be broken down into linear transformation and nonlinear transformation. The linear transformation is the product of the weight vector and the input vector plus the bias variable, and the nonlinear transformation is the nonlinear activation. Then fi• can be expressed as follows: Here, wi and bi are the weight vector and bias variable of the ith layer, respectively. The specific process of the network structure is expressed in the following: where H is the hidden layer output matrix, wIn1 is the hidden layer historical state weight parameter group, wIn2 is the hidden layer historical action weight parameter group, b1 is the hidden layer offset parameter group, wOut is the output layer weight parameter group, and b2 is the output layer offset parameter group. The state input of the network is the concatenation of the state vectors of the past d timeslots, represented as sI, While the action input is the concatenation of the state vectors of the past d timeslots, represented as aI, The activation function used is Tanh(•), due to its fully differentiable, antisymmetric, and symmetric center at the origin characteristics, making it a commonly used activation function of the hidden layer. The network training process is formalized in where θ is the network parameter group, including wIn1, wIn2, b1, wOut, b2. The network parameters are adjusted to minimize the gap between the reward of the network strategy and the expert strategy, obtaining the output result a^t of imitation learning. 3.4. Decision Making The output a^t of the ILNN is an N∗1 sequential array of decimal numbers. To reduce the influence of error on channel selection and make channels with similar values have similar selection probability, a^t is used as the selection probability of decision action after exponential compression and normalization, represented by where Pa^ is the selection probability of the anti-jamming action at obtained by channel selection according to Pa^ probability. 4. Simulation This section presents the simulation results of this manuscript from four aspects: jamming settings, simulation settings, performance experiments, and comparative experiments. The jamming settings part introduces a jamming environment where jamming appears in various patterns alternately. The simulation setting introduces the definition of several important concepts in this section. The performance experiments part demonstrates the impact of different numbers of hidden layers in the neural network and sample delays on the anti-jamming performance. The comparative experiments part compares the proposed method in this manuscript with a DQN-based anti-jamming method. However, our source code is unfortunately not available due to the requirements of the organization. We will actively communicate with any researcher who is interested in this paper and send us an email. 4.1. Jamming Setting In the scenario of this manuscript, the corresponding state samples are as follows. Taking M timeslots as a sample, each sample is divided into k parts with long random timeslot lengths and their jamming parameters are randomly generated. Then a jamming sample is given by: among them, there are k−1 time nodes, 1