Multi-Agent Reinforcement Learning-Based Computation Offloading for Unmanned Aerial Vehicle Post-Disaster Rescue Authors: Lixing Wang, Huirong Jiao Published: 2024 ABSTRACT Natural disasters cause significant losses. Unmanned aerial vehicles (UAVs) are valuable in rescue missions but need to offload tasks to edge servers due to their limited computing power and battery life. This study proposes a task offloading decision algorithm called the multi-agent deep deterministic policy gradient with cooperation and experience replay (CER-MADDPG), which is based on multi-agent reinforcement learning for UAV computation offloading. CER-MADDPG emphasizes collaboration between UAVs and uses historical UAV experiences to classify and obtain optimal strategies. It enables collaboration among edge devices through the design of the ’critic’ network. Additionally, by defining good and bad experiences for UAVs, experiences are classified into two separate buffers, allowing UAVs to learn from them, seek benefits, avoid harm, and reduce system overhead. The performance of CER-MADDPG was verified through simulations in two aspects. First, the influence of key hyperparameters on performance was examined, and the optimal values were determined. Second, CER-MADDPG was compared with other baseline algorithms. The results show that compared with MADDPG and stochastic game-based resource allocation with prioritized experience replay, CER-MADDPG achieves the lowest system overhead and superior stability and scalability. 1. Introduction Unmanned aerial vehicles (UAVs) are widely used in various fields, such as crime scene surveillance [1], habitat destruction assessment [2], crop monitoring [3], and vegetation mapping [4]. Recently, UAVs have been used in humanitarian rescue operations [5]. UAVs can be combined with artificial intelligence technology to improve the efficiency of natural disaster rescue efforts. UAVs are characterized by their low cost, high maneuverability, and flexible deployment, allowing them to easily enter disaster-stricken areas that are difficult to reach otherwise [6]. UAVs use onboard sensors and wireless communications to create digital maps of post-disaster areas and can be used as communication relays to effectively forward scheduling and commands of post-disaster rescue to ground rescue equipment [7]. Recent work has investigated the age of information (AoI)-minimal clustering, transmission, and trajectory co-design in UAV-assisted wireless-powered communication networks (WPCNs). In these networks, UAVs with limited battery capacity must manage energy-efficient clustering of islands and optimize their flight trajectories to ensure effective data transmission and long-term performance [8]. The use of UAVs in post-disaster rescue operations is subject to fundamental engineering challenges. On the one hand, UAVs have strict weight requirements, limiting the capacity of their onboard batteries [9]. On the other hand, some tasks have deadlines, and because of the limited computational capacity of UAVs, real-time applications, such as image processing and video streaming, often exceed the local data-processing capacity of UAVs. Therefore, UAVs must offload a large number of intensive computing tasks to nearby edge servers during post-disaster rescue to respond to rescue efforts in a timely manner and extend their lifecycle. These studies highlight the importance of designing intelligent offloading strategies to balance energy consumption and task performance in UAV-assisted systems. UAVs must consider the weighted sum of the task completion time and system energy consumption as the system overhead, and the goal of the joint offloading strategy is to minimize system utilization. Our motivation is to optimize the UAV’s computation offloading decisions to reduce system overhead. Minimizing system overhead can improve the overall task completion efficiency. System overhead includes both time and communication overheads. Therefore, the means to improve the overall task completion efficiency is to minimize time consumption and communication consumption, achieving the highest efficiency for completing computation offloading tasks. The use of multi-agent reinforcement learning methods to determine optimal offloading strategies for UAV computation offloading scenarios has emerged as a topic of significant research interest [10]. In this study, a mobile edge computing offloading system called the multi-agent deep deterministic policy gradient with cooperation and experience replay (CER-MADDPG) with multiple UAVs and servers is proposed, in which UAVs generate tasks with delay constraints and continuously send offloading requests to edge servers during their movement. We define the trade-off between the delay and energy consumption as the system overhead. To minimize the system overhead caused by the decisions made by the UAVs, a swarm of UAVs makes joint offloading decisions using CER-MADDPG. The main contributions of this study are summarized as follows:The UAV swarm makes joint offloading decisions by modeling the offloading environment, delegating offloading decision-making to UAVs, and considering the random mobility of edge server clusters, time-varying nature of channels, and signal blockage of UAVs. To avoid wasting server computing resources while ensuring the successful execution of user tasks, the system is modeled as an optimization problem that minimizes the system overhead using the weighted average of the task execution time and server energy consumption as the system overhead.The CER-MADDPG algorithm based on multi-agent reinforcement learning is proposed. The UAVs use this algorithm to obtain their next coordinates, select edge servers, and determine the task offloading ratios. The algorithm enhances the collaborative decisions between agents by considering both global and individual information. Additionally, the algorithm classifies historical experiences and learns according to different categories of experiences, allowing UAVs to fully utilize historical experiences in continuous decision-making, cooperation, and optimal joint decisions.Experiments were conducted using the PyTorch platform to simulate the proposed algorithm and compare it with MADDPG and stochastic game-based resource allocation with prioritized experience replays (SGRA-PERs), thereby verifying that the proposed algorithm is superior to other algorithms in terms of optimality, stability, and scalability. The UAV swarm makes joint offloading decisions by modeling the offloading environment, delegating offloading decision-making to UAVs, and considering the random mobility of edge server clusters, time-varying nature of channels, and signal blockage of UAVs. To avoid wasting server computing resources while ensuring the successful execution of user tasks, the system is modeled as an optimization problem that minimizes the system overhead using the weighted average of the task execution time and server energy consumption as the system overhead. The UAV swarm makes joint offloading decisions by modeling the offloading environment, delegating offloading decision-making to UAVs, and considering the random mobility of edge server clusters, time-varying nature of channels, and signal blockage of UAVs. To avoid wasting server computing resources while ensuring the successful execution of user tasks, the system is modeled as an optimization problem that minimizes the system overhead using the weighted average of the task execution time and server energy consumption as the system overhead. The CER-MADDPG algorithm based on multi-agent reinforcement learning is proposed. The UAVs use this algorithm to obtain their next coordinates, select edge servers, and determine the task offloading ratios. The algorithm enhances the collaborative decisions between agents by considering both global and individual information. Additionally, the algorithm classifies historical experiences and learns according to different categories of experiences, allowing UAVs to fully utilize historical experiences in continuous decision-making, cooperation, and optimal joint decisions. The CER-MADDPG algorithm based on multi-agent reinforcement learning is proposed. The UAVs use this algorithm to obtain their next coordinates, select edge servers, and determine the task offloading ratios. The algorithm enhances the collaborative decisions between agents by considering both global and individual information. Additionally, the algorithm classifies historical experiences and learns according to different categories of experiences, allowing UAVs to fully utilize historical experiences in continuous decision-making, cooperation, and optimal joint decisions. Experiments were conducted using the PyTorch platform to simulate the proposed algorithm and compare it with MADDPG and stochastic game-based resource allocation with prioritized experience replays (SGRA-PERs), thereby verifying that the proposed algorithm is superior to other algorithms in terms of optimality, stability, and scalability. Experiments were conducted using the PyTorch platform to simulate the proposed algorithm and compare it with MADDPG and stochastic game-based resource allocation with prioritized experience replays (SGRA-PERs), thereby verifying that the proposed algorithm is superior to other algorithms in terms of optimality, stability, and scalability. The remainder of the article is organized as follows: Section 2 reviews relevant related work. Section 3 describes the UAV edge server problem. Section 4 introduces an offloading decision model based on the CER-MADDPG algorithm. Section 5 presents an analysis of the performance of the CER-MADDPG algorithm using simulations, and Section 6 presents the conclusions. 2. Related Work Computation offloading is a core issue in edge computing. Zhang et al. [11] argued that the main solutions to computation offloading are tasks to be offloaded and offloading locations. Current offloading decisions often involve comprehensive delay, energy consumption, or user benefits as offloading objectives to meet the needs of real-time and low energy consumption. Because finding an optimal solution to the computation offloading problem is difficult, several studies have proposed algorithms based on searching for suboptimal solutions, such as metaheuristic-based algorithms. Among these, the genetic algorithm is most widely used, which has low computational cost and can obtain solutions very close to the optimal solution in some cases. Li et al. [12] used the genetic algorithm to solve a scenario with multiple mobile devices and one edge server, dividing the tasks of edge devices proportionally to minimize the overall task completion time. Al-habob et al. [13] modeled the computation offloading scenario as an optimization problem with binary scheduling decision variables and designed a genetic algorithm to solve the optimization problem. Another approach for solving optimization problems is based on deep reinforcement learning algorithms. Hu et al. [14] modeled the offloading problem as a Markov decision process and designed an offloading strategy based on deep deterministic policy gradient to dynamically adjust the offloading ratio, maximizing the system overhead considering server energy consumption and task duration. Huang et al. [15] proposed an online offloading framework based on deep reinforcement learning, optimizing the task offloading decision of a single wireless device according to the time-varying wireless channel. Yang et al. [16] divided the offloading decision into discrete actions, jointly deciding on task offloading, wireless channel allocation, and image compression rate selection to achieve higher average image recognition accuracy and lower average processing latency. In some scenarios, single-agent reinforcement learning methods are not applicable. For example, in large-scale disaster relief, UAVs have different sensors and capabilities. Multi-agent reinforcement learning allows multiple agents to make independent decisions, with each agent responsible for solving specific sub-problems, thus better adapting to the decomposition and decision-making of complex tasks. Nguyen et al. [17] used MADDPG for joint data offloading in multiple independent edge clouds. Peng et al. [18] used the MADDPG method to solve the decision-making problem of UAV computation offloading with the goal of maximum reduction in the number of tasks. Lu et al. [19] considered minimizing the task failure rate and improving system utility as objectives to solve the task offloading problem of edge devices, based on the MADDPG algorithm. Huang et al. [20] proposed the cost-aware collaborative task-execution model, which specifically considers long-term reward indicators such as the number of lost tasks and uses the MADDPG method to coordinate multiple energy harvesting mobile devices in an EH-D2D network to execute computing tasks. Kumar et al. [21] treated each vehicle as an intelligent agent with decision-making capabilities and classified tasks generated by vehicles into local execution queues or task offloading queues using a method known as the Lyapunov-based MADDPG, which can minimize energy consumption while maintaining queue stability. However, these studies have shortcomings considering the cooperation between edge devices, which can lead to suboptimal joint decisions made by a group of edge devices. Insufficient experience can lead to a system that does not achieve an optimal solution. Generally, reinforcement learning algorithms require a series of iterations to achieve the optimal solution. Argerich et al. [22] proposed using external knowledge to guide agent decisions, which improves performance during training using expert knowledge from a database in the form of a programmable function. Chen et al. [23] stored the historical experiences of edge devices in a priority experience buffer, repeatedly learning from key historical experiences to improve the learning efficiency and obtain higher-quality solutions. Wu et al. [24] prioritized the historical experiences of each UAV in a tree-like manner, repeatedly learning from less frequent but important experiences to improve the system performance. However, these methods have shortcomings in the utilization of historical experiences of edge devices, which is also one of the reasons why the joint decisions made by a group of edge devices fall into local optima. In recent years, with the widespread application of unmanned aerial vehicles (UAVs) in fields such as post-disaster rescue and environmental monitoring, multi-UAV and multi-MEC (mobile edge computing) collaborative systems have become a hot research topic. Multi-UAV systems are capable of providing flexible computational and communication resources, particularly in scenarios where traditional cloud and edge computing capabilities are limited, effectively compensating for the lack of computational resources. Asif et al. [25] proposed a Joint Data Aggregation and Computation Offloading (JDACO) scheme, utilizing multiple UAVs as MEC servers to optimize data aggregation and task offloading processes, minimizing energy consumption and latency, showcasing UAVs’ potential in providing computational support for IoT devices in post-disaster scenarios. Zhang et al. [26] introduced a device-edge-cloud collaborative computing model that optimizes system delay and energy consumption, addressing the limited computational power of UAVs. The study employs a SAC-based reinforcement learning algorithm (STS-UDCO) to improve system convergence speed and stability, offering an effective solution for multi-UAV computation offloading. Wang et al. [27] presented a multi-UAV-assisted MEC system, incorporating Reconfigurable Intelligent Surfaces (RIS) to enhance communication performance and reduce system delay. By optimizing the computation offloading strategy and UAV trajectory, and employing a multi-agent deep reinforcement learning algorithm (MATD3), the study significantly improves system fairness and delay performance. Although these studies provide valuable insights into multi-UAV and multi-MEC systems for post-disaster rescue and resource allocation, they share a common limitation. While collaboration mechanisms and task offloading optimization are addressed in each study, they do not fully consider the integration of collaboration with the effective utilization of historical experiences. Specifically, existing approaches typically focus on optimizing a single objective, neglecting the potential of using intelligent experience replay mechanisms and collaborative decision-making to enhance overall system performance. Although several studies have attempted to use multi-agent reinforcement learning methods to solve the UAV computation offloading problem, we found the following:1.Most current works consider scenarios with a single server and multiple UAVs, whereas real scenarios often involve multiple servers and multiple UAVs.2.Existing research does not fully consider cooperative decision-making between UAVs. For example, when multiple UAVs observe the same environmental state in the same time slot, they make the same decisions. This homogenized decision-making is not necessarily a joint optimal decision, which may lead to joint offloading decisions made by a group of UAVs to fall into local optima.3.Existing studies have not sufficiently considered the utilization of historical UAV decision-making experiences. Current research focuses on prioritizing historical UAV experiences or introducing external knowledge bases. The UAV group cannot learn quickly from high-quality historical decision-making experiences and avoid poor experiences. This can also lead to joint offloading decisions made by a group of UAVs to fall into local optima. 1.Most current works consider scenarios with a single server and multiple UAVs, whereas real scenarios often involve multiple servers and multiple UAVs. Most current works consider scenarios with a single server and multiple UAVs, whereas real scenarios often involve multiple servers and multiple UAVs. 2.Existing research does not fully consider cooperative decision-making between UAVs. For example, when multiple UAVs observe the same environmental state in the same time slot, they make the same decisions. This homogenized decision-making is not necessarily a joint optimal decision, which may lead to joint offloading decisions made by a group of UAVs to fall into local optima. Existing research does not fully consider cooperative decision-making between UAVs. For example, when multiple UAVs observe the same environmental state in the same time slot, they make the same decisions. This homogenized decision-making is not necessarily a joint optimal decision, which may lead to joint offloading decisions made by a group of UAVs to fall into local optima. 3.Existing studies have not sufficiently considered the utilization of historical UAV decision-making experiences. Current research focuses on prioritizing historical UAV experiences or introducing external knowledge bases. The UAV group cannot learn quickly from high-quality historical decision-making experiences and avoid poor experiences. This can also lead to joint offloading decisions made by a group of UAVs to fall into local optima. Existing studies have not sufficiently considered the utilization of historical UAV decision-making experiences. Current research focuses on prioritizing historical UAV experiences or introducing external knowledge bases. The UAV group cannot learn quickly from high-quality historical decision-making experiences and avoid poor experiences. This can also lead to joint offloading decisions made by a group of UAVs to fall into local optima. As discussed in this section, several architectures and technologies exist to solve the computation offloading problem, each with its own advantages and disadvantages. Among these, technology based on multi-agent reinforcement learning is the most promising. Therefore, we propose a UAV edge-computing offloading method based on multi-agent reinforcement learning, aiming to jointly optimize the task completion time and server energy consumption. Existing studies do not fully consider cooperative decision-making between UAVs or the full utilization of historical UAV decision-making experiences, resulting in suboptimal system overhead. To address these shortcomings, we propose the CER-MADDPG algorithm, which uses a neural network design to achieve cooperative decision-making between UAVs and guides the joint decision-making of a UAV group toward an optimal solution through the classification of historical UAV decision-making experiences. 3. Problem Description In this section, we introduce the UAV computation offloading system. The system is divided into the UAV and edge layers, and the movement of devices in both layers is modeled with the devices able to move in their respective layers. Information exchange can occur between UAVs and edge devices. The system comprises three sub-models of mobile edge computing: network communication, task computation, and task transfer. 3.1. MEC Environment Model The system comprises two layers of devices, as shown in Figure 1, namely the UAV layer and edge layer. In this structure, the devices in the edge layer provide computing resources for the UAV group, and the UAVs can fly within the system architecture. Our work focuses on effective offloading decisions of UAVs. Let the set of UAVs be N={1,2,⋯,n} and the set of edge devices be M={1,2,⋯,m}. After each UAV randomly generates a task, it determines the distance and angle of flight and offloading ratio of the task and selects an edge device to send the offloading request in each time interval. The corresponding device performs the offloading computations. The main components of the MEC environment model include the UAV mobility, edge server mobility, and UAV task generation models, as described below. 3.1.1. UAV Mobility Model Let the flight angle and flight distance of the UAV be θ and dist, respectively, and the current coordinates of the UAV be uavx,uavy. The position of the UAV in the next time slot is given by (1):(1)uavx=uavx+dist·cos(θ)uavy=uavy+dist·sin(θ) The energy consumption battery of the UAV flight to the next destination is given by (2). (2)battery=battery−r·fuav2·x·taskSize·s where fuav represents the computation frequency of the UAV, x represents the computation offloading ratio, taskSize represents the size of the UAV task, s represents the number of CPU cycles required to process one bit, and r represents the impact factor of the chip structure on the CPU processing. 3.1.2. Edge Server Mobility Model The current coordinates of the edge server are uex,uey. At each time step, it randomly moves one step. The possible coordinates of the edge server in the next time step are as follows:uex−1,ueyuex+1,ueyuex,uey−1uex,uey+1uex,uey 3.1.3. UAV Task Generation Model Each UAV generates task Tn in the initial state, including the task generation time, expiration time, and task size. In each time slot, each UAV selects an angle and distance to move and an edge server for the partial offloading of tasks. 3.2. Communication Model We assume that the wireless channel bandwidth for transmission from the UAV to the edge server is B Hz. Additionally, the transmission loss power is defined as pnoisy, and the reference channel gain when the distance between UAV and edge server is 1 m is defined as α0. We denote the uplink transmission power as puplink. The distance between the UAV and selected edge server was calculated based on the positions of the UAV and edge server as dist(uav,ue). The channel gain g(uav,ue) and uplink transmission rate transRate are given by Formulas (3) and (4), respectively. (3)g(uav,ue)=α0dist(uav,ue)2 (4)transRate=B·log21+puplink·g(uav,ue)pnoisy The communication delay ttr and communication energy consumption etr are given by Formulas (5) and (6), respectively, where taskSize represents the task size of UAV n, x represents the task offloading ratio, transRate is the uplink transmission rate obtained from (4), and P1 represents the power consumed for communication computation. (5)ttr=x·taskSizetransRate (6)etr=x·taskSizetransRate·P1 3.3. Computation Offloading Model 3.3.1. Local Model The tasks selected for local computation are provided with computing resources by the UAV. The local computation delay tlocal and local energy consumption elocal are given by Formulas (7) and (8), respectively. (7)tlocal=(1−x)·taskSizefuav·s (8)elocal=(1−x)·taskSizefuav·s·P2 where x is the offloading ratio, taskSize represents the task size of UAV n, fuav is the computing frequency of the UAV, s is the number of CPU cycles required to process one bit of the task, and P2 represents the power consumed for local computation. 3.3.2. Server Model The tasks selected for server computation are provided with computing resources using an edge server. The server computation delay tedge and server energy consumption eedge are given by Formulas (9) and (10), respectively. (9)tedge=x·taskSizefuav·s (10)eedge=x·taskSizefuav·s·P3 Therefore, the total time consumption tcom is given by Formula (11), and the total energy consumption ecom is given by Formula (12), where x is the offloading ratio, taskSize represents the task size of UAV n, fue is the computing frequency of the edge server, s is the number of CPU cycles required to process one bit of the task, and P3 represents the power consumed for the server computation. (11)tcom=maxttr+tedge,tlocal (12)ecom=etr+eedge+elocal The objective of the optimization problem for the optimal offloading decision is to minimize the system cost through rational decision-making of the UAV fleet. The system cost comprises the total time consumption tcom and total energy consumption ecom. The specific optimization problem is shown in Formula (13). MinimizeD=∑i=1N[λ1·max(ttri+tedgei,tlocali) (13)+λ2·max(etri+eedgei,elocali)] (13a)subjecttomax(ttri+tedgei,tlocali)maxr¯,PBlow The experiences of the UAV swarm are classified by the classifier, with high-quality experiences entering PB and poor-quality experiences entering NB. Both high- and poor-quality experiences enter the general experience buffer, RB. Experiences in the high-quality experience buffer PB were used to train the MINE network, whereas experiences in the poor-quality experience buffer NB were used to train the CLUB network. The overall reward for training the critic network comprises both the environmental reward and estimated mutual information value. 4.3. Training Process The training process using CER-MADDPG to solve the UAV computation offloading problem proposed in this study is as follows: The Algorithm 1 initialization is described in lines 1–3. First, the action networks, value networks, MINE, and CLUB were initialized for each UAV. Additionally, the experience buffers PB, RB, and NB, and finally, the positions of the UAVs and edge devices are initialized. Training starts in line 4, where each UAV observes its state sn from the environment. Each UAV obtains an action from the action network based on its current state. The UAVs then execute joint actions and receive the current reward rn and the next state sn′. The (s,a,r,s′) tuples of the UAVs are stored in their respective experience replay buffers, RB. The experiences of the UAV swarm are classified by the classifier to determine whether to store them in the high-quality experience buffer PB or the low-quality experience buffer NB. When there are sufficient experiences with RB, a small batch is sampled from the replay buffer. Based on this batch, the critic and actor networks are updated. When there are sufficient experiences in both PB and NB, experiences are extracted from both to update the MINE and CLUB networks. Algorithm 1 The CER-MADDPG Training Procedure in the UAV Computation Offloading SystemRequire: Replay buffer RB, positive buffer PB, negative buffer NB, time budget T, exploration probability ε, discount factor γ, update step ξ Ensure: The optimal policy πθn* 1:Initialize actor and critic networks for each UAV, MINE and CLUB.2:Initialize experience replay buffer RB, PB, and NB.3:Initialize the location of UAVs and edge servers.4:for episode=1 to M do5: Initialize the state s0←Ni,Dji=1,j=1N,M,t06: for t=1 to T do7: Select joint actions ut of UAVs based on policy πθi(oti)8: Execute joint actions ut9: Receive the team reward rt and the new state o′10: Store trajectory v = v={ut,ot,ut+1,rt} to RB.11: end for12: if rsum>maxPBlow,r¯ then13: Add v to PB.14: else15: Add v to NB.16: end if17: Update the actor and critic networks.18: Update MINE and CLUB with every k episodes.19:end for 1:Initialize actor and critic networks for each UAV, MINE and CLUB. Initialize actor and critic networks for each UAV, MINE and CLUB. 2:Initialize experience replay buffer RB, PB, and NB. Initialize experience replay buffer RB, PB, and NB. 3:Initialize the location of UAVs and edge servers. Initialize the location of UAVs and edge servers. 4:for episode=1 to M do for episode=1 to M do 5: Initialize the state s0←Ni,Dji=1,j=1N,M,t0 Initialize the state s0←Ni,Dji=1,j=1N,M,t0 6: for t=1 to T do for t=1 to T do 7: Select joint actions ut of UAVs based on policy πθi(oti) Select joint actions ut of UAVs based on policy πθi(oti) 8: Execute joint actions ut Execute joint actions ut 9: Receive the team reward rt and the new state o′ Receive the team reward rt and the new state o′ 10: Store trajectory v = v={ut,ot,ut+1,rt} to RB. Store trajectory v = v={ut,ot,ut+1,rt} to RB. 11: end for end for 12: if rsum>maxPBlow,r¯ then if rsum>maxPBlow,r¯ then 13: Add v to PB. Add v to PB. 14: else else 15: Add v to NB. Add v to NB. 16: end if end if 17: Update the actor and critic networks. Update the actor and critic networks. 18: Update MINE and CLUB with every k episodes. Update MINE and CLUB with every k episodes. 19:end for end for 5. Simulation, Results and Analysis In this section, we describe extensive simulations conducted to validate the performance of the proposed CER-MADDPG scheme in UAV computation offloading and edge–server resource allocation problems. 5.1. Simulation Environment and Parameter Setting We simulated an MEC environment using Python, where MEC servers were allocated alongside roads and some edge servers moved randomly in the communication area. In the simulation, all UAVs selected angles and distances to move within the communication range and edge servers for offloading requests based on the CER-MADDPG method. The simulation was performed using Python on a PyTorch platform. All simulation parameters used are shown in Table 1 and Table 2, which were used in our simulation experiments. Referring to the works [14,30,31,32,33,34], similar values are set in our simulations. The UAV’s computation frequency, fuav is set to 1.2 GHz. The task size, taskSize is randomly generated when adjusting parameters, with values ranging from 30 to 40 Mbits. The number of CPU cycles required to process one bit, denoted as s, is set to 1000 cycles/bit. The UAV’s initial battery level is set to 500,000 J. These values are based on typical UAV parameters. In our model, the location of the edge servers changes dynamically, simulating the potential movement of servers over time in a post-disaster environment. We use a simple linear mobility model to simulate server movement, where the servers can randomly select paths within a predefined area. The server’s movement speed is set to 1 m/s, and the server’s computation frequency, fue is set to 5 GHz. This model can be further improved in the future with more complex random walk models to adapt to more complicated environmental changes. 5.2. Results and Analysis 5.2.1. Selection of Hyperparameters Setting the hyperparameters is crucial in the CER-MADDPG algorithm, because different hyperparameters can affect the optimization, convergence, and stability of the algorithm. Experiments were conducted to determine the optimal hyperparameters for the proposed algorithm. In the CER-MADDPG algorithm, the learning rates of the actor and critic networks influence the training and updating of the neural networks. A high learning rate can lead to poor optimization and stability, whereas a low learning rate can result in poor optimization and slow convergence. In general, because the critic guides the update of the actor, the critic’s learning rate should be set slightly higher than that of the actor to ensure faster convergence. As shown in Figure 5, when the critic network’s learning rate is 1.2×10−5 and that of the actor network is 6×10−6, the reward increases with the number of iterations. However, the optimal solution here is not as good as that with a critic network’s learning rate of 1.2×10−6 and the actor network’s learning rate is 6×10−7. This is because a high learning rate causes both the critic and actor networks to update significantly, whereas the optimal solution requires smaller updates. When the learning rate for the critic network is 1.2×10−6 and that for the actor network is 6×10−7, the system’s cost can achieve a converged and stable optimal solution. Although a locally optimal solution was obtained after 170 rounds of training, a globally optimal solution could still be achieved by increasing the number of iterations. When the learning rate of the critic network is 1.2×10−7 and that for the actor network’s learning rate is 6×10−8, the system reward cannot reach the optimal solution, as the lower learning rates result in slower neural network updates, requiring more iterations to converge. Therefore, in this study, the optimal learning rate for the critic network is 1.2×10−6, and that for the actor network is 6×10−7. Additionally, it can be observed from the figure that after 100 rounds there was a significant change in the system cost. This is because at this point, the experience replay buffer is full, and the neural networks have sufficient useful information for training. The experience replay buffer affected the training time, optimality, and convergence of the algorithm. As shown in Figure 6, a local optimal solution was obtained with an experience replay buffer size of 1000 and 320 training rounds; however, a global optimal solution could not be achieved. This is because a small experience replay buffer can affect the extraction of data features, leading to an inability to learn the optimal policy. When the experience replay buffer was increased to 10,000, a globally optimal solution was achieved after 480 rounds. However, with an experience replay buffer of 100,000, the optimal solution could not be obtained, because the buffer was too large and the data could not be updated, thus preventing the attainment of the best solution. Additionally, the algorithm with an experience replay buffer size of 10,000 trains was three times faster than that with an experience replay buffer size of 100,000. Therefore, we selected an optimal experience replay buffer size of 10,000. The selection of the mutual information parameters α1 for the lower bound of the high-quality experience buffer and α2 for the upper bound of the poor-quality experience buffer is shown in Figure 7. The choice of mutual information parameters affects the optimality and stability of the algorithm. When α1 is chosen as 0.5 and α2 is chosen as 0.5, the system reward converges steadily to the optimal solution. When α1 is chosen as 0.9 and α2 is chosen as 0.5, the system converges to a local optimal solution after 220 rounds. When α1 is chosen as 0.5 and α2 is chosen as 0.9, the system reward first decreases and then increases, and there is no stable convergence to the optimal solution. This is because neither a dominant weight for the upper bound of the poor-quality experience buffer nor a dominant weight for the lower bound of the high-quality experience buffer leads to convergence to the optimal solution. We summarize the experimental results in Table 3. Through comparative experiments, it was verified that the experience buffer size should be set to 104, learning rate of the actor network should be set to 10−7, learning rate of the critic network should be set to 10−6, α1 should be set to 0.5, and α2 should be set to 0.5. 5.2.2. Performance Comparison and Analysis We compared the CER-MADDPG method with MADDPG and SGRA-PERs to demonstrate the effectiveness of our approach. Brief introductions to these two methods are presented below. MADDPG: A policy gradient algorithm in which each agent has actor and critic networks. The critic network can access the states and actions of other agents during training, whereas the actor network requires only its own information. Consequently, the critic network is trained centrally, whereas the actor network is executed in a distributed manner. MADDPG: A policy gradient algorithm in which each agent has actor and critic networks. The critic network can access the states and actions of other agents during training, whereas the actor network requires only its own information. Consequently, the critic network is trained centrally, whereas the actor network is executed in a distributed manner. SGRA-PERs: Utilizes a prioritized experience replay mechanism for each UAV’s experience buffer, allowing important but infrequent experiences to be repeatedly utilized for learning, thereby enhancing system utility. SGRA-PERs: Utilizes a prioritized experience replay mechanism for each UAV’s experience buffer, allowing important but infrequent experiences to be repeatedly utilized for learning, thereby enhancing system utility. As shown in Figure 8, with an increase in iterations, the proposed CER-MADDPG algorithm achieved the lowest system cost and optimal stability after 480 rounds. In comparison, the MADDPG algorithm achieved stable system utility after 460 rounds, and the SGRA-PERs achieved stable system utility after 520 rounds of training. However, the solutions obtained by these algorithms are locally optimal and do not achieve the best results. This is because the proposed CER-MADDPG considers the cooperation between agents more deeply and utilizes the classification of good and bad decision-making experiences of UAVs. The bidirectional guidance of joint decision-making by UAVs tends to exhibit good behavior and avoid bad behavior, thus achieving the best system utility. SGRA-PERs prioritizes historical experiences for each UAV, which leads to a lower system utility compared with that of MADDPG. It can guide UAVs to learn better strategies to some extent but still has limitations. Therefore, compared with other algorithms, the CER-MADDPG proposed in this study has advantages in terms of system cost and can obtain solutions with lower system costs. To further validate the performance of the CER-MADDPG algorithm, Figure 9 illustrates the relationship between the initial task size of the UAVs and time required to complete the tasks. As the initial task size of the UAVs increases, the time required to complete the tasks also increases. Regardless of the initial task size, the CER-MADDPG required the shortest time to complete the tasks compared to those of the MADDPG and SGRA-PERs. Additionally, as the initial task size of the UAVs increased, CER-MADDPG exhibited the smallest increase in completion time, indicating higher stability than that of the MADDPG algorithm. This is because the CER-MADDPG algorithm proposed in this study fully utilizes good and bad experiences, enabling the UAV group to make more reasonable joint decisions and demonstrating advantages in stability and effectiveness. We compared the system energy consumption of the four algorithms for different numbers of UAVs. We set the number of UAVs N to 2, 3, 5, 10, and 20. As shown in Figure 10, with an increase in the number of UAVs, the system costs of the different algorithms increase to some extent. The CER-MADDPG algorithm outperformed the other two algorithms in controlling system utility. This is because, as the number of UAVs increases, the total task processing time and energy consumption of the UAV group increase, resulting in an overall increase in system costs. The growth rate of CER-MADDPG was approximately constant, whereas those of MADDPG and SGRA-PERs were higher than that of CER-MADDPG. This is because CER-MADDPG utilizes an experience classification learning mechanism and considers both global and individual critic networks, enabling a more comprehensive consideration of the collaborative nature of joint decisions, thereby improving the system utility. In summary, the CER-MADDPG algorithm has significantly lower system costs than those of the other offloading methods, and this advantage increases with the number of UAVs. We summarize the experimental results in Table 4. As shown in Table 4, irrespective of the number of UAVs, the system overhead of CER-MADDPG is smaller than those of SGRA-PERs and MADDPG. Additionally, from the changes in the system overhead, it can be observed that as the number of UAVs increases, the system overhead of the CER-MADDPG becomes more stable than those of the SGRA-PERs and MADDPG. Irrespective of the initial task of the UAVs (30 Mbits or 1000 Mbits), CER-MADDPG has the shortest average completion time for UAVs compared to those of the SGRA-PERs and MADDPG. 6. Conclusions To address the limitations of UAV battery life and computing capabilities, this study considers a scenario with multiple UAVs and servers and proposes a UAV-assisted post-disaster rescue computation offloading scheme based on multi-agent reinforcement learning. The algorithm proposed herein delegates decision-making to UAVs, and is designed to consider cooperative decision-making among multiple UAVs. Additionally, the algorithm focuses on guiding UAVs to make decisions based on good experiences and avoid bad experiences, enabling them to make optimal joint offloading decisions and ultimately improving system performance. The simulation results demonstrate that the proposed CER-MADDPG algorithm achieves better effectiveness and stability in joint decision-making compared with SGRA-PERs and MADDPG, effectively reducing the system overhead. The CER-MADDPG can generate optimal strategies based on time-varying channel conditions and edge server states. Fault tolerance in UAV edge computing systems [35] is a new field that must be explored in the future. In this study, the UAVs, channels, and edge devices were assumed to be honest. In future research, we will consider security issues, improve the existing models, and design a model that is more suitable for practical applications.