UAV-Assisted Resilience in 6G and Beyond Network Energy Saving: A Multi-Agent DRL Approach Dao Lan Vy Dinh 1 , Anh Nguyen Thi Mai 1 , Hung Tran 1 , Giang Quynh Le Vu 1,6 , Tu Dac Ho 2 , Zhenni Pan 3 , Vo Nhan Van 4 , Symeon Chatzinotas 5 , Dinh-Hieu Tran 5 Abstract This paper investigates the unmanned aerial vehicle (UAV)-assisted resilience perspective in the 6G network energy saving (NES) scenario. More specifically, we consider multiple ground base stations (GBSs) and each GBS has three different sectors/cells in the terrestrial networks, and multiple cells are turned off due to NES or incidents, e.g., disasters, hardware failures, or outages. To address this, we propose a Multi-Agent Deep Deterministic Policy Gradient (MADDPG) framework to enable UAV-assisted communication by jointly optimizing UAV trajectories, transmission power, and user-UAV association under a sleeping ground base station (GBS) strategy. This framework aims to ensure the resilience of active users in the network and the long-term operability of UAVs. Specifically, it maximizes service coverage for users during power outages or NES zones, while minimizing the energy consumption of UAVs. Simulation results demonstrate that the proposed MADDPG policy consistently achieves high coverage ratio across different testing episodes, outperforming other baselines. Moreover, the MADDPG framework attains the lowest total energy consumption, with a reduction of approximately 24% compared to the conventional all GBS ON configuration, while maintaining a comparable user service rate. These results confirm the effectiveness of the proposed approach in achieving a superior trade-off between energy efficiency and service performance, supporting the development of sustainable and resilient UAV-assisted cellular networks. I Introduction The rapid proliferation of mobile devices and and the shift toward 6G and beyond networks have driven an unprecedented surge in energy consumption. This escalating energy demand not only increases operational expenditure for mobile network operators but also amplifies the environmental impact through higher carbon emissions. Consequently, improving the energy efficiency (EE) and sustainability of wireless infrastructures has become a central design goal for next-generation networks. Beyond energy concerns, modern communication systems must remain resilient to failures caused by natural disasters, backhaul congestion, or equipment malfunctions, necessitating rapid and reliable service recovery under dynamic conditions [ 2 ] . Unmanned Aerial Vehicles (UAVs) have recently emerged as key enablers for green and resilient communications due to their high mobility, adaptive deployment, and strong line-of-sight (LoS) connectivity. When functioning as Aerial Base Stations (ABSs), UAVs can promptly compensate for service disruptions in Cell Outage Compensation (COC) scenarios or temporarily replace low-traffic Ground Base Stations (GBSs) placed in sleep mode to reduce network energy consumption [ 3 ] . However, jointly determining UAV trajectories, transmit power levels, and the on/off states of GBSs constitutes a complex, non-convex, and temporally coupled optimization problem that challenges traditional model-based methods. Several works have studied UAV-assisted energy-efficient (EE) architectures via classical optimization. Chang et al. [ 12 ] propose a UAV-GBS scheme for small cells, using a heuristic integer linear program (ILP) to jointly schedule GBS sleep and place UAVs, improving EE while preserving coverage. Alsharoa et al. [ 13 ] optimize a three-tier heterogeneous network (HetNet) with solar-powered drone cells via a per-block binary program, reducing total energy through joint drone deployment and dynamic GBS activation under minimum received-power (QoS) constraints. Li et al. [ 3 ] formulate a unified problem over GBS sleep states, UAV trajectories, and transmit powers; their block-coordinate descent with successive convex approximation (SCA) and branch-and-bound balances flight, transmit, and GBS operational energy. Gaddam et al. [ 14 ] analyze ground radio station infrastructures for UAV links, showing throughput–coverage–EE trade-offs under coordinated scheduling and sleep control. Collectively, these studies have demonstrated the potential of integrating UAV deployment and GBS sleep mechanisms to achieve greener communication. However, their reliance on deterministic or heuristic optimization techniques, such as ILP, SCA, or block-coordinate methods, limits their scalability and adaptability to highly dynamic network environments. Moreover, none of the above frameworks incorporate learning-based mechanisms that allow agents to autonomously adapt to temporal traffic variations, user mobility, and stochastic wireless conditions. In contrast, deep reinforcement learning (DRL) has emerged as a powerful approach for complex, non-convex control in wireless networks. Liu et al. [ 4 ] use DRL for energy-efficient UAV coverage control, and Wang et al. [ 5 ] apply Deep Deterministic Policy Gradient (DDPG) to UAV-assisted computation offloading, illustrating DRL’s ability to learn adaptive policies. Likewise, [ 1 ] shows that Deep Q-Learning (DQN) can tune GBS configurations for energy savings while preserving QoS. However, few works jointly leverage UAV-assisted architectures and DRL to co-optimize UAV mobility, transmit power, and GBS sleep scheduling within a unified green-communication framework. To address these limitations, this paper makes four contributions in a single unified framework. First, to the best of our knowledge, it is the first to study UAV-assisted resilience for 6G-and-beyond NES using multi-agent DRL. Second, we model a multi-cell radio access network with scheduled inactive cells, distributed users, and user rate demands with occasional sudden surges. To ensure these surges reflect realistic usage, the traffic patterns are synthesized based on the characteristics of the open 5G dataset from Choi et al. [ 15 ] , which contains real-world traffic from diverse applications such as video conferencing (e.g., Google Meet, MS Teams) and video streaming. Under this model, we aim to maximize user coverage while minimizing UAVs’ energy consumption to ensure both network resilience and UAV sustainability. Third, we employ a Multi-Agent Deep Deterministic Policy Gradient (MADDPG) approach with centralized training and decentralized execution to jointly optimize UAV trajectories, transmit power, and user–UAV association. Finally, extensive simulations show that, compared with benchmark schemes, our learning-driven method offers superior adaptation, scalability across heterogeneous environments, and robustness to unpredictable network dynamics. II System Model We considers a downlink communication network in which UAVs provide service to ground users when nearby cells are temporarily turned off to save energy or due to power outages or failures. The goal is to simultaneously optimize the UAV trajectory and transmit power to maximize user coverage while minimizing the UAV’s energy consumption. As shown in Fig. 1 , each GBS consists of three sectors or cells. Each cell serves the associated mobile users within its coverage area. When the cells encounter problems such as hardware failures, natural disasters, or energy-saving problems, but the traffic demand from mobile users in cells suddenly increases unexpectedly, turning on the cells when they are in turn-off/deep sleep modes will take many minutes. Even when turned on, these cells are still overloaded and cannot serve all users in time. Therefore, we use UAVs acting as ABSs to serve users in a timely manner. We design the scheduling so that users are allocated to the UAV with the best SINR. This ensures that each user is connected to the most favorable serving UAV based on the instantaneous channel quality and the UAV’s location. II- 1 Cells There are K fixed cells, each is a hexagonal cell. The cell operational state is given by s_{k}(t)\in\{0,1\} , where 1 denotes ON and 0 denotes OFF. The cell ON/OFF schedule is predetermined during data synthesizing process and provided as input. The set of inactive cells requiring UAV assistance at time t is: A(t)=\{k|s_{k}(t)=0\} . II- 2 UAVs There are N UAVs in the network with each UAV i flies at a fixed altitude H with position q_{i}(t)=[x_{i}(t),y_{i}(t)] . UAVs can move within a bounded area, with movement constrained by: ||\Delta q_{i}(t)||\leq v_{max}. (1) Each UAV has limited transmit power: 0\leq P_{i}(t)\leq P_{max}. (2) The total system transmit power throughout flying time must satisfy: \sum_{i}P_{i}(t)\leq P_{max}. (3) II- 3 Users M users are distributed randomly within the coverage area. Each user j has a fixed location (x_{j},y_{j}) and a heterogeneous, dynamic rate requirement R_{req,j} . II-A Communication Model The formulation for the communication model, including the distance, Rician fading channel, SINR, and achievable data rate (Eqs. ( 4 ) – ( 9 )), is adopted from the model presented in [ 10 , 11 ] . Given the location (x_{j},\ y_{j},\ 0) of each user j\in M at ground level and the current UAV location (x_{i},\ y_{i},\ H) in time slot n, the distance d_{ij} between the UAV and the user is: d_{ij}(t)=\sqrt{H^{2}+(x_{i}-x_{j})^{2}+(y_{i}-y_{j})^{2}}. (4) The channel between the user j and the UAV i follows a Rician fading channel model with a factor K, where the channel coefficient h_{ij} can be written as: h_{ij}(t)=\sqrt{w_{ij}(t)}.\tilde{h}_{ij}(t), (5) where w_{ij}(t) represents for the large-scale fading effects and h_{ij}(t) accounts for Rician small-scale fading coefficient. Specifically, w_{ij}(t) can be modeled as: w_{ij}(t)=w_{0}.d_{ij}^{-\alpha}(t), (6) where w_{0} is the average channel power gain at the reference distance d=1m , and \alpha\geq 2 is the path loss exponent for the Rician fading channel. The small scale fading \tilde{h}_{ij} with an expected value \mathop{\mathbb{E}}[|\tilde{h}_{ij}^{2}|]=1 , is given by: \tilde{h}_{ij}(t)=\sqrt{\frac{G}{1+G}}\bar{h}_{ij}(t)\ +\sqrt{\frac{1}{1+G}}\hat{h}_{ij}(t), (7) where \bar{h}_{ij}(t) and \hat{h}_{ij}(t)\sim\mathcal{CN}(0,\,1) denote the deterministic LoS and non-line-of-sight (NLoS) component (Rayleigh fading) during time slot t, respectively; G is the Rician factor. The SINR of each user j is: \gamma_{ij}(t)=\frac{P_{i}(t).|h_{ij}(t)|^{2}}{\sigma^{2}+\sum_{k\neq i}P_{k}(t).|h_{kj}(t)|^{2}}\ , (8) where P_{i}(t) is the power of the UAV i at the timeslot t, \sigma^{2} is the thermal noise power which is linearly proportional to the allocated bandwidth, interference includes other UAVs and active GBSs. Thus, the data rate achievable for user j when served by UAV i at time t is: \displaystyle R_{ij}(t)=B\log_{2}(1+\gamma_{ij}(t)), (9) where B is the frequency range allocated to this communication link. Each user is associated with the UAV that provides the highest SINR: a_{ij}(t)=\begin{cases}1,&\text{if }i=\displaystyle\arg\max_{k}\gamma_{kj}(t),\\ 0,&\text{otherwise.}\end{cases} Each user can be served by at most one UAV: \sum_{i}a_{ij}(t)\leq 1,\quad\forall j. (10) To ensure satisfactory communication quality, each user must achieve a minimum data rate requirement R_{ij}(t)\geq R_{\mathrm{req},j},\quad\forall j (11) where R_{\mathrm{req},j} denotes the required data rate threshold of user j . The total network throughput at time t is expressed as R_{\mathrm{total}}(t)=\sum_{i}\sum_{j}a_{ij}(t)R_{ij}(t). (12) The objective of the UAV communication system is typically to maximize R_{\mathrm{total}}(t) under the above user assignment and QoS constraints. II-B UAV Energy Model The energy consumption of UAV i at time t consists of both propulsion and communication components: E_{i}(t)=E_{\text{prop},i}(t)+E_{\text{comm},i}(t), (13) where E_{\text{prop},i}(t) and E_{\text{comm},i}(t) are formulated, respectively, as follows: \displaystyle E_{\text{prop},i}(t)=\alpha_{1}||\Delta q_{i}(t)||^{2}+\alpha_{2}||\Delta q_{i}(t)||, (14) \displaystyle E_{\text{comm},i}(t)=P_{i}(t)\Delta t, (15) with \Delta t is a timestep duration. Accordingly, the total energy of all UAVs is defined as: \displaystyle E_{\text{UAV}}(t)=\sum_{i=1}^{n}E_{i}(t). (16) II-C Cell ON/OFF Energy Model For each cell c , instantaneous cell power is modeled as: P_{c,t}=z_{c,t}(P_{\text{static}}+\Delta_{p}^{\text{cell}}\phi_{c,t}) with cell ON or OFF at timestep t : z_{c,t}\in\{0,1\} , load fraction: \phi_{c,t}\in[0,1] and load dependent slope: \Delta_{p}^{\text{cell}} as referred to [ 8 ] [ 9 ] . Episode energy per cell is calculated as: E_{c}=\sum_{t=1}^{T}P_{c,t}\Delta t . Network energy will include the static power drawn when any cell linked to the GBS is ON: E_{\text{total}}=\sum E_{c}+\sum E_{s} with E_{s}=\sum_{t}P_{s,t}\Delta t . II-D Problem Formulation The optimization objective of maximizing the users coverage while minimizing the total UAVs’ energy consumption by optimizing the UAVs’ trajectory, UAVs’ transmit power, and user-UAV association. Our problem is formulated as: \displaystyle\mathcal{P}_{1}: (17) \displaystyle\max_{\{q_{i}(t),P_{i}(t),a_{ij}(t)\}}\sum_{t=1}^{T}\sum_{i=1}^{N}\sum_{j=1}^{M}a_{ij}(t)R_{ij}(t)-\lambda\sum_{t=1}^{T}\sum_{i=1}^{N}E_{i}(t) (18) s.t. \displaystyle R_{ij}(t)\geq R_{\mathrm{req},j},\quad\forall i,j,t, (18a) \displaystyle\sum_{i}a_{ij}(t)\leq 1,\quad\forall j,t, (18b) \displaystyle a_{ij}(t)\in\{0,1\},\quad\forall i,j,t, (18c) \displaystyle||\Delta q_{i}(t)||\leq v_{max},\quad\forall i,t, (18d) \displaystyle 0\leq P_{i}(t)\leq P_{max},\quad\forall i,t, (18e) \displaystyle\sum_{i}P_{i}(t)\leq P_{max},\quad\forall t, (18f) where \lambda is a trade-off coefficient balancing coverage maximization and energy minimization. Constraint ( 18a ) ensures that each served user meets its minimum QoS requirement, while ( 18b ) guarantees that each user is associated with at most one UAV. Constraints ( 18d )-( 18f ) enforce UAV mobility and power limitations. III POMDP Formulation for UAV-Assisted Resilience in 6G NES III-A POMDP Formulation We model the control problem as a partially observable Markov decision process (POMDP) [ 6 ] . Each UAV i observes a local state s_{i}(t) while a global state s_{global}(t) is available for centralized training. III-A 1 Local State of UAV i \displaystyle s_{i}(t) \displaystyle=[q_{i}(t),P_{i}(t),q_{\text{other}_{i}}(t),P_{\text{other}_{i}}(t), (19) \displaystyle{(x_{j},y_{j},R_{req,j})}_{j\in U_{i}(t)},{s_{k}(t)}_{k=1}^{K}], where q_{i}(t) is position of UAV i at time t ; P_{i}(t) is transmit power of UAV i at timestep t ; q_{\text{other}_{i}}(t),P_{\text{other}_{i}}(t) are positions and powers of other UAVs; {(x_{j},y_{j},R_{req,j})}_{j\in U_{i}(t)} is information of 6 nearest users within UAV i coverage diameter; {s_{k}(t)}_{k=1}^{K} is states of all cells. III-A 2 Global State s_{\text{global}}(t)=[\{q_{i}(t),P_{i}(t)\}_{i=1}^{N},\{{(x_{j},y_{j},R_{req,j})}\}_{j=1}^{M},s_{k}(t)_{k=1}^{K}]. (20) III-A 3 Action of UAV i a_{i}(t)=[\Delta x_{i}(t),\Delta y_{i}(t),P_{i}(t)]. (21) III-B Reward Function The reward emphasizes coverage maximization and energy minimization, with coverage rate is measured based on successfully served UAV-needed users: \displaystyle R_{i}(t)=\omega_{1}\cdot\mathcal{R}^{C}_{i}(t)+\omega_{2}\cdot(1-\mathcal{P}_{i}^{E}(t)), (22) where \mathcal{R}^{C}_{i}(t) and \mathcal{P}_{i}^{E}(t) are calculated, respectively, as \displaystyle\mathcal{R}^{C}_{i}(t)=\frac{|\{j:R_{ij}(t)\geq R_{\text{req},j},\,s_{\text{cell},j}(t)=0\}|}{|\{j:s_{\text{cell},j}(t)=0\}|}, (23) \displaystyle\mathcal{P}_{i}^{E}(t)=\frac{E_{i}(t)}{E_{max}}, (24) with E_{max}=P_{max}\Delta t+\alpha_{1}v_{max}^{2}+\alpha_{2}v_{max} . III-C Objective Each UAV learns a policy \pi_{i}(a_{i}|s_{i}) that maximizes its expected cumulative discounted reward: \displaystyle\max_{\pi_{1},\pi_{2}}E\left[\sum_{t=0}^{T}\gamma^{t}R_{i}(t)\right]. (25) Fig. 1: System Model Illustration IV The Proposed CTDE-MADDPG Framework IV-A CTDE-MADDPG Framework We adopt a Centralized Training with Decentralized Execution (CTDE) variant of MADDPG for cooperative UAVs control with better memory and training efficiency as shown 2 . Each UAV is modeled as an agent with its own actor–critic pair. During training, each agent’s critic has access to the global state s_{\text{global}}(t) and actions of all agents, enabling coordinated learning. However, during execution, each UAV only uses its local observations s_{i}(t) through the actor network, making the system scalable and robust to partial observability. Fig. 2: CTDE-MADDPG Framework IV-B Network Architecture Both actor and critic networks are implemented as fully connected multilayer perceptrons (MLPs) with ReLU activations. Actor outputs continuous actions [v_{x},v_{y},P] normalized to valid ranges via tanh activation, with v_{x},v_{y}\in[-v_{max},v_{max}] and P\in[0,P_{max}] . Critic receives concatenated joint observations and actions. Optimization is performed with the Adam optimizer and gradient norms are clipped to prevent instability. Target networks are softly updated using Polyak averaging. IV-C Training Mechanism IV-C 1 Prioritized Experience Replay (PER) Standard experience replay samples transitions uniformly, which can be inefficient when most experiences are not informative. To improve sample efficiency and stability, we integrate a prioritized replay buffer based on the proportional prioritization scheme. Each transition’s sampling probability is determined by its temporal-difference (TD) error magnitude, stored in a sum-tree structure for efficient O(logN) sampling and priority updates. IV-C 2 Exploration Strategy Besides dynamically adjusting exploration through PER, we also add temporally correlated Ornstein-Uhlenbeck (OU) Noise to actions during training to encourage stable exploration: N_{t}=N_{t-1}+\theta(\mu-N{t-1})\Delta t+\sigma W_{t} with noise parameter \sigma is decayed by 0.9999 per step with a minium of 0.01 [ 7 ] . After substantial number of episodes, exploration noise is disabled to exploit the learned policy. This could prevents erratic movements that could violate physical constraints, and produces smooth, correlated actions suitable for UAV dynamics. IV-C 3 Interference Management The system accounts for inter-UAVs interference when computing SINR values. Each UAV’s transmission contributes to the interference perceived by users connected to other UAVs. This coupling is reflected in the environment dynamics and achievable rate calculation. The mechanism encourages UAV agents to learn cooperative spatial and power control strategies that balance coverage maximization and energy minimmization. V Results and Discussions V-A Reward-based Analysis Fig. 3 illustrates the convergence of the proposed MADDPG framework. The x-axis represents the Training Episodes (up to 25,000), and the y-axis is the normalized average episodic reward (per step). The average reward rises sharply during the initial exploration phase and stabilizes around 0.032. It exhibits a clear stepped improvement around the 5,000-episode mark, which coincides with the learning-rate decay. This jump is followed by a brief period of high-variance fine-tuning. The policy then settles into its final converged state of approximately 0.039, and the stable plateau from roughly 8,000 episodes onward confirms robust convergence. Fig. 3: Training convergence curve, showing the average reward per step over the training process. As shown in Fig. 4 , the learned MADDPG policy maintains the highest average reward across all timesteps, outperforming both the k-nearest neighbors (KNN)-Fixed and Random baselines. The Random policy quickly collapses toward zero reward, while the KNN-Fixed method remains myopic and less adaptive. In contrast, the MADDPG agents, through cooperative learning, appears to find a more refined balance in spatial positioning and power control. This ability to make better sequential decisions yields a consistent, albeit small, performance gain throughout the episode. Fig. 4: omparison of instantaneous reward per step, averaged over 15 testing episodes. V-B Coverage Performance Evaluation Fig. 5: Comparison of Total Coverage Ratio per Episode TABLE I: Energy consumption comparison Method \mathbf{E_{\text{UAV}}} \mathbf{E_{\text{cell}}} \mathbf{E_{\text{total}}} Served Users Saving vs. MADDPG (Wh) (Wh) (Wh) (%) (%) KNN + Fixed Power 34.29 86.37 120.66 76.20 23.78 MADDPG (Ours) 5.60 86.37 91.97 78.42 – Random 34.32 86.37 120.69 63.13 23.80 All Cell ON – 93.28 121.06 77.66 24.03 Fig. 5 illustrates the total coverage ratio per episode for the proposed MADDPG policy compared against two baseline methods. It can be seen from Fig. 5 that the MADDPG policy (blue line) achieves the highest total coverage ratio in all 15 testing episodes. This superior performance suggests that the agents have successfully learned a complex, cooperative strategy, optimizing their trajectories and power control to serve the maximum number of users in the sleeping cells. The KNN-Fixed policy (green line), which serves as a stable heuristic, is consistently outperformed by MADDPG. This performance gap can be explained by the policy’s myopic and non-cooperative nature. Lacking coordination, multiple agents can be independently drawn to the same dense user cluster. This fact results in redundant coverage in one area while neglecting users in other areas. The MADDPG agents, in contrast, appear to learn to divide this workload efficiently via the centralized training mechanism. Conversely, the Random policy (orange line) exhibits extremely poor and highly volatile performance, with its coverage ratio frequently dropping to zero. This result is expected, as its actions have no correlation with the system’s objective. The UAVs are just as likely to fly away from users as they are to fly towards them. V-C Energy Performance Evaluation Table I compares the overall energy consumption and user service performance of the proposed MADDPG framework against 3 baseline methods. The total energy consumption is decomposed into the UAV operational energy E_{\text{UAV}} and the energy consumed by active cell E_{\text{cell}} . The All Cell ON baseline represents the scenario in which all GBSs remain fully active at fixed transmission power, consuming the highest amount of energy ( 121.06 Wh). In contrast, other methods that involve UAV assistance or selective GBS activation demonstrate energy savings by dynamically adjusting coverage and transmission behavior. The KNN + Fixed Power and Random baselines consume over 120 Wh due to their static or non-optimized decision-making, leading to redundant UAV movement or inefficient transmission scheduling. Our MADDPG-based approach achieves the lowest total energy consumption ( 91.97 Wh) while maintaining a comparable user service rate ( 78.42\% ). This indicates that the trained agents effectively balance between activating UAVs and GBSs to minimize energy usage without sacrificing network service quality. Specifically, MADDPG reduces the total energy consumption by approximately 24\% compared to the All Cell ON configuration, and by around 23.8\% compared to the random policy, while serving a higher percentage of users. It is worth noting that the total served user percentage slightly differs from the coverage rate in the previous section. Here, the metric accounts for all users successfully served by either UAVs or active GBSs, whereas the coverage rate previously measured only the UAV-needed coverage proportion. Overall, the MADDPG-based coordination strategy achieves the best trade-off between energy efficiency and service performance, confirming its capability to dynamically adapt to traffic variations and power constraints in green UAV-assisted cellular networks. VI Conclusion This paper presents a UAV-assisted approach to green, resilient wireless networks using a MADDPG framework with CTDE. The method jointly optimizes UAV trajectories, transmit power, and user-UAV association under a sleeping-cell strategy to maximize user coverage while minimizing UAV energy; in simulations it outperforms conventional baselines, achieving the highest coverage and reducing total energy consumption by 24% versus an All-Cell-ON configuration. Future work includes lightweight policy distillation and transfer learning to speed convergence; integrating adaptive rate control, user prioritization, and dynamic bandwidth allocation to better satisfy per-user QoS; and jointly optimizing the GBS sleep schedule with UAV trajectories and transmit power. Overall, integrating MADRL with UAV-assisted architectures is a promising path toward sustainable, autonomous, and resilient 6G-era networks.