Selective entropy-fused proximal policy optimisation with federated reinforcement learning for intelligent multi-UAV trajectory and communication optimisation Authors: M Velmurugan, T Velmurugan Published: 2026 ABSTRACT AbstractThe rapid evolution of 5G and emerging 6G networks has increased the demand for wireless communication systems that deliver high capacity, low latency, and adaptability. However, conventional terrestrial infrastructure remains costly and inflexible, particularly in dynamic or remote environments. This article develops a new Federated Reinforcement Learning (FRL)-based UAV communication system using Selective Entropy-Fused Proximal Policy Optimization (SEF-PPO) is proposed to enhance the performance of locally-on-policy learning in real-time decision-making environments. In contrast to existing digital twin or offline-trained deep reinforcement learning (DRL) methods, the proposed solution eliminates the need for replay buffers, thereby reducing memory and computational requirements for UAV platforms. UAVs learn collaboratively while preserving data privacy and maintaining robustness to non-IID user distributions through federated aggregation with a High-Altitude Platform (HAP). The framework integrates trajectory planning, user association, energy-efficient resource allocation, and handover management within a unified adaptive architecture. Experimental results demonstrate significant improvements in throughput, fairness, latency, and energy efficiency compared with baseline methods, including DMTD, DRL-EC3, and greedy and random algorithms. Overall, the proposed design enables scalable, energy-aware, and environment-responsive UAV coordination, offering a deployment-ready solution for next-generation wireless networks without requiring simulation-based pretraining.Keywords: Unmanned Aerial Vehicle (UAV), Federated Reinforcement Learning (FRL), Selective Entropy Fusion, Trajectory Optimization, Energy-efficient CommunicationSubject terms: Engineering, Mathematics and computing velmurugan.m2024@vitstudent.ac.in1, tvelmurugan@vit.ac.in2*. Introduction The rapid development of wireless communication technologies, including fifth-generation (5G) and emerging sixth-generation (6G) networks, has significantly increased the demand for ubiquitous, high-capacity, and low-latency wireless services. Nevertheless, the deployment and maintenance of terrestrial infrastructure remain costly and inflexible, particularly in challenging terrains, temporary event settings, and disaster-affected areas. To address these limitations, unmanned aerial vehicles (UAVs) have emerged as a promising solution for augmenting ground-based networks through their use as aerial base stations (UAV-BSs) or relay nodes This approach enables improved coverage, increased throughput, and rapid network recovery1,2. In contrast to fixed ground base stations, UAVs offer high mobility, cost-effectiveness, and the ability to establish reliable line-of-sight (LoS) links with user equipment (UEs)3. These characteristics make UAV-enabled communication networks particularly effective in dynamic scenarios such as emergency response operations, rural broadband deployment, and high-density urban environments4,5. By dynamically adjusting their three-dimensional (3D) positions, UAVs can respond to fluctuating user demands, balance traffic loads, and enhance overall quality of service (QoS). Despite these advantages, the highly dynamic and distributed nature of UAV networks introduces significant challenges. In particular, trajectory planning, energy-efficient communication, user association, and interference management must be jointly optimised across multiple UAVs operating within shared airspace6. Classical optimisation techniques, such as convex programming, struggle to address these non-stationary, high-dimensional problems in real time. Previous studies have applied a range of optimisation approaches, including convex and non-convex programming, mixed-integer linear programming (MILP), and Lagrangian relaxation, to jointly optimise power control and UAV trajectories7. Heuristic and metaheuristic methods, such as particle swarm optimisation (PSO), genetic algorithms (GA), and ant colony optimisation (ACO), have also been employed for load balancing and energy-efficient route planning in aerial networks8,9. Game-theoretic and Stackelberg models have further been explored to manage resource competition and enable cooperative spectrum sharing10. However, these conventional methods are generally limited by high computational complexity, slow convergence, and poor adaptability to dynamic or atypical scenarios. To address these limitations, deep reinforcement learning (DRL) has recently emerged as a promising approach for autonomous UAV control, enabling agents to learn environment-dependent policies through interaction with their surroundings11. However, current DRL-based UAV communication systems still exhibit significant limitations when deployed in real-world environments. Most existing approaches rely on offline training or digital twin simulations, which restrict adaptability and responsiveness in non-stationary conditions. In addition, conventional reinforcement learning algorithms (e.g. DQN and DDPG) require large replay buffers and substantial computational resources, rendering them impractical for energy- and memory-constrained UAV platforms. Furthermore, the presence of non-IID data across regions, arising from heterogeneous and non-independent user behaviour, complicates centralised training. At the same time, swarm-level coordination remains an open challenge due to limited communication bandwidth and privacy constraints12,13. In recent years, advances in large-scale intelligent models and cross-domain learning frameworks have been proposed to address complex, multimodal, and distributed decision-making problems. Vision–language–action models aligned with cognitive processes, such as CogVLA14, and semantic sparsification approaches, such as SemanticVLA15, highlight the importance of efficient representation learning and adaptive model routing in large-scale systems. Similarly, hyperscale multi-sensor environments leveraging large-model-driven data fusion have been explored, alongside the development of self-adaptive and self-healing mechanisms in distributed physical systems16,17. These developments underscore the importance of scalable, adaptable, and communication-efficient learning paradigms, thereby motivating the proposed federated and entropy-aware UAV learning architecture. To address these challenges, this article introduces a federated reinforcement learning (FRL) framework combined with a Selective Entropy-Fused Proximal Policy Optimisation (SEF-PPO) algorithm for real-time UAV-enabled wireless communication systems. In contrast to traditional digital twin or offline reinforcement learning (RL) approaches, the proposed method enables UAVs to learn directly from real-world interactions in a local, on-policy manner, without requiring replay buffers or pretraining. Through federated aggregation at a high-altitude platform (HAP), UAVs collaboratively optimise their policies while preserving data locality and ensuring user privacy. Motivation Enhancing fairness and performance in dynamic wireless networks requires the joint optimisation of UAV trajectories and personalised resource allocation. By integrating entropy into proximal policy optimisation, UAVs can adaptively reposition to strengthen line-of-sight links, thereby improving both system throughput and fairness. Personalised resource allocation further supports fairness by weighting users according to demand and channel conditions, reducing service imbalance. Together, these strategies enable scalable, energy-efficient connectivity aligned with emerging AI-driven approaches in 6G and beyond. Main contributions The key contributions of this article are as follows: We introduce Selective Entropy-Fused Proximal Policy Optimisation (SEF-PPO), a locally on-policy, entropy-augmented variant of proximal policy optimisation, designed for real-time UAV learning without the need for replay buffers or simulation-based pretraining. We introduce Selective Entropy-Fused Proximal Policy Optimisation (SEF-PPO), a locally on-policy, entropy-augmented variant of proximal policy optimisation, designed for real-time UAV learning without the need for replay buffers or simulation-based pretraining. UAVs learn online directly from the environment using federated reinforcement learning (FRL), in contrast to existing approaches that rely on digital twins, thereby reducing setup costs and improving adaptability to real-world conditions. UAVs learn online directly from the environment using federated reinforcement learning (FRL), in contrast to existing approaches that rely on digital twins, thereby reducing setup costs and improving adaptability to real-world conditions. The integration of FRL with SEF-PPO training results in a memory- and compute-efficient framework that is well suited to UAV constraints, as it avoids off-policy sampling and large buffer management. The integration of FRL with SEF-PPO training results in a memory- and compute-efficient framework that is well suited to UAV constraints, as it avoids off-policy sampling and large buffer management. Each UAV learns an individualised policy tailored to local user patterns, interference, and mobility, while also contributing to global policy aggregation via a high-altitude platform (HAP). This enables scalable cooperative behaviour among UAV swarms, even under non-IID user distributions and high network loads. Each UAV learns an individualised policy tailored to local user patterns, interference, and mobility, while also contributing to global policy aggregation via a high-altitude platform (HAP). This enables scalable cooperative behaviour among UAV swarms, even under non-IID user distributions and high network loads. The proposed model addresses trajectory planning, user association, energy-efficient resource allocation, handover management, and swarm coordination as a unified optimisation problem, rather than treating them as separate tasks. The proposed model addresses trajectory planning, user association, energy-efficient resource allocation, handover management, and swarm coordination as a unified optimisation problem, rather than treating them as separate tasks. The locally on-policy FRL training architecture enables continuous learning, making the system suitable for real-world environments with dynamic conditions, without requiring offline bootstrapping. The locally on-policy FRL training architecture enables continuous learning, making the system suitable for real-world environments with dynamic conditions, without requiring offline bootstrapping. Section organization The remainder of this paper is structured as follows. Section 2 reviews existing studies on UAV optimisation. Section 3 presents the system model and details the proposed trajectory optimisation algorithm. Section 4 describes the implementation and analyses the results. Section 5 discusses the findings, outlines limitations, and suggests directions for future work. Literature review With the rapid deployment of unmanned aerial vehicles (UAVs) in next-generation wireless networks, considerable research attention has been devoted to optimising trajectory planning, resource allocation, and communication reliability using machine learning (ML) and reinforcement learning (RL) techniques. Early optimisation-based approaches relied on convex models and heuristic methods to maximise coverage and energy efficiency. Liu et al.18 proposed a deep reinforcement learning (DRL)-based energy-efficient control framework for coverage and connectivity (DRL-EC3). Their approach employs two deep neural networks to model environmental dynamics and optimise an energy efficiency objective that accounts for communication distance, fairness, energy consumption, and connectivity. Similarly, Zhan et al.19 investigated UAV-assisted communication in a three-dimensional (3D) environment, where multiple UAVs serve multiple users. They proposed a multi-agent proximal policy optimisation (MAPPO)-based RL method for online UAV control, demonstrating improved cumulative rewards and faster convergence through adaptive trajectory and speed control. Extending this line of work, He et al.20 introduced a distributed 3D trajectory design framework to maximise system performance across multiple UAVs. Their DRL-based approach, termed DMTD, enables UAVs to explore optimal flight altitudes and coverage regions through iterative interaction with the environment. Ji et al.21 developed a UAV-enabled multicast system in which a UAV acts as a mobile transmitter, supported by a reconfigurable intelligent surface (RIS) to enhance service quality under power constraints. They formulated a joint optimisation problem involving UAV trajectory, RIS configuration, and beamforming, and proposed a multi-pass deep Q-network (BT-MP-DQN) algorithm. Simulation results demonstrated improved achievable rates and effective multicast performance, further enhanced by RIS deployment. Gong et al.22 applied heuristic optimisation to balance energy consumption and data queue sizes in multi-UAV systems, incorporating adaptive network construction. Bayesian optimisation was used to improve learning efficiency and convergence by leveraging historical trajectory data. Gao et al.23 proposed a multi-agent federated reinforcement learning (MAFRL) framework for data collection from mobile ground devices (GDs), jointly optimising UAV trajectories and communication scheduling. Their model incorporates multi-step propagation and a duelling network architecture, resulting in improved convergence speed and stability. Song et al.24 introduced personalised federated deep reinforcement learning (PF-DRL) for multi-UAV trajectory optimisation in complex environments, proposing a PF-MADDPG-based method that minimises both total energy consumption and individual UAV flight costs. Yang et al.25 developed a collaborative trajectory optimisation approach for high-density user scenarios, integrating graph neural networks (GNNs) with proximal policy optimisation (PPO). Their reward design captures path efficiency, energy consumption, collision avoidance, and cooperative behaviour among UAVs. Despite these advances, many existing approaches rely on periodic global synchronisation or static aggregation, which can limit real-time responsiveness in highly dynamic environments. To improve policy stability and exploration efficiency, entropy-based DRL methods have gained increasing attention. Su et al.26 proposed a lightweight entropy-integrated DRL framework for fixed-wing UAV navigation, incorporating high-quality replay buffer samples into gradient updates to guide policy optimisation. This approach introduces a soft imitation mechanism, while an adaptive entropy regularisation term balances exploration and exploitation. More recent studies by Betalo et al.27–30 have explored dynamic coverage control using RIS, AI-driven task allocation in space–air–ground integrated networks (SAGIN), and data-driven optimisation strategies involving dynamic recharging and energy harvesting. These works reflect a growing focus on intelligent, energy-aware, and cooperative UAV systems. Lv et al.31 proposed an information-theoretic exploration strategy, Entropy Explorer (EE), which generates intrinsic rewards based on state and action entropy to address sparse reward challenges in UAV path planning. Combined with the twin delayed deep deterministic policy gradient (TD3) algorithm, this approach improves sampling efficiency and policy stability. Evaluation in AirSim demonstrated superior performance compared with benchmark methods, highlighting the effectiveness of entropy-guided optimisation for real-time UAV learning models such as SEF-PPO. Several recent studies have incorporated digital twin (DT) models to emulate real-world environments and accelerate UAV policy training. Zhao et al.32 combined DT with an incremental and distributed update (IDU) mechanism to optimise UAV trajectory and energy consumption, while a self-adaptive trajectory decision (STD) scheme generates candidate trajectories based on predicted conditions. Training is performed using duelling DQN with prioritised experience replay (PER). Shen et al.33 proposed DNQMIX, a multi-agent DRL framework for cooperative target search, supported by a DT-driven training architecture combining centralised training, decentralised execution, and continuous evolution (CTDECE). However, reliance on DT introduces scalability challenges due to modelling complexity, latency, and the need for high-fidelity environmental replication. Zhang et al.34 proposed a partially observable weighted mean field DDPG (PO-WMFDDPG) algorithm to address large-scale UAV swarm coordination, achieving improved success rates and convergence speed compared with baseline methods. Wang et al.35 explored conventional optimisation techniques, including convex optimisation and block coordinate descent (BCD), alongside a DRL-based real-time trajectory control algorithm (RAT) enhanced by prioritised experience replay. Huang et al.36 introduced a representation enhancement PPO (RE-PPO) framework, incorporating observation memory improvement (OMI) and dynamic relative position–attitude reshaping (DRPAR) to improve feature extraction and reduce collisions under partial observability. Experimental results in 3D environments demonstrated faster convergence, higher success rates, and fewer collisions. He et al.37 examined task offloading in multi-UAV systems, where ground devices select target UAVs for computation. Analytical methods were used to ensure fairness in UAV selection and task allocation, while a MADDPG framework optimised UAV trajectory. Wang et al.38 proposed a duelling double deep Q-network (D3QN)-based path planning method capable of adapting to varying IoT node distributions, data volumes, and UAV configurations. Overall, the literature indicates that DRL has become a foundational approach for UAV trajectory optimisation, user association, and energy management. However, several key research gaps remain. Many DRL and DT-based models depend heavily on offline training or simulated environments, limiting adaptability to real-time dynamics. Federated and multi-agent approaches continue to face challenges related to communication overhead and privacy, particularly under non-IID data distributions. Fully decentralised swarm coordination remains underexplored, with most models relying on centralised aggregation or fixed synchronisation intervals. Furthermore, critical tasks such as trajectory planning, handover management, energy optimisation, and bandwidth allocation are often addressed in isolation rather than within a unified framework. Motivated by these limitations, this study proposes a Selective Entropy-Fused Proximal Policy Optimisation (SEF-PPO) algorithm integrated with a federated reinforcement learning (FRL) framework to enable real-time, on-policy, and distributed UAV learning. Proposed system model The proposed system presents a federated reinforcement learning (FRL)-based communication and control framework for a multi-UAV aerial network operating in dynamic user environments. The architecture (Fig. 1) comprises multiple low-altitude UAVs that collaboratively provide real-time coverage to widely distributed mobile user equipment (UE) across a large geographic area. Each UAV functions as an autonomous agent employing the Selective Entropy-Fused Proximal Policy Optimisation (SEF-PPO) algorithm, enabling it to learn optimal flight trajectories, user association strategies, and resource allocation policies through continuous interaction with the environment. A high-altitude platform (HAP) serves as the federated coordination server, aggregating model parameters from distributed UAVs while preserving data privacy and locality. UAVs establish air-to-ground communication links with UEs and transmit encrypted model updates to the HAP via air-to-air backhaul channels. The system adopts a time-division multiple access (TDMA) protocol to ensure collision-free communication between UAVs and UEs. The overall objective is to maximise network coverage, fairness, and capacity, while minimising delay, interference, and energy consumption through an adaptive, on-policy FRL framework. System setup. System setup and user environment The simulation models a dynamic air-to-ground communication network within a 1000 m × 1000 m square area, which is subdivided into equally sized grid cells for spatial analysis. A fleet of 2–7 UAVs provides coverage for 100–255 mobile user equipment (UE), following heterogeneous mobility patterns, as illustrated in Fig. 1. User mobility is modelled using Gaussian, random waypoint, clustered, and long-path mobility models to capture diverse spatiotemporal dynamics. The UAVs operate at altitudes ranging from 50 to 300 m, with a maximum velocity of 30 m/s and defined acceleration constraints, ensuring a realistic operational scenario. The network operates under both line-of-sight (LoS) and non-line-of-sight (NLoS) channel conditions, with additive white Gaussian noise ranging from − 90 dBm to − 109 dBm and a carrier frequency of 2.5 GHz. The total system bandwidth is 10 MHz and is dynamically allocated among active UEs. Each simulation epoch consists of 60 time slots with durations of 0.5–1 s, and training is conducted over 200 epochs. The HAP coordinates UAV updates across federated learning rounds. A signal-to-interference-plus-noise ratio (SINR) threshold of ≥ 10 dB is maintained to ensure reliable communication links. UAVs minimise redundant coverage overlap while optimising their actions to maximise user coverage, energy efficiency, and throughput. During execution, UAVs dynamically adapt transmission power, bandwidth allocation, and trajectory to serve mobile UEs while avoiding collisions and reducing redundant coverage. The UAV–UE links are predominantly LoS, with probabilistic NLoS components governed by urban density. The received signal power at UE ufrom UAV iis modelled as where and denote antenna gains, is the wavelength, is the distance between UAV i and UE u, and represents the environment-dependent attenuation factor. These mobility and environmental settings form the testbed for the Selective Entropy-Fused FRL framework, evaluated under diverse UE distributions. To formalise the communication model, a three-dimensional (3D) Cartesian coordinate system is adopted. The position of UAV i is given by where denotes the flight altitude. Each ground UE u is located at and the HAP, acting as the federated aggregation server, is positioned at where , typically on the order of several kilometres. The distance between UAV i and UE u is given by The inter-UAV distance, used for collision avoidance and safe separation, is given by The UAV–UE communication link follows a probabilistic LoS/NLoS channel model based on 3GPP TR 36.777 for aerial communications. The LoS probability between UAV iand UE udepends on the elevation angle , defined as and modelled as where a and b are environment-dependent parameters. In this study, a=9.61 and b=0.16, corresponding to an urban environment as defined in 3GPP TR 36.777. All simulations are conducted under urban air-to-ground propagation conditions with dense obstacles and moderate NLoS probability. Although the proposed SEF-PPO model is not restricted to a specific environment, variations in a and b significantly influence LoS probability and SINR distribution, thereby affecting reward dynamics. The learning mechanism remains robust to such variations, as SEF-PPO adaptively updates its policy based on observed channel and reward conditions. While absolute performance metrics (e.g. throughput and coverage) may vary across suburban or rural settings, relative performance gains and convergence behaviour remain consistent. The path loss (in dB) between UAV i and UE u is expressed as where is the carrier frequency, c is the speed of light, and represents environment-specific attenuation factors (e.g., ). The received power at UE u from UAV i becomes The instantaneous SINR is given by where denotes the noise power spectral density and Bis the bandwidth allocated per UE. These formulations collectively describe the geometric relationships and propagation characteristics governing the UAV–UE–HAP communication environment. minimum inter-UAV separation of m is enforced to ensure safe aerial spacing and collision avoidance during trajectory optimisation. These formulations provide the analytical foundation for the optimisation framework described in Sect. 3.3. Problem formulation Based on the channel and mobility formulations in Sect. 3.1, the UAV–UE–HAP network optimisation is formulated as a constrained Markov decision process (MDP). The objective of the proposed federated UAV architecture is to jointly optimise user association, resource allocation, and trajectory design within a dynamic three-dimensional (3D) airspace. Each UAV operates as an autonomous decision-maker over a time horizon T, updating its behaviour via a local policy . The problem is therefore modelled as an MDP, where the policies aim to maximise the long-term expected cumulative reward: where denotes the multi-objective reward, incorporating coverage gain, energy efficiency, and user fairness, and is the discount factor. The local reward is defined as a weighted combination of performance and cost terms: where denotes the coverage ratio, defined as the fraction of active users successfully served by UAV i; represents the spectral efficiency, measured as throughput per unit bandwidth; is the fairness index, ensuring balanced resource allocation among users; corresponds to energy consumption, including both propulsion and transmission components; and denotes the number of handovers, penalising excessive mobility-induced switching. The coefficients are objective-specific weights that control trade-offs among these performance metrics. This aggregate reward formulation promotes the maximisation of service coverage and data rate, while penalising interference, energy consumption, and redundant handovers. The components , , and contribute positively, encouraging communication performance and fairness, whereas and contribute negatively, discouraging inefficient and unstable behaviour. The optimisation problem is subject to the following constraints. The achievable per-UE data rate served by UAV i is given by where is defined according to the channel model in Sect. 3.1. Note that represents the instantaneous data rate and is distinct from the reward function in (14). Minimum quality-of-service constraints are imposed as where is the SINR threshold and is the minimum required data rate per UE. The total system bandwidth is constrained by Transmission power per UAV is bounded as The UAV energy dynamics are modelled as The energy consumption term used in (14) is derived from the transmission and propulsion components in (19). The cumulative energy constraint is given by Where denotes propulsion power, typically dependent on velocity and is the energy capacity of UAV i. To ensure safe operation, inter-UAV distances must satisfy where is determined by operational constraints (typically 20–50 m in urban environments). A soft constraint is introduced to limit excessive handovers within a sliding time window , where is a design parameter. Load balancing and fairness are defined by letting denote the number of UEs served by UAV i. The relative load fraction is Fairness is enforced using Jain’s fairness index The fairness component in (14) is computed using the index in (24). The resulting multi-objective optimisation problem is solved via scalarisation of the expected reward. The SEF-PPO algorithm learns policies that approximate the constrained optimum by incorporating both hard constraints and soft penalties within the learning process. Local UAV training using SEF-PPO Each UAV operates as an individual PPO agent, learning to optimise its three-dimensional (3D) trajectory, bandwidth and power allocation, and handover decisions under mobility-induced link variations. A central parameter server aggregates model updates across agents. The state space is defined as where denote position, is the transmit power, is the allocated bandwidth, and is the number of associated users. The action space is given by To improve computational efficiency, the multi-objective reward in (14) is simplified by grouping related terms. Coverage, spectral efficiency, and fairness are incorporated into a unified rate-based reward, while energy consumption and handover penalties remain explicit. The resulting reward function used in SEF-PPO training is: where is computed using (15). The global model parameters are updated via federated aggregation as Standard proximal policy optimisation (PPO) is inherently on-policy, often sample-inefficient, and may exhibit unstable convergence in real-time multi-agent coordination. To address these limitations, PPO is enhanced with a selective entropy fusion (SEF) mechanism, resulting in the SEF-PPO algorithm. The SEF module augments PPO with entropy-aware advantage estimation and dynamic entropy modulation, enabling a more effective balance between exploration and exploitation without relying on replay buffers. The training objective is defined as: In conventional PPO, the entropy term acts as an additive regulariser with a fixed or heuristically scheduled coefficient and is not directly linked to reward dynamics. Consequently, exploration remains largely independent of performance feedback. The standard PPO objective is: where is a constant entropy coefficient independent of reward dynamics. denotes the entropy term, which encourages exploration, and is a time-varying entropy coefficient dynamically modulated by the SEF mechanism, defined as: where represents the temporal difference in reward between consecutive time steps. In contrast to traditional entropy scheduling schemes, the proposed formulation directly links entropy modulation to changes in reward. Specifically, large variations in reward increase , thereby encouraging exploration in uncertain or rapidly changing environments, whereas stable reward signals reduce entropy, promoting exploitation. This mechanism establishes a feedback loop in which exploration is driven by performance variability rather than predetermined schedules. Unlike conventional adaptive entropy approaches, in which or depends solely on policy entropy, the proposed formulation explicitly correlates reward variation with the intensity of exploration. The proposed SEF mechanism differs fundamentally from existing entropy-based approaches by introducing a bi-directional coupling between reward dynamics and entropy control. First, entropy is incorporated directly into the reward function (Eq. 33), thereby modifying the effective optimisation objective rather than acting solely as a regulariser. Second, entropy coefficients () are dynamically adjusted based on reward variance, establishing a direct link between exploration and environmental feedback. Third, the sensitivity term in Eq. (37) explicitly relates the gradients of entropy and reward, enabling entropy to influence policy updates in proportion to reward sensitivity. In contrast, standard PPO employs fixed or externally scheduled entropy terms, while soft actor–critic (SAC) regulates entropy using a target entropy constraint that is independent of reward dynamics. Accordingly, SEF introduces a reward–entropy co-adaptation mechanism that is not present in conventional PPO or SAC frameworks. As High reward variance, which reflects environmental uncertainty or sparse rewards, leads SEF to increase entropy and promote exploration. As the reward signal stabilises, entropy is gradually reduced, shifting the focus towards exploitation and convergence. This adaptive entropy modulation enables stable exploration in non-IID and heterogeneous swarm environments, while incorporating lightweight local memory caching to smooth policy updates and mini-batch clipping to prevent excessively large gradient steps. It also supports sample-efficient learning without the need for experience replay. As a result, each UAV agent independently adapts its policy to changing airspace conditions and user mobility patterns, enabling robust, energy-efficient operation and seamless handover management. The SEF mechanism constitutes a lightweight extension to PPO, introducing minimal computational overhead while enabling adaptive entropy regulation based on direct reward feedback. In contrast to fixed-entropy PPO variants, SEF continuously monitors environmental stability through reward variance and automatically adjusts the level of exploration. This mitigates premature convergence and supports stable policy updates in federated, non-IID UAV environments. SEF-PPO is conceptually inspired by entropy–reward coupling mechanisms explored in recent reinforcement learning literature. For example, Dong et al.39 introduced the Adaptive Joint Entropy Reward (AJER), which provides effective guidance for agents and enhances decision-making in sparse-reward environments. Wu et al.40 analysed policy optimisation stability and convergence under entropy-driven dynamics, while Choe and Kim41 developed maximum-entropy actor–critic methods incorporating entropy–advantage estimation. Building on these insights, SEF-PPO introduces a dynamic entropy-weighted reward formulation that supports sample-efficient, on-policy learning and robust convergence in multi-UAV federated settings, without requiring replay buffers or offline pretraining. Figure 2 illustrates the SEF-PPO architecture implemented within each UAV agent, including the policy network, value network, and SEF controller responsible for dynamic entropy modulation. The overall training framework of the proposed method is shown in Fig. 3. SEF-PPO architecture. Training structure of the proposed SEF-PPO based FRL system. SEF introduces a dynamic entropy-weighted reward defined as where . Here, corresponds to the reward defined in Eq. (27), while represents the entropy-augmented reward used in SEF-PPO optimisation. The entropy coefficient is dynamically modulated as where denotes the reward variance. This formulation increases entropy when reward variance is high, thereby encouraging exploration, and reduces it when reward variance stabilises, promoting exploitation. The parameter governs entropy modulation based on reward variance, while is the effective entropy coefficient used in the loss function, and represents the sensitivity term. The gradient update is given by: To integrate entropy with reward sensitivity, define: The selective entropy fusion loss is then given by where is a dynamically modulated entropy coefficient designed to maintain low computational overhead. The derivative of the loss with respect to the entropy coefficient is given by The entropy coefficient is then updated dynamically using gradient feedback The entropy modulation dynamics in SEF are expressed as The system constraints, in addition to the power constraints defined in Sect. 3.2, are as follows. Load balancing (traffic fairness) is defined as The rate constraint is given by The per-UAV energy consumption is which is consistent with the energy model defined in Eq. (19). The corresponding energy efficiency is These constraints are incorporated into PPO via a penalty-adjusted reward formulation. The average reward per episode is defined as The average number of steps per positive reward is given by where denotes the indicator function. SEF-PPO achieves stable convergence through lightweight entropy–reward coupling, without requiring replay buffers or off-policy correction. The training configuration uses a learning rate of , a batch size of 128, a clipping parameter , and four PPO update epochs per iteration. SEF-PPO agent training. Global entropy weighted aggregation In the proposed Selective Entropy-Fused Proximal Policy Optimisation (SEF-PPO) framework, all UAVs participate in a federated reinforcement learning (FRL) paradigm to enable scalable, privacy-preserving, and decentralised training. Rather than sharing raw trajectory or user data, each UAV agent locally optimises its SEF-PPO policy using its own experience—such as signal quality, energy state, and user density—and periodically transmits only encrypted model parameters to the high-altitude platform (HAP), which acts as the federated aggregation server. The HAP performs asynchronous global aggregation using a customised federated averaging (FedAvg) scheme: This weighting scheme does not restrict local exploration, as entropy modulation is preserved at the UAV level through the SEF mechanism (Eqs. 32–34). Instead, it mitigates the impact of high-variance local updates on the global model, thereby improving convergence stability under heterogeneous (non-IID) conditions. Policy uncertainty and gradient variance are reflected in the entropy coefficient . Higher-entropy policies produce more stochastic action distributions, which can lead to high-variance gradient updates. Accordingly, Eq. (48) can be interpreted as an inverse-uncertainty weighting scheme, , which reduces the influence of high-variance updates during aggregation while preserving their contribution. The aggregation rule in Eq. (47) extends standard FedAvg by incorporating entropy-aware weighting (Eq. (48)), thereby assigning lower weights to UAVs exhibiting higher exploration and promoting stable global convergence. This weighting strategy does not restrict agent-level exploration. Individual UAVs continue to explore using locally adapted entropy coefficients (Eqs. 32–34). Instead, aggregation acts as a moderating mechanism, limiting the global impact of highly exploratory agents whose updates may be noisy or unstable. This design is inspired by variance-reduction principles in distributed optimisation, where stabilising global updates is critical under heterogeneous (non-IID) data distributions. In this context, client contributions are inversely proportional to update uncertainty, improving convergence stability while preserving local exploration. The proposed federated architecture offers several advantages, including privacy preservation (as no raw environmental or user data are shared), scalability (through independent and asynchronous UAV training), and enhanced swarm coordination (via shared model parameters that promote global situational awareness, reduce redundant trajectories, and improve coverage efficiency). SEF-PPO adopts a locally on-policy training paradigm rather than a globally synchronised on-policy approach. Each UAV collects trajectories based on its current policy and performs PPO updates locally before federated aggregation. The aggregation step is applied to updated model parameters rather than raw experience samples, thereby avoiding the mixing of off-policy data. Although parameter aggregation introduces slight policy heterogeneity across agents, this effect is mitigated through periodic synchronisation and entropy-driven updates, enabling convergence while retaining the practical benefits of on-policy learning. Each UAV communicates model parameters or gradients of dimensionality representing the total number of parameters transmitted per federated round. The SEF-PPO model employs a lightweight multi-layer perceptron (MLP) with approximately trainable parameters, making it suitable for deployment on resource-constrained UAV platforms. The communication latency is given by highlighting the scalability implications of the federated framework. Global model stability depends on both local learning variance and communication delay. The convergence bound for SEF-PPO-FRL is expressed as where is the strong convexity constant, denotes local gradient variance, and captures entropy-induced fluctuations. This iterative process—local learning, global aggregation, and personalised policy updates—enables collective intelligence sharing while preserving data confidentiality. To further align exploration across heterogeneous UAVs, SEF employs entropy-weighted aggregation, assigning lower weights to agents with higher uncertainty or exploration levels. This stabilises convergence under non-IID conditions while maintaining effective exploration. The overall federated objective is to maximise the collective efficiency of the UAV swarm in terms of global reward. where the global objective aggregates the unified reward terms defined in Eq. (14) and reformulated in Eq. (27), ensuring consistency between local and federated optimisation. The global optimisation problem is defined as To account for variations in UAV load and link capacity, the aggregation weights can be normalised as Across successive federated rounds, the UAV swarm adapts cooperatively to user mobility, interference dynamics, and energy constraints. The HAP thus functions as an integrated coordination layer, consolidating UAV-specific learning outcomes into a unified global policy. Personalized federated adaptation Following each global aggregation round, UAVs perform personalised local adaptation to refine the shared policy in accordance with their specific regional conditions, mobility patterns, and load variations. While the aggregated global policy provides generalised knowledge, the non-IID nature of UAV trajectories and UE distributions necessitates fine-grained personalisation. The personalisation function is defined as: where performs gradient-based local policy fine-tuning using the global policy and the UAV-specific trajectory–reward dataset . The adaptation process is formulated as a meta-learning-based optimisation: where is the local adaptation rate, and denotes the SEF-PPO loss evaluated on the dataset of UAV i. To prevent overfitting under non-IID conditions, a regularised personalisation objective is employed. where is the personalisation regularisation coefficient. This term ensures that UAV-specific policies remain aligned with the global policy while allowing local adaptation. The adaptation phase can be extended to multi-step local fine-tuning, enabling UAVs to iteratively refine their policies based on updated trajectory rewards where is the local adaptation step size. The update is based on the same SEF-PPO loss defined in Eq. (29), ensuring consistency between local training and personalised adaptation. This personalised update mechanism supports adaptive load balancing, region-specific quality-of-service (QoS) optimisation, and seamless mobility adaptation, while maintaining alignment with global model stability. Mobility-aware handover management As UAVs continuously serve mobile user equipment (UE), maintaining uninterrupted connectivity is essential. Each UAV monitors the signal-to-interference-plus-noise ratio (SINR) and traffic load to determine when a handover should be initiated. A handover is triggered when: where is the SINR between UAV iand user u, is the SINR threshold, is the distance between the UAV and the user, and denotes the coverage radius of UAV i. The target UAV for handover is selected based on proximity, available bandwidth, and load balancing considerations, ensuring both connectivity continuity and energy efficiency. To prevent unnecessary handovers and support smooth mobility, a penalty term is incorporated into the reward function: where is the penalty coefficient and represents the number of recent handovers. This penalty is included within the handover component of the overall reward function defined in Eq. (27). Although the primary handover trigger based on link quality and coverage constraints is defined in Eq. (59), the actual handover decision is implicitly governed by the SEF-PPO policy through the reward function. In particular, load balancing, bandwidth availability, and handover stability (including ping-pong prevention) are incorporated via the load term in Eq. (41) and the handover penalty in Eq. (60). Consequently, UAV agents are trained to avoid overloaded nodes, maintain stable links, and minimise unnecessary switching. SINR-based communication constraints, as defined in Eqs. (12) and (50), act as indirect indicators of link and backhaul quality, thereby influencing both policy updates and reward dynamics. As a result, handover decisions are not governed solely by fixed thresholds but are instead optimised within the reinforcement learning framework. This approach reduces excessive switching and stabilises trajectory planning. By integrating these factors into the SEF-PPO reward mechanism, UAVs learn to maximise communication continuity and energy efficiency, while minimising link failures and congestion. Furthermore, the system supports dynamic prediction of user mobility patterns, enabling proactive handover decisions that promote fairness and balanced load distribution across UAVs. Reward function design The SEF-PPO model formulates each UAV as a Markov decision process (MDP), defined as . Here, Sdenotes the state space, including UAV position, residual energy, user density, and the SINR map; Arepresents the action space, comprising power control, trajectory adjustment, and bandwidth allocation; is the state transition probability; Ris the multi-objective reward function, as defined earlier; and is the discount factor. Each UAV agent aims to learn an optimal policy that maximises the expected cumulative discounted reward . The PPO optimiser employs a clipped surrogate loss function where is the probability ratio, is the advantage estimate, and is the entropy coefficient, adaptively modulated by the SEF mechanism. The baseline PPO objective is given in Eq. (61), while the SEF-PPO loss function used in this work extends it through entropy-adaptive modulation, as described in Eq. (29). Reward design plays a central role in balancing communication efficiency, energy consumption, and user experience. In this framework, each UAV optimises a multi-objective reward that jointly accounts for coverage, spectral efficiency, fairness, energy consumption, and handover stability, as defined in Eq. (14). The integration of SEF-PPO introduces adaptive entropy modulation during training, promoting exploration when reward signals are sparse or unstable, and favouring exploitation as convergence is achieved. This adaptive reward–entropy interaction enables stable and efficient policy learning under non-IID conditions and dynamically changing aerial environments. Swarm-level coordination and complexity analysis The proposed SEF-PPO-guided federated reinforcement learning (FRL) architecture enables emergent coordination at the swarm level through decentralised policy learning and collective parameter aggregation. Each UAV updates its SEF-PPO policy locally based on environmental feedback, while the high-altitude platform (HAP) performs periodic FedAvg aggregation to integrate collective knowledge across the swarm. This iterative process naturally leads to coordinated behaviour, whereby UAVs learn to self-organise their three-dimensional (3D) trajectories, transmission power, and bandwidth allocation to achieve joint resource optimisation, including trajectory control, load balancing, and interference mitigation. Through repeated training–aggregation–adaptation cycles, the swarm evolves into an adaptive, self-regulating communication network capable of responding dynamically to user mobility, energy constraints, and channel variations. The total computational cost per communication round is , where Nis the number of UAVs, Mdenotes the number of SEF-PPO updates per UAV, and Erepresents the number of training epochs. Despite the distributed nature of the framework, scalability and communication efficiency are maintained through the use of lightweight model architectures, sparse parameter updates, and model compression techniques. These features enable the FRL system to operate effectively in bandwidth-constrained environments and support large-scale UAV deployments. Results and discussion Simulation experiments were conducted to evaluate the effectiveness of the proposed SEF-PPO framework under realistic aerial communication scenarios. The environment was modelled using a high-fidelity Python-based simulator incorporating UAV mobility, radio propagation, and computational dynamics in UAV-assisted networks. All simulations were performed on a high-performance computing platform (Intel Core i9 processor with 64 GB RAM), using TensorFlow 2.12 and Python 3.10. Key performance metrics, including convergence rate, average throughput, coverage ratio, energy efficiency, and latency, were evaluated across different configurations. The proposed method was compared against baseline approaches, including DMTD20, DRL-EC318, and conventional random and greedy schemes, all implemented within the same federated learning framework for consistency. Table 1 summarises the principal system parameters described earlier in Sect. 3.1. Table 1Simulation and system parameters.ParameterSymbolDescriptionRangeSimulation areaL × L timesOperational field size1000 m × 1000 mUAV count Number of aerial nodes2–7UE count Number of users100–255Environment type—Channel model settingUrbanCarrier frequency Transmission carrier2.5 GHzBandwidth B Total available bandwidth10 MHzNoise power AWGN level−90 to − 109 dBmUAV altitude Flight height per UAV50–300 mUAV velocity Maximum flight speed20–30 m/sTransmission power UAV transmit power0–20 dBmSINR threshold Minimum link quality≥ 10 dBTime-slot duration Duration per step0.5–1 sTraining epochs Number of learning rounds200TDMA protocol—Access methodSequential UE scheduling Simulation and system parameters. Training dynamics and convergence analysis The convergence performance of the proposed SEF-PPO-based federated UAV control framework is evaluated in terms of training reward progression, policy stability, Q-value variance, entropy adaptation, and federated learning efficiency. Each training step corresponds to a single local SEF-PPO update, consisting of approximately 60 environment time steps (i.e. one episode). During each iteration, UAV agents generate trajectories and update their policies using mini-batch optimisation. Federated aggregation is performed after every eight local update steps, such that the overall training horizon of 500 iterations comprises approximately 60 global aggregation rounds. Consequently, Fig. 4 illustrates cumulative local learning progress, with multiple local policy updates occurring between successive global aggregations. Average reward vs. training iterations. The reward curves in Fig. 4 illustrate the average cumulative reward achieved by all UAV agents over the training iterations. During the initial phase, the proposed SEF-PPO approach exhibits a steep increase in average reward, whereas DMTD20 and DRL-EC318 demonstrate slower convergence. SEF-PPO attains higher reward levels in fewer iterations, indicating improved sample efficiency and enhanced policy stability compared with the baseline methods. The entropy adaptation mechanism ensures a balanced exploration–exploitation trade-off, supporting effective trajectory and resource optimisation. Figure 5 presents the standard deviation of rewards across training iterations as a measure of policy stability. All results are averaged over five independent simulation runs, with variability (not shown) represented by ± 1 standard deviation. SEF-PPO achieves the lowest standard deviation, indicating smoother policy updates and more stable convergence across UAV agents. The integrated SEF mechanism effectively mitigates gradient oscillations and prevents unstable policy shifts during on-policy learning. Convergence policy stability. Figure6 illustrates the variance of Q-values across training iterations, serving as an indicator of the stability of action–value estimation. The lower Q-value variance observed for SEF-PPO indicates greater confidence in value estimation and more stable policy convergence. SEF-PPO converges within approximately 180–220 iterations, whereas DRL-EC3 and DMTD require more than 350 iterations to achieve comparable reward levels. This corresponds to an improvement in convergence speed of approximately 35–45%. The SEF mechanism effectively regularises short-term update variability, enabling stable learning performance under non-stationary and dynamic environments. Convergence variance. Figure 7 depicts the evolution of the mean entropy coefficient of SEF-PPO over 60 federated rounds. The entropy coefficient is initially high, encouraging exploratory behaviour. As training progresses, it decreases substantially and subsequently stabilises, reflecting the adaptive nature of the SEF mechanism. This behaviour enables a gradual transition from exploration to exploitation, where early exploration supports coverage expansion, while later exploitation ensures precise trajectory optimisation and efficient power allocation. Figure 8 presents the average per-step reward across federated rounds. The limited fluctuation in reward indicates stable global model aggregation via the HAP. Furthermore, the personalised adaptation layer enables UAV-specific convergence despite non-IID data distributions, demonstrating the robustness of the proposed federated learning framework. SEF entropy coefficient over federated rounds Federated training progress. UAV trajectory optimisation and mobility patterns UAV trajectory optimisation is central to ensuring uninterrupted connectivity, optimal coverage, and reduced interference in dynamic aerial networks. The simulation environment is initialised within a 1000 m × 1000 m area, with UAV operating altitudes extending up to 300 m. Figure 9 illustrates the initial three-dimensional (3D) spatial distribution of UAVs and UEs under different mobility models, including Gaussian, random waypoint, and long-path patterns. These models generate heterogeneous user distributions, characterised by densely clustered regions (Gaussian), uniformly distributed mobility (random waypoint), and structured movement trajectories (long-path). As a result, each UAV operates under distinct local observation conditions, influenced by variations in user density, mobility dynamics, and link quality. This setup reflects a realistic federated learning scenario with non-identically distributed (non-IID) local experiences across UAV agents. Initial 3D setup with UAV and UE. Figure 10 presents the UAV trajectory adaptations obtained under various baseline methods and the proposed SEF-PPO approach. Conventional methods, such as random and greedy strategies, fail to maintain consistent coverage under heterogeneous conditions, resulting in unstable and inefficient movement patterns. DRL-based baselines exhibit improved adaptability; however, they often display oscillatory behaviour or converge to locally suboptimal trajectories due to variability in user distributions. 3D trajectories subplot. The figure compares (a) DMTD20, (b) DRL-EC318, (c) random, (d) greedy, and (e) the proposed SEF-PPO method. Each subplot illustrates UAV flight trajectories in X–Y–Z coordinates, where red markers indicate UAV positions and black markers represent UEs. (a) DMTD demonstrates relatively smooth but limited trajectory adaptation, partially aligning with user clusters and indicating moderate dynamic tracking capability. However, trajectory curvature reflects convergence to local optima, resulting in incomplete area coverage. (b) DRL-EC3 exhibits significant oscillations in both altitude and spatial positioning, indicating aggressive exploration but reduced trajectory smoothness. These oscillations lead to instability in service continuity. (c) Random displays highly unstable and inefficient flight paths, characterised by abrupt directional changes and redundant revisits, leading to increased energy consumption and poor UE association consistency. (d) Greedy prioritises immediate proximity to users, resulting in zigzag movement around dense clusters while neglecting sparsely distributed UEs. This behaviour limits overall coverage balance and reduces system efficiency. The visualised trajectories demonstrate that heuristic and conventional approaches (DMTD, random, greedy, and DRL-EC3) lack global optimisation and smooth trajectory characteristics, as shown in Fig. 10(a–d), resulting in imbalanced energy consumption and inefficient flight behaviour. In contrast, the proposed SEF-PPO method in Fig. 10(e) generates smoother, more adaptive, and better-balanced trajectories by integrating entropy-aware exploration with federated policy synchronisation. This enables UAVs to dynamically adjust their flight patterns in response to user mobility and network conditions. The SEF-PPO model consistently produces stable and coherent trajectories across heterogeneous mobility scenarios. Despite the non-IID nature of local observations, UAVs trained under the federated framework exhibit coordinated movement patterns. This indicates that the proposed approach effectively integrates diverse local experiences, ensuring consistent policy behaviour across a wide range of spatial and temporal user distributions, and demonstrating strong robustness to non-IID environments. Performance comparison under different configurations Figure 11 compares task delay and total energy consumption across the evaluated methods. The proposed SEF-PPO approach achieves the lowest average delay and energy consumption, outperforming all baseline methods. This improvement reflects the effectiveness of jointly optimising the energy–latency trade-off through entropy-controlled exploration and federated learning-based personalisation. The adaptive learning mechanism enables efficient resource allocation and trajectory planning, resulting in reduced communication delays and lower energy expenditure. Task delay with energy consumption. As shown in Fig. 12, the proposed method achieves a user coverage rate of 90% and an area utilisation of 88%, outperforming all baseline approaches. Through cooperative trajectory planning and personalised policy updates, UAVs attain enhanced spatial coverage and improved load balancing. The SEF-PPO framework continuously adapts to user mobility dynamics, minimising service gaps and reducing redundant coverage. As a result, it effectively optimises both coverage density and resource utilisation. Coverage performance. Energy efficiency, measured in bits per joule, is presented in Fig. 13. The proposed SEF-PPO-FRL approach achieves the highest efficiency across all configurations, with an average improvement of approximately 41% over DRL-EC3 and 31% over DMTD, depending on the operating conditions. These results are averaged over five independent simulation runs, with low variability (± 2–3%), indicating strong statistical reliability. For clarity of presentation, confidence intervals are not shown in Fig. 13. The federated coordination mechanism enables UAVs to share policy updates, reducing redundant energy consumption while maintaining high throughput. In addition, the SEF mechanism guides exploration towards energy-efficient actions, resulting in sustained, high-efficiency communication performance in dynamic and non-stationary environments. Energy efficiency. Energy consumption and operational efficiency This section evaluates the energy and operational performance of the proposed framework using metrics such as the ratio of flying to hovering time, energy component breakdown, and power–bandwidth utilisation. The results demonstrate how the integration of SEF, PPO-driven trajectory optimisation, and personalised federated adaptation contributes to enhanced energy efficiency and improved operational balance across all UAV-assisted configurations. Figure 14 illustrates the average flying and hovering times of UAVs across five configurations: the proposed SEF-PPO-FRL, DMTD, DRL-EC3, random, and greedy methods. The proposed approach achieves the lowest flying and hovering times, indicating efficient trajectory planning and minimal idle operation. Flying vs. hovering time. Figure 15 presents the trade-off between average power consumption (W) and bandwidth utilisation (%) for the evaluated methods. The proposed model achieves the lowest average power consumption while maintaining the highest bandwidth efficiency, demonstrating its ability to deliver high data rates at reduced power levels. These results highlight the superior resource scheduling capability of the proposed framework, where PPO-based optimisation combined with federated coordination enables effective energy management and improved spectral efficiency. Power vs. bandwidth usage. Fairness, handover, and scalability analysis This section examines the fairness, handover stability, and scalability of the proposed system under dynamic UAV–UE network conditions. The evaluated metrics include Jain’s fairness index, access delay, handover frequency and success rate, task delay scalability, and task offloading performance across different mobility models. Figure 16 compares Jain’s fairness index and access delay for five methods: the proposed SEF-PPO-FRL, DMTD20, DRL-EC318, random, and greedy approaches. The proposed method achieves the highest fairness index, indicating balanced task distribution among UAVs, while also maintaining the lowest access delay. Figure 17 illustrates the trade-off between handover frequency and success rate (%). The proposed model demonstrates smooth and efficient handover behaviour, achieving a favourable balance between switching frequency and reliability. In contrast, baseline methods exhibit higher handover frequencies, often associated with redundant or unsuccessful handovers, leading to reduced stability and increased signalling overhead. Fairness access delay. Handover frequency and success rate. Figure 18 compares the average task delay (ms) as the number of UAVs increases. In the proposed model, task delay decreases with increasing UAV density, indicating strong scalability and efficient task distribution. This trend suggests that swarm-based UAV scheduling combined with distributed policy learning enables near-linear reductions in delay as the number of UAVs grows. Average task delay with scalability. Figure 19 illustrates the variation of the fairness index with increasing numbers of UAVs. The consistent fairness observed under scaling conditions can be attributed to the personalised federated learning mechanism, which coordinates global model updates while adapting to heterogeneous UAV–UE interactions. Scalability with fairness index. Figure 20 presents the impact of scalability on network throughput (Mbps). The results indicate that SEF-based cooperative energy management not only enhances energy efficiency but also improves data rate performance across UAV clusters. Scalability with throughput. The policy network is implemented as a lightweight actor–critic architecture with approximately parameters in the SEF-PPO-FRL framework. Assuming a 32-bit floating-point representation (4 bytes per parameter), the model size per UAV is approximately 0.8 MB. Based on the communication cost formulation in Eq. (49), the communication overhead per federated round can be estimated. For the largest configuration considered ( N=7 UAVs): Total transmitted parameters . Each parameter (FP32) . Communication overhead . For larger swarm sizes: These results indicate that communication overhead scales linearly with the number of UAVs. However, this overhead is mitigated in the proposed framework through several design choices. Selective Entropy Fusion (SEF) reduces the impact of high-variance updates, thereby avoiding unnecessary transmissions. Periodic (non-continuous) aggregation limits the frequency of communication rounds. HAP-based centralised aggregation prevents excessive peer-to-peer communication overhead. In addition, the compact on-policy PPO architecture eliminates the need for replay buffers, further reducing model size. While the proposed framework performs efficiently for moderate swarm sizes (up to approximately 50 UAVs), ultra-large-scale deployments may require additional communication-efficient strategies, such as model compression, parameter sparsification, or event-driven update mechanisms. Ablation analysis A quantitative assessment of the contribution of each component is conducted through an ablation study, in which individual modules are selectively disabled while all other settings are held constant. Three configurations are considered: (i) SEF-PPO-FRL (full model), (ii) No-SEF (removal of the entropy fusion mechanism), and (iii) No-Personalisation (removal of local adaptation following aggregation). Figure 21 illustrates the contribution of each component to overall system performance. The integrated design proves most effective, with the full SEF-PPO-FRL model achieving the highest average reward (approximately 95) and the lowest delay (approximately 0.45 s). Ablation study analysing SEF mechanism and personalisation. When the SEF mechanism is removed (No-SEF), the average reward decreases significantly (approximately 82), while the delay increases to around 0.60 s. This demonstrates that SEF plays a critical role in enhancing exploration efficiency and stabilising policy learning in dynamic environments. Similarly, removing the personalisation module (No-Personalisation) leads to a notable reduction in performance, with the average reward dropping to approximately 85 and the delay increasing to around 0.52 s. This highlights the importance of adapting the global model to local UAV conditions, particularly under non-IID user distributions. Overall, the ablation results support the hypothesis that SEF improves learning stability and exploration efficiency, while personalisation enhances adaptability to heterogeneous environments. The combination of all components yields the best overall performance. Conclusion This study proposes a novel federated reinforcement learning (FRL)-based UAV networking framework enhanced by a Selective Entropy-Fused Proximal Policy Optimisation (SEF-PPO) algorithm for real-time, adaptive, and energy-efficient wireless communication. The proposed system addresses the limitations of conventional digital twin and offline reinforcement learning approaches by enabling on-policy learning without reliance on replay buffers or pretraining. Within a federated framework, UAVs collaboratively optimise their policies through aggregation at a high-altitude platform (HAP), ensuring scalability while preserving user data privacy. The model jointly addresses key network challenges, including trajectory optimisation, user association, energy-efficient resource allocation, handover management, and swarm coordination under heterogeneous conditions. Experimental results demonstrate significant improvements in throughput, fairness, latency, and energy efficiency compared with baseline methods such as DMTD, DRL-EC3, and conventional greedy and random strategies. The SEF-PPO mechanism facilitates stable convergence and efficient resource utilisation, enabling UAVs to adapt autonomously to non-stationary user demands and dynamic interference conditions. Despite these advantages, the framework presents certain limitations, including communication overhead associated with federated aggregation and the use of simplified air-to-ground channel models in simulation. Future work will focus on systematically evaluating the impact of non-IID data heterogeneity by varying user mobility distributions and analysing its effects on convergence stability, policy divergence, and overall network performance. Additional research directions include the integration of communication-efficient techniques, such as model compression and parameter sparsification, to reduce aggregation overhead, and the extension of the framework to multi-tier aerial–terrestrial hybrid networks. Further development will also explore the incorporation of edge intelligence, adaptive model pruning, and real-world UAV deployment experiments to validate scalability and practical applicability in large-scale 6G communication systems. Overall, this work establishes a robust, decentralised, and learning-driven paradigm for next-generation UAV-enabled wireless networks. Author contributions Conceptualization, methodology, formal analysis, investigation, software and validation by M. Velmurugan. Resources and data curation, writing original draft preparation, writing review and editing, by T. Velmurugan. All authors have read and agreed to the published version of the manuscript. Funding Open access funding provided by Vellore Institute of Technology. Data availability The data that support the findings of this study are available from the corresponding author upon reasonable request. Declarations Competing interests The authors declare no competing interests. Footnotes Publisher’s note Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations. References Associated Data Data Availability Statement The data that support the findings of this study are available from the corresponding author upon reasonable request.