Perception-Aware Cooperative Path Planning for Multi-UAV Systems in Urban Wind Fields via Deep Reinforcement Learning Authors: Jie Ding, Linshen Wang, Shuxin Jin, Di Wang Published: 2026 ABSTRACT The safe deployment of multiple Unmanned Aerial Vehicles (UAVs) in complex urban environments relies heavily on accurate environmental perception and efficient cooperative path planning. However, executing multi-UAV operations in low-altitude airspaces faces severe challenges due to the dual constraints of complex building clusters and steady-state wind field disturbances. These dynamic environmental factors frequently distort sensory expectations, inducing trajectory drift and degrading policy robustness. To address these limitations, this paper proposes an enhanced Dueling Double Deep Q-Network (D3QN) algorithm, termed NPD3QN, tailored for perception-aware multi-UAV cooperative path planning. By formulating the perceived environmental data (e.g., wind speed, obstacle distances, and inter-UAV states) into a Markov Decision Process, an N-step update strategy is integrated to enhance the characterization of long-term returns. Simultaneously, an improved Prioritized Experience Replay (PER) mechanism is developed to actively filter negative experiences and assign dynamic weights to critical state-action samples, thereby significantly elevating training stability. A 3D urban kinematic environment incorporating a steady-state simulated wind field is constructed. Extensive ablation and comparative results demonstrate that NPD3QN effectively maps high-dimensional state perceptions to robust control commands. In wind-disturbed scenarios, it generates highly streamlined cooperative trajectories, reducing the total path length by approximately 11.7% compared to the standard D3QN baseline. While currently evaluated within steady-state simulated constraints, this study establishes a robust, sensor-driven methodological foundation for autonomous multi-UAV cooperative path planning in wind-disturbed airspaces. 1. Introduction In recent years, the rapid advancement of intelligent autonomous systems in sensing, computation, and decision-making has catalyzed the integration of Unmanned Aerial Vehicles (UAVs) into various civilian applications [1]. Driven by the burgeoning requirements of Urban Air Mobility (UAM), UAVs are increasingly deployed for low-altitude logistics delivery, emergency medical rescue, real-time traffic monitoring, and critical infrastructure inspection [2]. Within these contexts, the urban low-altitude airspace has emerged as a vital spatial medium for complex mission execution. A fundamental challenge in such operations is enabling UAVs to achieve autonomous obstacle avoidance and precise target reaching within unfamiliar and constrained environments. This capability hinges on autonomous decision-making to navigate around obstacles and identify optimal trajectories—a challenge formulated essentially as a complex path planning problem [3]. In this context, cooperative navigation serves as the overarching sensor-driven framework, while path planning acts as its core decision-making and execution mechanism. Effective path planning is not only critical for ensuring the operational safety of UAV fleets but also serves as a primary determinant of mission efficiency. Consequently, developing efficient and reliable path planning methodologies for complex urban low-altitude environments is of paramount scientific and practical significance. To achieve this, modern UAVs heavily rely on diverse onboard sensors (e.g., LiDAR, IMU, and anemometers) to acquire real-time environmental states. The efficient processing and fusion of these multi-modal sensory inputs into safe navigational commands remain a core challenge in autonomous system design. Compared to ground-based path planning, low-altitude path planning offers enhanced 3D spatiality and maneuverability; however, urban environments introduce significantly more rigorous constraints [4]. First, the high density and structural complexity of building clusters severely restrict the traversable airspace, necessitating obstacle avoidance within extremely limited safety margins. Second, local wind fields are highly susceptible to urban topography and building layouts, where significant variations in wind speed and direction can induce path deviations and increase flight distance, even escalating to collision risks. Therefore, UAV path planning in urban airspace must explicitly account for wind field disturbances to ensure the flyability and safety of the generated trajectories [5]. Furthermore, urban missions typically involve multi-UAV systems operating in shared airspace. Beyond individual obstacle avoidance, these systems require sophisticated spatiotemporal coordination to resolve conflicts while maintaining global operational efficiency—for instance, by minimizing redundant detours and reducing path crossing risks. When subjected to steady-state wind disturbances, the resulting path drift further complicates the relative motion dynamics between drones [6], making it increasingly difficult to avoid collisions and coordinate maneuvers. Thus, multi-UAV collaborative path planning that incorporates both structural building constraints and steady-state wind-field disturbances represents a high-value research frontier. Current methodologies for solving path planning problems are generally categorized into three paradigms: traditional algorithms, intelligent metaheuristic algorithms, and deep reinforcement learning (DRL) [7]. Traditional algorithms typically assume static and interference-free environments [8]; while performant in simple or well-mapped scenarios, they exhibit significant limitations in complex urban settings. Specifically, when facing external perturbations such as wind-field disturbances, traditional methods often lack the requisite adaptability and real-time responsiveness for practical deployment. Intelligent metaheuristic (bionic) algorithms, characterized by their simulation of natural swarming behaviors and local search mechanisms, offer a viable pathway for trajectory optimization under multi-objective and multi-constraint conditions [9]. However, these approaches rely heavily on iterative stochastic optimization, which often leads to prohibitive computational overhead, slow convergence in high-dimensional state spaces, and a tendency to become trapped in local optima. Such bottlenecks fundamentally limit the efficiency and precision of path planning in time-sensitive urban scenarios. The integration of Deep Reinforcement Learning (DRL) into sensor-based path planning represents a paradigm shift from traditional bio-inspired algorithms. Unlike heuristic-based approaches that rely on pre-defined search strategies, DRL agents learn optimal decision-making policies through continuous interaction with the environment, effectively mapping high-dimensional raw sensory inputs directly to continuous or discrete control actions [10]. This inherent adaptability is particularly advantageous for Urban Air Mobility (UAM), where UAVs must navigate steady-state low-altitude wind fields and complex urban geometries with high flexibility. Recent advancements have established DRL as a cornerstone of autonomous navigation research. For instance, Tang et al. developed an enhanced Dueling Double Deep Q-Network (D3QN) framework utilizing Prioritized Experience Replay (PER) for UAV path planning in dynamic scenes, demonstrating significant performance gains over classical methods [11]. Similarly, Farid et al. integrated PER with L2-regularization within a Deep Q-Network (DQN) architecture to improve the success rate of autonomous UAV agents in 3D obstacle-cluttered environments, addressing the limitations of baseline DQN models in high-dimensional state spaces [12]. The efficacy of DRL extends to robust obstacle avoidance and path planning in multi-objective scenarios. Traditional analytical methods often struggle with complex control laws, a challenge overcome by the DQN-based collision avoidance algorithms [13]. For quadrotor UAVs, autonomous obstacle avoidance can be achieved through two-stage architectures comprising perception and decision modules, even in the absence of prior environmental information [14]. To balance efficiency and collision risk, globally guided reinforcement learning methods combine global path planning with DRL-based local planners, utilizing novel reward structures that generalize across diverse dynamic environments [15]. Performance in these dynamic settings can be further optimized by combining D3QN with PER to enhance sampling efficiency and value estimation stability, while Gök demonstrated that this combination significantly reduces path length and travel time for mobile robots without compromising safety [16]. Advanced architectural modifications have further addressed the constraints of partially observable and continuous dynamic environments. For mobile robots in unknown settings, reward-modified Double DQN (DDQN) algorithms utilize weighted reward functions and multi-level thresholds to stabilize the training process against erratic reward signals caused by moving obstacles [17]. In maritime contexts, integrating Long Short-Term Memory (LSTM) networks with DRL enables drones to mitigate decision biases stemming from partial observability by preserving long-term temporal dependencies [18]. Efficient exploration and faster convergence are also facilitated by incorporating prior knowledge and rule-based constraints into the DQN framework [19]. Furthermore, modern hybrid approaches have successfully addressed complex mission requirements. Liu et al. proposed an enhanced DRL strategy for UAVs in complex dynamic environments that incorporates a novel data storage system and an artificial potential field (APF) strategy to maintain high mission success rates [20]. These innovations collectively enable UAVs to navigate high-density dynamic obstacles while maintaining path optimality and mission reliability. Despite the theoretical advancements in UAV trajectory optimization, multi-UAV cooperative navigation in complex urban airspaces remains an evolving research frontier. Recent paradigms have begun to bridge game theory with multi-agent systems (MASs) to model strategic interactions and robust path optimization under high-dimensional uncertainties [21]. To address the real-time constraints and network complexity in large-scale swarms, autonomous connection selection mechanisms based on network formation games have been proposed to optimize positioning accuracy and computational efficiency [22]. Furthermore, hierarchical frameworks integrating LSTM-based dynamic predictors and multi-source consistency evaluation (e.g., wind-speed and airflow data) have demonstrated enhanced formation stability in challenging or GNSS-denied environments [23]. While these efforts have significantly improved cooperative reliability, current research specifically addressing multi-UAV operations under steady-state wind-field disturbances in dense urban canyons remains relatively sparse. Existing frameworks often overlook the tight kinematic coupling between 3D obstacle avoidance and multi-agent safety constraints when subjected to environmental drift. To address these limitations, this paper proposes NPD3QN, an architecture tailored for robust multi-UAV coordination in wind-disturbed urban airspaces. While the core algorithmic components—D3QN, N-step targets, and PER—are established reinforcement learning techniques, the main contributions of this work are summarized as follows:Kinematic-Aware Decision Framework: We integrate domain-specific kinematic wind field coupling into the state transition and reward design. This multi-objective reward explicitly incentivizes tailwind utilization and crosswind rejection, enabling robust navigation that standard DRL frameworks fail to capture.Task-Oriented Experience Replay: We propose an improved PER mechanism that regulates sampling density based on wind-induced drift patterns and collision outcomes. This mitigates conservative policy freezing and ensures safe exploration in high-variance environments.Enhanced Cooperative Trajectory Optimality: By synergizing the N-step update strategy with kinematic perception, the proposed NPD3QN effectively maps high-dimensional sensor data to streamlined, collision-free maneuvers, significantly reducing redundant detours under complex multi-agent constraints. Kinematic-Aware Decision Framework: We integrate domain-specific kinematic wind field coupling into the state transition and reward design. This multi-objective reward explicitly incentivizes tailwind utilization and crosswind rejection, enabling robust navigation that standard DRL frameworks fail to capture. Kinematic-Aware Decision Framework: We integrate domain-specific kinematic wind field coupling into the state transition and reward design. This multi-objective reward explicitly incentivizes tailwind utilization and crosswind rejection, enabling robust navigation that standard DRL frameworks fail to capture. Task-Oriented Experience Replay: We propose an improved PER mechanism that regulates sampling density based on wind-induced drift patterns and collision outcomes. This mitigates conservative policy freezing and ensures safe exploration in high-variance environments. Task-Oriented Experience Replay: We propose an improved PER mechanism that regulates sampling density based on wind-induced drift patterns and collision outcomes. This mitigates conservative policy freezing and ensures safe exploration in high-variance environments. Enhanced Cooperative Trajectory Optimality: By synergizing the N-step update strategy with kinematic perception, the proposed NPD3QN effectively maps high-dimensional sensor data to streamlined, collision-free maneuvers, significantly reducing redundant detours under complex multi-agent constraints. Enhanced Cooperative Trajectory Optimality: By synergizing the N-step update strategy with kinematic perception, the proposed NPD3QN effectively maps high-dimensional sensor data to streamlined, collision-free maneuvers, significantly reducing redundant detours under complex multi-agent constraints. To contextualize our contributions within the evolving landscape of UAV navigation, a nuanced comparative summary of related works is provided in Table 1. Recent studies have made significant strides in optimizing single-agent obstacle avoidance, enhancing sampling efficiency, and addressing foundational multi-agent coordination. Building upon these robust methodological foundations, our work specifically investigates the localized synergy between multi-agent IQL coordination and kinematic-aware reward structures under modeled steady-state wind disturbances. 2. Methodology 2.1. Problem Formulation The decision-making process for UAV path planning is formulated as a Markov Decision Process (MDP), which provides a rigorous mathematical framework for modeling sequential decision problems under uncertainty. Formally, the MDP is characterized by a tuple 〈S,A,P,R,γ〉. Here, S denotes the state space, representing the set of all possible configurations the UAV may occupy within the operational environment. A signifies the action space, encompassing the set of all executable maneuvers available to the UAV. The reward function R(st,at,st+1) quantifies the immediate feedback received upon executing action at in state st and transitioning to a subsequent state st+1. The transition probability P(st+1|st,at) defines the likelihood of shifting to state st+1 given the current state st and action at. Finally, γ∈[0,1] is the discount factor, which determines the present value of future rewards and governs the agent’s horizon of optimization. In the context of path planning, the agent interacts with the environment to acquire the necessary information to solve the MDP, a process facilitated through reinforcement learning (RL). The primary objective of RL is to derive an optimal policy that maximizes the cumulative reward, ensuring that the agent consistently selects the action that maximizes the Q-value. The optimal Q-value function is expressed via the Bellman equation:(1)Q*(st,at)=ER(st,at,st+1)+γmaxat+1Q*(st+1,at+1) where at+1 represents the potential action to be taken in the subsequent state st+1 to maximize the expected future value. By leveraging this framework, the agent evaluates the utility of various paths by synthesizing environmental data with its own internal state, thereby guiding the UAV to make informed and autonomous path planning decisions in complex scenarios. 2.2. Kinematic Environment and Wind Field Modeling In UAV path planning, the wind field is a critical environmental factor that significantly influences flight efficiency and operational safety. While tailwinds can minimize energy consumption and enhance operational efficiency, head-winds and cross-winds often induce trajectory drift, potentially compromising obstacle avoidance and the quality of multi-UAV collaborative path planning. To ensure that the learned path planning strategies remain executable and robust under atmospheric disturbances, this study incorporates a wind field into the RL training environment. The wind field is integrated as a core component of the environmental dynamics and reward feedback, enabling the agent to perform adaptive decision-making based on meteorological conditions. The wind field is modeled as a steady-state spatial vector field in three-dimensional space, defined as:(2)Vwind(x,y,z)=[U(x,y,z),V(x,y,z),W(x,y,z)]T where U, V, and W represent the wind velocity components along the three principal axes, respectively. Within the RL framework, wind field information is embedded into multiple stages of the decision-making pipeline:State Perception: Wind conditions are incorporated into the state vector, allowing the agent to directly perceive the wind direction and magnitude at its current coordinates.State Transition: The environment accounts for geometric offsets by superimposing the wind-induced drift onto the intentional displacement generated by discrete actions, thereby simulating realistic aerodynamic effects on the UAV trajectory.Reward Design: Directional feedback related to the wind field is introduced. This encourages the policy to satisfy obstacle avoidance and goal-reaching constraints while favoring advantageous wind directions and suppressing ineffective yaw caused by cross-winds, leading to more efficient and stable path generation under UAM constraints. State Perception: Wind conditions are incorporated into the state vector, allowing the agent to directly perceive the wind direction and magnitude at its current coordinates. State Perception: Wind conditions are incorporated into the state vector, allowing the agent to directly perceive the wind direction and magnitude at its current coordinates. State Transition: The environment accounts for geometric offsets by superimposing the wind-induced drift onto the intentional displacement generated by discrete actions, thereby simulating realistic aerodynamic effects on the UAV trajectory. State Transition: The environment accounts for geometric offsets by superimposing the wind-induced drift onto the intentional displacement generated by discrete actions, thereby simulating realistic aerodynamic effects on the UAV trajectory. Reward Design: Directional feedback related to the wind field is introduced. This encourages the policy to satisfy obstacle avoidance and goal-reaching constraints while favoring advantageous wind directions and suppressing ineffective yaw caused by cross-winds, leading to more efficient and stable path generation under UAM constraints. Reward Design: Directional feedback related to the wind field is introduced. This encourages the policy to satisfy obstacle avoidance and goal-reaching constraints while favoring advantageous wind directions and suppressing ineffective yaw caused by cross-winds, leading to more efficient and stable path generation under UAM constraints. For clarity and to support the interpretation of simulation results, the spatial wind field is sampled on a grid and visualized using sparse wind vectors. Figure 1 and Figure 2 illustrate the 2D and 3D visualizations of the wind field, respectively. The arrow directions indicate wind orientation, while the color and length represent the magnitude at each grid point. Darker colors and longer vectors signify higher velocities, with the wind speed ranging from 5 m/s to 12 m/s, corresponding to the “Gentle Breeze” to “Strong Breeze” categories on the Beaufort scale. This specific range is supported by meteorological studies of low-altitude urban airspaces, where the “urban canyon effect” significantly amplifies ambient wind speeds between high-rise clusters [26]. According to aerospace engineering guidelines for small-to-medium UAVs, 12 m/s represents a critical operational threshold for maintaining control stability [27]. The selection of a steady-state model over a time-varying one is a deliberate methodological choice. As highlighted in recent MARL surveys [28], utilizing a time-averaged steady-state field is essential during the initial multi-agent coordination phase to isolate kinematic coupling effects and prevent high-frequency stochastic gusts from exacerbating environmental non-stationarity. While we utilize a steady-state 3D wind field as a foundational step to validate the kinematic-aware reward function, the logical progression of Figure 1, Figure 2 and Figure 3 explicitly addresses this setup: Figure 1 and Figure 2 establish the baseline isolated wind kinematics, while Figure 3 illustrates the coupled wind-terrain environment used for the final multi-agent training. However, we acknowledge that this represents a time-averaged simplification of complex urban turbulence. Furthermore, a realistic urban architectural environment is constructed as the experimental scenario for multi-UAV collaborative path planning. This urban model is superimposed with the 3D wind field within a unified coordinate system to create a comprehensive simulation environment. Figure 3 provides a schematic of the urban 3D scene with the superimposed wind field, where gray entities represent buildings and blue vectors visualize the wind field distribution. 2.3. State Space and Action Space In this study, RL is utilized to optimize action selection for multi-UAV path planning. To facilitate effective path planning and RL training within a discrete 3D grid environment, the agent’s action space is discretized into a 26-neighbor movement set. To accurately characterize the impact of wind field disturbances on UAV trajectories, the state transition model is formulated by superimposing the drift displacement caused by the wind field onto the intentional displacement generated by the 26-neighbor actions. Let the position of the UAV in the 3D grid space at time step t be denoted as:(3)pt=(xt,yt,zt) In this study, each UAV is abstracted as a point agent located at the center of a grid cell. The corresponding displacement vector for action at is given by:(4)Δp(at)=(Δx,Δy,Δz) To account for the displacement induced by the wind field, the grid resolution l (measured in m/cell) is introduced as:(5)l=gridresolution The relationship between the physical coordinates and the grid indices is expressed as:(6)rt=pt·l Consequently, the intentional displacement of the 26-neighbor actions within the physical coordinate system can be defined as:(7)Δraction=Δp(at)·l The 3D wind field is modeled as a velocity vector field:(8)Vwind(r)=[U(r),V(r),W(r)]T where r represents the physical coordinate position. Given a single time step duration Δt, the drift displacement caused by the wind field at the current position rt during step t is formulated as:(9)Δrwind=Vwind(rt)·Δt Therefore, the state transition considering wind field disturbances can be represented as a 3D vector summation:(10)rt+1=rt+Δraction+Δrwind Subsequently, rt+1 is mapped back to the 3D grid index to obtain the next state pt+1:(11)pt+1=roundrt+1l Assuming the system contains M UAVs, the individual state of the i-th UAV at time step t is denoted as sti, and the joint state space of the environment is defined as:(12)St={st1,st2,…,stM} where St denotes the collection of all UAV individual states maintained by the environment at time step t. It is used to update the multi-UAV environment and check inter-UAV interactions, whereas each agent receives only its own local state sti as the neural network input under the IQL formulation. In practical sensor-driven path planning frameworks, the individual state sti serves as a comprehensive perceptual vector. It encapsulates the current kinematic data perceived by onboard sensors, including the relative distances to dynamic obstacles, the proximity to neighboring UAVs, and the localized wind velocity vectors Vwind(rt) detected by wind sensors. By formulating these multi-dimensional sensory inputs as the MDP state space, the DRL agent acts as an intelligent processing hub, directly mapping raw environmental perceptions to optimal collision-free maneuvers. To make the observation space fully reproducible, the individual observation of the i-th UAV at time step t is explicitly defined as a 9-dimensional vector:(13)sti=[Δxg,ti,Δyg,ti,Δzg,ti,dobs,ti,duav,ti,Uti,Vti,Wti,vwind,ti]T The first three components represent the relative displacement from the current position of the i-th UAV to its assigned target in the global coordinate frame:(14)Δxg,ti=xgoali−xti,Δyg,ti=ygoali−yti,Δzg,ti=zgoali−zti A comprehensive summary of the 9-dimensional observation vector is provided in Table 2. All UAVs use the exact same observation definition, component ordering, and coordinate frames. The absolute UAV position is not directly used as a neural-network input feature; rather, it is implicitly utilized by the environment to compute the relative target vector, local wind components, and distances. Furthermore, heading and action histories are excluded to maintain a strictly memoryless Markov property. To ensure gradient stability, the raw observations are processed using clipping and Min-Max normalization. Specifically, the local distance features (dobs,ti and duav,ti) are truncated at the maximum sensing range of 15.0 m. Subsequently, all 9 components are normalized into a [−1,1] range based on their respective theoretical or physical boundaries before being fed into the neural network. Accordingly, the individual action selected by the i-th UAV at time step t is denoted as ati, while the joint action of the multi-UAV system is expressed as:(15)At={at1,at2,…,atM} where At denotes the collection of individual actions executed simultaneously by all UAVs at time step t. The action collection At is an environment-transition variable used to apply wind-induced motion and check collisions, rather than a centralized decision variable. As shown in Figure 4, the proposed multi-UAV learning process follows a decentralized IQL formulation. Each UAV selects its action according to its own local observation:(16)ati∼πi(·|sti) where πi denotes the local policy of the i-th UAV. The environment receives the action collection and updates the UAV state collection according to:(17)St+1∼Penv(St+1|St,At) where Penv denotes the environment transition, including the 26-neighbor action displacement, wind-induced drift, obstacle checking, and inter-UAV collision checking. After the transition, each UAV receives its own local feedback:(18)(st+1i,rti,dti) where st+1i, rti, and dti denote the next local observation, local reward, and termination flag of the i-th UAV, respectively. The Q-network of the i-th UAV takes sti as input and estimates action values over its own discrete action set:(19)Qi(sti,a;θi),a∈Ai Thus, each UAV independently maintains and updates its own action-value function:(20)Qi(sti,ati;θi) As intuitively illustrated in Figure 5, this formulation ensures that the learning variables used for policy evaluation and action selection are consistently defined at the individual UAV level. 2.4. Reward Function In the context of multi-UAV path planning, the design of the reward function is of paramount importance as it guides the agents toward learning optimal policies. To enhance the autonomous exploration capability of the agents and ensure a thorough investigation of the state space, an ϵ-greedy strategy is integrated into the learning process. This strategy balances exploration and exploitation by allowing each agent to select a random action with a probability of ϵ and the current optimal action with a probability of 1−ϵ. Specifically, when the action of the i-th agent is chosen stochastically, the selection probability is defined as ϵ/|A(s)|. Conversely, if ati corresponds to the optimal action, the probability is formulated as 1−ϵ+ϵ/|A(s)|, where 1−ϵ represents the probability of intentional exploitation and ϵ/|A(s)| accounts for the probability of selecting the optimal action through random exploration. If ati is neither the optimal action nor selected randomly, the policy π(ati|sti)=0, indicating that such an action will not be executed. This mechanism is defined as follows:(21)π(ati|sti)=1−ϵ+ϵ|A(s)|,ifati=arg maxa Qi(sti,a)ϵ|A(s)|,otherwise where A(s) denotes the set of all feasible actions in state s, and |A(s)| represents the cardinality of this set (i.e., the number of available actions). The term Qi(sti,a) signifies the action-value function for the i-th agent taking action a in state sti. Regarding the reward structure, the formulation incorporates several critical factors: the number of steps taken by the agent, the alignment between the action direction and the wind field, the proximity to obstacles, potential collisions, and the attainment of the goal. To incentivize the agents to move toward their respective target points, a small positive reward is granted for reducing the distance to the goal, while a negative penalty is imposed for each step taken to encourage efficiency. Consequently, the reward increases as the agent approaches the target point and minimizes the total steps. This relationship can be mathematically expressed as:(22)Rdist=ω1d(pt−1i,pgoali)−d(pti,pgoali)−cstep where cstep=0.01 is the time penalty per step designed to promote shorter decision sequences. This specific scalar was empirically selected to provide a gentle heuristic push toward optimal paths without overshadowing the primary task reward (Rgoal=100). Preliminary sensitivity analyses indicate that the system’s convergence stability is robust to minor variations (e.g., within [0.005,0.05]). However, it is fundamentally sensitive to order-of-magnitude changes: an excessively large penalty (e.g., cstep>1.0) disrupts the gradient landscape, potentially inducing “reward hacking” where agents intentionally collide to terminate the episode prematurely and halt penalty accumulation. The term d(pti,pgoali) denotes the Euclidean distance from the current position to the target point for agent i in state sti, and ω1=0.1 is a weighting factor used to balance the distance reward. Considering the influence of wind field disturbances, agents moving in the direction of the wind can achieve higher speeds and improved energy efficiency. Therefore, the angle between the agent’s action direction and the wind flow is integrated into the reward function. A smaller angle θ results in a larger cosine value cosθ, yielding a higher reward, which is formulated as follows:(23)Rwind=ω2cosθ(ati,Vwind) where θ(ati,Vwind) represents the angular difference between the intended direction of action for agent i at time step t and the local wind velocity vector. The associated weight ω2 is set to 0.5. To facilitate effective obstacle avoidance, substantial negative rewards are assigned for collisions, while proximity-based penalties are applied to keep the agents at a safe distance from obstacles. The distance-dependent penalty for obstacles is expressed as:(24)f(d)=0,d≥dsafe1d+0.1,d0 Therefore, if the i-th UAV and the j-th UAV collide, both UAVs receive the inter-UAV collision penalty in their own local rewards. UAVs not involved in that collision do not receive this penalty. If multiple collisions occur at the same time step, the penalty of each UAV is determined by its own collision indicators Cobs,ti and Cuav,ti. The inter-UAV reward component for agent i is thus:(29)Ruav=−cuavCuav,ti−ω4∑j≠if(dij,t) where the weighting factors ω4 and cuav are assigned values of 0.03 and 15, respectively. Upon reaching the target destination, agent i receives a significant positive reward Rgoal. In all other instances, the composite reward is the summation of the aforementioned components, expressed as:(30)rti=Rgoal,iftargetisreachedRdist+Rwind+Robs+Ruav,otherwise The reward rti is the local reward used for the Q-network update of the i-th UAV. Each UAV updates its action-value function using its own local reward rather than the summed reward of all UAVs. The summed reward of the multi-UAV system is recorded as:(31)Rtsys=∑i=1Mrti where Rtsys denotes the summed reward used to describe the overall training performance of the multi-UAV system. It is not used as the TD target for any individual Q-network. The learning update of the i-th UAV is based on rti. This comprehensive reward scheme guides the multi-agent system to achieve efficient and collaborative path planning by accounting for diverse environmental and operational factors. Given the varying scales and magnitudes of different reward components, normalization is necessary to prevent any single term from dominating the learning process. The normalized reward is defined as follows:(32)Rnorm=R−RminRmax−Rmin where Rmax and Rmin are the maximum and minimum cumulative returns observed during the training phase over multiple episodes. By normalizing the composite reward, the feedback signal becomes numerically comparable across different agents, thereby facilitating global learning stabilization and policy optimization for the multi-UAV system. 2.5. Improved NPD3QN Algorithm In urban low-altitude environments characterized by wind field disturbances, cooperative path planning for multiple UAVs involves a high-dimensional joint state space. Under this setting, traditional tabular reinforcement learning struggles to generalize effectively in large-scale state spaces. Deep neural networks, however, can achieve high-dimensional mapping of action values through function approximation, offering an effective solution to this problem. This paper introduces deep neural networks to address low-altitude environments containing wind field disturbances, a method referred to as deep reinforcement learning. The Deep Q-Network (DQN) is a representative algorithm in the field of deep reinforcement learning. Its core objective is to minimize the loss function, thereby reducing the discrepancy between the estimated value and the actual return. However, DQN relies on a single neural network to perform the Q-value estimation task, which leads to overestimation of Q-values and deviation from the true values. Double DQN (DDQN) innovatively adopts a dual-network architecture, decoupling action selection and value evaluation into two separate networks, which successfully resolves the Q-value overestimation issue. Dueling Double DQN (D3QN) builds upon the dual-network architecture of DDQN by further designing the network structure into two parallel branches. One branch focuses on evaluating the value of the state itself, while the other emphasizes measuring the relative advantage of taking different actions in a specific state. D3QN not only inherits the advantages of DDQN in reducing estimation bias and improving stability but also enables a more detailed and precise analysis of the value of various states and actions in complex urban low-altitude environments. This allows the agent to make wiser and more efficient decisions in continuous and high-dimensional spaces, greatly expanding the potential of deep reinforcement learning in practical applications. Traditional D3QN algorithms only consider single-step updates, making it difficult to effectively utilize long-term return information. To enhance the ability to characterize long-term returns, this paper introduces an N-step update strategy based on the D3QN algorithm. At each decision point, the agent estimates the cumulative reward over multiple future steps and the state value estimate after the N-th step to update the value function of the current state. By comprehensively considering future information, the agent can plan an action path that is relatively optimal over a future period, avoiding the loss of larger subsequent returns due to immediate short-term interests. The return equation is expressed as:(33)yt=∑k=0N−1γkrt+ki+γN(1−done)Qtargetst+N,arg maxa Qonline(st+N,a;θi);θi− where rti is the immediate reward obtained by agent i at time step t. γ is the discount factor, used to balance the influence of current and future rewards on decision-making. st+N is the state of agent i after N steps from the current state. done is the termination flag, which takes the value of 1 when termination conditions such as reaching the target or collision occur, and 0 otherwise. Qonline is the action value function output by the online Q-network of agent i, with parameters θi. Qtarget is the action value function output by the target Q-network of agent i, with parameters θi−. The PER mechanism is a strategy in reinforcement learning for optimizing experience sampling. Compared with uniform sampling, the PER mechanism uses the Temporal Difference-error (TD-error) as a metric to evaluate the priority of sampled data. The TD-error refers to the difference between the online Q-value and the target Q-value; a larger error indicates that the data is more helpful for updating network parameters. Therefore, the larger the absolute value of the TD-error of a sample, the greater its guiding role in learning, and the more likely it is to be sampled. On this basis, the PER mechanism can accelerate the convergence speed of the network. The priority setting based on TD-error can be expressed as:(34)δ=yt−Qonline(st,at;θi) where δ represents the error of the current experience, that is, the deviation between the agent’s estimate of the optimal behavior in the current state and the actual return. To ensure that all experiences have a chance to be sampled and to avoid the priority being zero due to excessively small errors, a small offset ζ is added to the priority calculation of the l-th experience, expressed as:(35)pl=|δl|+ζ The existence of priority directly affects the sampling probability of the l-th experience, namely:(36)P(l)=plα∑kpkα where α is a tuning parameter used to control the degree of influence of priority on sampling. When α=0, PER degenerates into uniform sampling; when α>0, experiences with larger TD-errors are assigned higher sampling probabilities. When adopting the PER mechanism, non-uniform sampling changes the probability distribution of samples being replayed. To mitigate the sampling bias caused by this, importance sampling weights are usually introduced to weight and correct the gradient update, expressed as:(37)wl=(N·P(l))−βmaxjwj where N is the size of the experience replay buffer. The hyperparameter β controls the strength of the importance sampling correction. To explicitly mitigate the variance amplification effects caused by extreme priority distributions during early learning, β is linearly annealed to 1.0 as training progresses. This ensures that the non-uniform sampling bias is fully compensated for in the later stages, effectively stabilizing the gradient updates. However, when the priority distribution becomes highly skewed, extremely small replay probabilities may induce excessively strong importance sampling correction, which in turn enlarges the fluctuation of weighted updates and undermines training stability. To improve the numerical robustness of the replay mechanism, a stabilization treatment is further introduced in this study. The effective replay probability is regularized as:(38)P˜(l)=max(P(l),ϵp) where ϵp is a small probability floor used to prevent excessively small replay probabilities. Based on P˜(l), the stabilized importance sampling weight is computed as:(39)ω^l=(N·P˜(l))−β and the final normalized weight is defined as:(40)ωl=clip(ω^l,ωmin,ωmax)maxjclip(ω^j,ωmin,ωmax) where ωmin and ωmax denote the lower and upper clipping bounds of the importance sampling weights, respectively. By constraining the correction magnitude associated with extremely small replay probabilities, this treatment alleviates the variance amplification effect under severely imbalanced replay distributions and improves the stability of weighted updates. In multi-UAV scenarios with wind field disturbances and building constraints, the stochastic interaction between the agent and the environment leads to an imbalance in the proportion of positive and negative experiences in the replay buffer. This imbalance directly affects the learning direction of the agent, making it prone to repeatedly sampling a large number of negative experiences, thereby reducing the reuse efficiency of key experiences. This paper improves the PER mechanism by introducing a combined mechanism of marking experience types, controlling the probability of entering the replay buffer, and weighting sampling weights. This paper denotes a single experience transition as et and defines a positive experience indicator variable Ipos. To provide a precise and granular evaluation of environmental feedback, experience transitions are systematically categorized into four classes: (1) Task-progression transitions (reaching the target or safely reducing the distance to the goal); (2) Near-miss transitions (avoiding collision but entering the critical safety margin 0