An energy aware Q-learning framework for comprehensive coverage path planning in unknown complex environments Authors: Yao Xue, Chee Keong Tan, Wai Peng Wong Published: 2025 ABSTRACT AbstractIn post-disaster search and rescue scenarios, robotic path planning must operate in unpredictable, dynamic environments where conventional coverage path planning (CPP) algorithms often struggle to adapt. To address this challenge, we propose an intelligent path planning algorithm called PERM-QN (Q-learning with priority experience replay and memory network), designed for energy-aware, complete area coverage in uncertain terrains. PERM-QN integrates a dynamic weight reward function, priority experience replay, and a memory network to enable efficient, comprehensive exploration in complex, obstacle-laden environments. The dynamic weight reward function adaptively balances coverage, path length, and energy consumption across different phases of operation, and the priority experience replay mechanism accelerates learning convergence by focusing on high-value past experiences. Finally, the memory network expedites route planning in regions with similar terrain, reducing redundant exploration. Experiments in simulated post-disaster environments of varying complexity demonstrate that PERM-QN achieves more efficient and comprehensive exploration than traditional methods while maintaining robust performance. These findings highlight PERM-QN as an effective path planning solution for robotic search in complex, dynamic environments.Keywords: Reinforcement learning, Energy Expenditure, Complex and unknown environments, Full coverage path planningSubject terms: Computer science, Electrical and electronic engineering Introduction The post-disaster rescue operation is a race against time and is also characterized by inherent hazards. The complexities of collapsed infrastructure, dynamically evolving environments, and the urgent need for survivor extraction demonstrate the challenges and exigency of these scenarios. Additionally, the potential of secondary disasters and the presence of unknown environmental factors pose significant risks to traditional human-led search and rescue efforts. This context highlights the substantial potential for the advancement and deployment of autonomous robotic systems in disaster response. Recent progress has yielded many robots equipped with sophisticated navigation systems and sensor suites, enabling obstacle traversal and survivor localization based on pre-existing spatial information. However, the acquisition and modeling of information pertaining to previously unmapped environments remain a significant impediment to the further intelligent autonomy of rescue robots. Traditional full-coverage path-planning algorithms1 for rescue robots frequently rely on pre-existing environmental maps. However, in post-disaster scenarios, the prevalence of debris, unforeseen obstacles, and intricate terrain structures significantly impede the adaptability of these traditional algorithms to the dynamic and uncertain operational needs. Such environments are typically characterized by the following attributes, the primary being environmental unpredictability. Drastic environmental changes following a disaster event often render cartographic data obsolete, resulting in a completely unknown environmental structure at the mission’s outset. Furthermore, the operational context necessitates the rescue robots to be equipped with real-time exploration and mapping capabilities. Secondly, environmental complexity, characterized by a significant density of obstacles and non-Euclidean terrain, severely restricts the robot accessibility and maneuverability2. A large number of unknown obstacles are likely to cause the robots executing pre-planned paths to fall into unproductive cyclic behaviors, greatly limiting the detection efficiency. Finally, resource constraints, particularly the limited energy capacity of the robots, impose significant constraints on the overall exploration ability. Complex obstacle configurations are prone to induce redundant exploratory efforts, exacerbating energy depletion. Consequently, it is imperative for the robot to maximize the area of coverage achieved within its constrained energy budget. In response to the above problems, traditional methods such as the A* algorithm and the cattle farming algorithm exhibit notable limitations in terms of efficiency and adaptability within the complexities of post-disaster environments3. In recent years, reinforcement learning (RL) has been widely used in robot navigation tasks because of its inherent ability to adaptively learn optimal strategies. However, the realization of comprehensive full-coverage path planning within complex unknown environments using an RL framework remains a challenging research problem. Taking into account the above issues, this paper proposes a novel full-coverage path planning algorithm leveraging on Q-learning with priority experience replay and memory network abbreviated as PERM-QN (Priority Experience Replay and Memory Network with Q-learning), specifically considering the robot’s finite energy resources to improve coverage efficiency and meet its detection needs for unknown post-disaster environments. The primary contributions of this work are as follows. The formulation of a novel dynamic weight-reward function that comprehensively integrates coverage, path length, and robot energy consumption, thereby achieving a balanced trade-off between exhaustive exploration and operational efficiency. The formulation of a novel dynamic weight-reward function that comprehensively integrates coverage, path length, and robot energy consumption, thereby achieving a balanced trade-off between exhaustive exploration and operational efficiency. The design of a prioritized replay mechanism based on exploration efficiency and temporal decay factor to accelerate learning convergence and improve training efficacy. The design of a prioritized replay mechanism based on exploration efficiency and temporal decay factor to accelerate learning convergence and improve training efficacy. The incorporation of a memory network architecture that enables the robot to rapidly generate efficient paths in previously encountered environmental configurations, significantly reducing redundant exploration. The incorporation of a memory network architecture that enables the robot to rapidly generate efficient paths in previously encountered environmental configurations, significantly reducing redundant exploration. The proposed scheme has been rigorously validated through extensive simulations across diverse environments, encompassing randomly generated graphs of varying scales and obstacle densities. The empirical results demonstrate that the presented method effectively synthesizes traditional path planning principles with key performance indicators such as coverage area, energy consumption, and path length, proving its ability for comprehensive exploration in complex and uncertain environments. Related work In post-disaster search and rescue missions, Full Coverage Path Planning (FCPP)4 plays a crucial role in enabling autonomous robots to systematically explore and navigate complex and hazardous environments with unknown obstacles and dynamic terrain conditions. Traditional heuristic-based methods for FCPP, although extensively employed, exhibit limitations, including inefficiency in dynamic environments, lack of inherent adaptability5, and elevated computational demands6. Concurrently, RL-based approach, particularly Q-learning, has emerged as a promising alternative due to its ability to learn optimal navigation policies without a priori environmental knowledge. This section provides a comprehensive review of both traditional and reinforcement learning-driven FCPP methods. Traditional FCPP methods Classical Coverage Path Planning (CPP) methods primarily address systematic environmental coverage in the absence of a prior environmental knowledge. Early investigations into FCPP heavily relied on grid-based methods, where in the operational environment is discretized into cellular structure, and robotic agents systematically traverse all grid locations. A seminal contribution in this domain was the Path Transform method proposed by Zelinsky et al. (1993)7, which generated coverage paths through the creation of a potential field that directed the robot towards unexplored regions. However, these methods often exhibited redundant coverage patterns and demonstrated inefficiency in completely unknown environments. To improve operational efficiency, frontier-based exploration, introduced by Yamauchi (1997)8, guided robots to the boundary between known and unknown regions, ensuring incremental coverage. Although effective in structured environments, these methods struggled in dynamic and cluttered disaster areas, characterized by frequent obstacle reconfiguration. A significant advancement in FCPP involved the cell decomposition technique, which partitions the environment into manageable sections to facilitate systematic coverage. For example, in 2002, a method coined as Boustrophedon Cellular Decomposition (BCD)9 was proposed as a method for partitioning the operational space to enable systematic coverage. Nevertheless, these techniques require reliable real-time environmental updates to effectively address unknown terrain. Currently, tree-based span algorithms were proposed10 to optimize coverage paths; however, this approach often fails in environments of irregular shape. In recent years, to overcome some of the inherent constraints of classical algorithms, bio-inspired methods, such as the Bio-neural Network (BNN)11 have been introduced to achieve real-time adaptive coverage, thereby reducing redundant path generation. These approaches model the coverage task as a neural activation process, where robots are repelled from previously covered areas and attracted towards unexplored regions, leading to smooth, real-time navigation. Although demonstrating effectiveness in cluttered environments, BNNs necessitate meticulous parameter tuning and can incur substantial computational overhead. In addition, swarm intelligence and evolutionary algorithms, such as Ant Colony Optimization (ACO)12 and Artificial Bee Colony (ABC) optimization13, have been applied to FCPP problem. These algorithms improve path efficiency by leveraging multi-agent collaboration, but they often require prior knowledge of the environment or a partially known map, limiting their adaptability in fully unknown scenarios. To further improve coverage efficiency, recent metaheuristic approaches have introduced multi-objective optimization. For example, Zhou et al. (2023) proposed a cooperative path planning strategy with universal conflict resolution (UCR-FCE) and a behavior regulation mechanism, allowing large teams of robots to avoid collisions and maintain coverage efficiency in complex scenarios14. Similarly, a salp swarm-based exploration framework demonstrated significantly better area coverage, reduced redundant traversal, and improved multi-robot coordination compared to earlier single-objective methods15. Notably, Tran et al. (2023) addressed the re-planning challenge by proposing a swarm-based algorithm that combines frontier-led exploration with pheromone signaling, enabling multi-robot teams to repeatedly cover unknown, dynamic environments while avoiding moving obstacles16. Despite the advancements of these techniques, traditional FCPP methods often suffer from computational complexity, inefficient re-planning capabilities in dynamic environments, and a lack of real-time adaptability17. This has consequently driven increased interest in the exploration of RL-based approaches. RL-based FCPP methods RL methodologies empower robotic agents to acquire optimal coverage strategies through iterative interaction with the environment, rendering them particularly well-suited for operation in untrued and unknown environments. Q-learning, a prevalent RL algorithm, has been successfully applied to the FCPP problem by allowing robots to incrementally refine their navigation policies. In 2019, Piardi, L.18 introduced the application of Q-learning to coverage tasks, demonstrating its potential to reduce redundant coverage. However, this method struggled with scalability limitations due to the exponential expansion of the state space with increasing environmental complexity. To address this challenge, in 2023, Carvalho, JP, developed an online CPP algorithm based on tabular temporal difference learning methods19. Meanwhile, Deep Q networks (DQN) have been widely used to further improve the ability of Q-learning-based methods. One of the main advantages of DQN-based FCPP lies in its capacity for adaptive decision-making within dynamic environments. For instance, in 2024, Zhang et al. trained a DQN-based model to dynamically adjust coverage paths in response to the addition or removal of obstacles in real time20. In contrast to classical planners, RL-based agents react instantly to environmental changes without requiring complete path re-computation. In the same year, Jonnarth et al.21 used DQN to train the robot to determine navigation paths in previously unencountered environments. In addition, policy-gradient deep RL approaches have been explored. Garrido-Castaeda et al. (2025) showed that actor-critic algorithms can effectively train robots for complete coverage by learning navigation policies through direct interaction in a grid-modeled environment22. Recent literature has also frequently addressed the application of RL algorithms within the context of post-disaster scenarios23–26, highlighting their utility in navigating the specific complexities of such environments. For example, Ramezani et al. (2024) recently developed a fault-tolerant multi-agent RL framework for coverage planning, in which a team of drones and unmanned ground vehicles operates under a hybrid centralized-decentralized control scheme; this approach achieves resilient coverage performance despite intermittent communication loss and sensor failures27. Despite the numerous advantages offered by the RL-based FCPP schemes, several fundamental challenges persist. Unlike classical methods, RL methods do not inherently ensure complete area coverage, requiring the incorporation of sophisticated reward shaping functions and constraints28. To improve coverage completeness and policy generalization, Carvalho and Aguiar (2025) introduced a value-based RL framework with a safe action masking scheme and curriculum training, allowing zero-shot coverage planning across varying map sizes while ensuring comprehensive and collision-free exploration29. Moreover, RL models typically demand substantial offline training time and significant computational resources prior to deployment in real-world applications30. Recent studies have also explored deep reinforcement learning and the heuristic search for emergency task allocation in aerial vehicle systems. For example, Guo et al.31 proposed GENIS, a graph neural neighborhood search algorithm that integrates dual-graph convolutional networks with reinforcement learning to improve scheduling efficiency under energy and routing constraints. Similarly, Yu et al.32 introduced a multi-start collaborative variable neighborhood search (MCVNS) framework with knowledge transfer for multi-aircraft collaborative task assignment. Although these studies also target post-disaster missions and achieve strong results in scheduling and allocation, they assume prior knowledge of task locations and demands, rendering them less directly applicable to completely unknown post-disaster environments where robots must autonomously explore, perceive, and map the scene under stringent energy budgets. Methodology To enable full-coverage exploration in autonomous robotic systems operating in complex unknown environments, this paper proposes PERM-QN, a novel reinforcement learning approach that leverages Q-learning enhanced with prioritized experience replay and a memory network for efficient navigation and path optimization. By integrating Q-learning and memory network mechanisms, the proposed approach enables robots to achieve efficient path planning and exploration capabilities in complex unknown dynamic environments. Furthermore, the methodology incorporates multi-objective optimization considerations, specifically addressing coverage efficiency, energy consumption, and path length minimization. In essence, Fig. 1 shows the overall system architecture and the entire operational flow in this paper. The PERM-QN algorithm tackles the challenge of enabling autonomous robots to fully explore and efficiently navigate unknown complex post-disaster environments. The process begins with the initialization of a Q-table, which serves as a value function, estimating the long-term reward for each state-action pair. The robot then enters a cycle where it selects an action based on its current state and the Q-table, often employing exploration-exploitation strategies. This action is then implemented in the environment. Robot exploration using Q-learning informed by path length, coverage, and energy levels, with prioritized experience replay and memory network. The robot receives a reward that reflects the effectiveness of its action with respect to several key objectives: minimizing path length, maximizing coverage of the environment, and managing energy levels (indicated as High or Low cost). These factors collectively shape the learning signal. The resulting experience (current state, chosen action, received reward, and the subsequent next state) is stored, and a Prioritized Experience Replay mechanism strategically samples these memories, prioritizing significant transitions to accelerate learning. Simultaneously, a Memory Network is utilized to store and retrieve information about the explored environment, enhancing the robot’s ability to make informed decisions in complex or partially observable scenarios. The received reward and the information from the memory network are then used to update the Q-table, refining the estimated values of state-action pairs. This update rule forms the core of the Q-learning process. The updated Q-table, along with the insights from the memory network, guides the robot’s future action selection, creating a feedback loop that enables the autonomous system to progressively learn an optimal policy for comprehensive exploration and efficient navigation within the unknown grid world. Environment modeling Grid environment This research uses Matlab and its RL toolbox to model and simulate the entire algorithm. The robotic exploration scenario is emulated through a discrete grid-based representation of the environment. This grid environment is modeled as a two-dimensional discrete state space, where each cell represents a potential location for robot movement or exploration. To simulate autonomous exploration of completely unknown scenes, the initial state of all cells within the grid map is designated as the “unknown” state, which implies that the robot can only gradually reveal the unknown area through continuous movement and sensor acquisition. Within this simulated environment, the location and quantity of obstacles are randomly generated. The robotic agent learns the layout of the obstacle in the environment solely through exploratory actions, specifically by attempting to navigate to the target locations and detecting the presence or absence of obstacles in those locations. As shown in Fig. 2, the left panel illustrates the obstacle configuration generated using MATLAB, while the right panel presents a schematic representation of the simulated post-disaster environment derived therefrom. This simulation is designed to mimic the complexities and uncertainties encountered in real-world post-disaster scenarios, including randomly distributed obstacles and initially unknown terrain. Through sufficient number of simulation training episodes, the robotic agent is enabled to learn effective navigation strategies for diverse obstacle shapes and dynamic spatial arrangements. This learning process enhances the robot’s capacity to effectively operate in completely unknown post-disaster environments. Environment for simulation: (Left) Grid-based occupancy, (Right) Continuous space analogy. Robot capabilities In this research, the robot’s operation space is discretized into four cardinal directions of motion: forward (up), backward (down), leftward and rightward. The robot’s energy autonomy is simulated by imposing a finite limit on the total number of permissible movement steps. At the same time, the robot’s perceptual capabilities are modeled to emulate the function of onboard sensors in actual exploration. Within the scope of this study, the robot’s perception is restricted to the immediately adjacent cell in its forward direction, as shown in Fig. 2. Energy-aware Q-learning framework The core of the algorithm in this research is to use the Q-learning algorithm to acquire an optimal strategy for achieving full coverage search of the robot within a completely unknown environment. This is primarily accomplished through iterative updates to the Q-value table and subsequent strategic decision-making based on the learned values. The Q-value update rule is formally defined by Eq. 1. where Q(s, a) is the Q-value associated with selecting action a under current state s; denotes the learning rate, which governs the step size of the Q-value update; r signifies the instant reward; is the discount factor, which balances the influence of immediate rewards and long-term rewards; and represents the maximum Q-value attainable in the subsequent state , characterizing the optimal long-term reward. Reward function The reward function is a critical component of RL, directly affecting the direction of learning and the efficiency of the agent. Traditional reward functions, often based on a fixed, single-objective design, lack the adaptability and flexibility needed for dynamic environments. Consequently, they may be suitable for simple scenarios, but exhibit significant limitations when applied to the complexities of our setting. In our approach, the design of the reward function incorporates a multi-objective optimization framework. By introducing dynamic weights, non-linear functions, and path weights, we propose a novel reward mechanism that better addresses the needs of robots operating in complex environments. The reward function is mainly composed of the three components: coverage reward, energy reward, and path length reward. Coverage rewards The first component is the coverage reward, designed to encourage the robot to cover as much of the unexplored area as possible while discouraging revisiting previously explored region. The coverage reward function is determined by Eq. 2. In Eq. 2, StepCount denotes the number of steps elapsed within the current episode, while the parameter T defines the periodicity of the sinusoidal modulation. This sinusoidal term is strategically introduced to act as a dynamic exploration bonus or reward shaping component within the coverage incentive. It serves as a soft perturbation mechanism that periodically amplifies or attenuates the agent’s immediate incentive to explore, thereby actively promoting behavioral diversity. This controlled, oscillatory behavior is particularly crucial during the mid-stage of exploration, where agents commonly encounter local loops, repetitive transitions, or premature stagnation in grid-based environments. By systematically varying the reward landscape, the sinusoidal modulation encourages the agent to re-evaluate its current local strategies and explore alternative, potentially more fruitful paths. The use of a sinusoidal function, specifically, ensures a smooth and bounded oscillation of this incentive, introducing controlled variability without destabilizing the learning process. This ultimately improves coverage completeness and effectively mitigates the aforementioned stagnation issues, facilitating a more thorough sampling of the environment. The typical range of T values employed in our experiments, summarized in Table 1, was empirically selected to align with the expected duration of this critical mid-stage exploration phase, ensuring optimal perturbation during the period when it is most beneficial for policy optimization. Table 1Key Hyperparameters in PERM-QN Framework.ParameterSymbolTypical Range / DescriptionLearning rate0.001–0.05Discount factor0.90–0.99Low-energy thresholdn0.15–0.25Sigmoid slope constantk8–12Penalty scaling constant0.5–2.0Coverage modulation periodT50–150 stepsPath success weight0.4–0.6Energy efficiency weight0.2–0.4Path length weight0.1–0.3 Key Hyperparameters in PERM-QN Framework. Path length reward The second component of the reward function is the path length reward, quantified by Eq. 3. where K is a regulatory factor that modulates the influence of step count on the penalty term. Energy reward In realistic post-disaster exploration scenarios, the robot energy status of the robot is also a critical consideration for rescue operations. Complex and unknown irregular terrains often lead the robot to engage in indiscriminate exploration of hard-to-reach regions, increasing the risk of energy depletion and potential mission failure. To address this, we introduce a novel energy-aware reward mechanism that adaptively modulates the robot’s exploration strategy according to its remaining energy, thereby encouraging extensive exploration when energy is sufficient to maximize coverage of unknown areas. Conversely, under low-energy conditions, exploration tends to be conservative, mitigating high-risk, long-distance exploration, and preserving energy for the robot’s return. The energy reward formula is defined in Eq. (4) where represents the percentage of remaining energy. Additionally, and are coefficients used to dynamically adjust the energy reward and penalty amplitude, respectively, which are determined by the following equations. where , and are constant. In our framework, n is primarily conceived as a configurable safeguard to ensure the robot retains sufficient power to reliably return to its base station without becoming lost or stranded within the exploration map. Rather than being derived from a formal theoretical model with a fixed value, n serves as a critical parameter that balances two competing objectives: maximizing coverage efficiency through deeper exploration, and minimizing the inherent risk of energy depletion before a successful return. A smaller value for n permits more aggressive and extensive exploration, but inherently increases the risk of the robot running out of energy. Conversely, a larger value prioritizes safety by ensuring greater energy reserves, though this may consequently limit the overall exploration range. This design intentionally offers flexibility, allowing for robust deployment in diverse environments where the desired trade-off between exploration aggressiveness and energy security can vary based on specific mission requirements and environmental characteristics. Through extensive simulation testing across a wide spectrum of obstacle densities and map sizes, we have empirically validated that setting n within a reasonable, empirically determined range consistently enables reliable return behavior without degrading overall coverage performance. Figure 3 illustrates the influence of energy optimization on robot exploration behavior at different energy levels. In each subfigure, the yellow star denotes the starting point, the green triangle marks the transition from high to low energy level, and the blue diamond represents the final position of the robot. The background shading indicates the spatial attractiveness of unexplored areas, where a darker blue corresponds to a higher attraction. In high-energy scenarios (Fig. 3a and c), the use of the segmented reward function enhances the attraction of remote unexplored regions, encouraging the robot to fully explore the map. This leads to more complete coverage and improved exploration efficiency. Notably, both the baseline and the energy-optimized strategies perform similarly under sufficient energy, as the robot has enough energy to explore and return. However, when the energy level drops to a critical threshold (Fig. 3b and d), the effect of energy optimization becomes more pronounced. Without energy-aware regulation (Fig. 3b), the robot continues to travel toward distant attractive areas, resulting in excessive energy consumption that could potentially jeopardize its return capability. In contrast, the energy-optimized strategy (Fig. 3d) dampens the attraction of distant targets in low-energy states. This prompts the robot to explore more cautiously, limiting needless motion and saving enough energy for the return journey. Conceptual diagram of the robot’s exploration performance within an unknown environment. a and b Performance without energy optimization, and c and d Performance with energy optimization. By combining the above three reward functions and the weight coefficients, the overall reward function of our algorithm can be formulated as denoted in Eq. (7). In Eq. 7, the coefficients , and serve to dynamically balance the three reward components: coverage incentive, energy-aware exploration, and path efficiency. These weights are formulated as adaptive functions of the robot’s battery level and exploration progress, allowing the agent to prioritize different behaviors at different stages of the mission. The effective ranges and modulation characteristics of each weight are summarized in Table 2. where , , and are constant offsets used to ensure that the weights have a minimum value in some states. The pseudo-code for Q-learning is outlined in Algorithm 1. Table 2Comparison of different methods in terms of path length, coverage, and search efficiency under a 3030 grid map.Map Complexity (%)MethodPath Length (Mean ± Std)Coverage (Mean ± Std)Efficiency (Mean ± Std)10Basic Q-learning970.55 ± 29.120.86 ± 0.02580.7177 ± 0.030510AntBot-EX972.45 ± 29.170.975 ± 0.02920.8121 ± 0.034410PERM-QN891.85 ± 26.760.9965 ± 0.02990.9050 ± 0.038410PPO980.00 ± 15.000.984 ± 0.00800.8133 ± 0.014115Basic Q-learning925.20 ± 27.760.835 ± 0.02500.6904 ± 0.029315AntBot-EX944.26 ± 28.330.956 ± 0.02870.7745 ± 0.032915PERM-QN858.80 ± 25.760.9959 ± 0.02990.8871 ± 0.037615PPO960.00 ± 20.000.972 ± 0.01200.7746 ± 0.018820Basic Q-learning865.65 ± 25.970.804 ± 0.02410.6687 ± 0.028420AntBot-EX965.83 ± 28.970.950 ± 0.02850.7082 ± 0.030020PERM-QN825.76 ± 24.770.9954 ± 0.02990.8679 ± 0.036820PPO950.00 ± 25.000.940 ± 0.02000.7124 ± 0.024125Basic Q-learning805.73 ± 33.170.735 ± 0.02210.6157 ± 0.031425AntBot-EX985.84 ± 29.580.866 ± 0.02600.5929 ± 0.025225PERM-QN790.77 ± 23.720.9707 ± 0.02910.8286 ± 0.035125PPO930.00 ± 35.000.880 ± 0.03500.6387 ± 0.035030Basic Q-learning809.64 ± 34.290.666 ± 0.02000.5182 ± 0.026930AntBot-EX1043.92 ± 31.320.804 ± 0.02350.4852 ± 0.020330PERM-QN747.00 ± 22.410.9421 ± 0.02830.7945 ± 0.033730PPO910.00 ± 45.000.810 ± 0.04000.5608 ± 0.039235Basic Q-learning792.04 ± 33.760.571 ± 0.01710.4217 ± 0.022035AntBot-EX1396.28 ± 41.890.900 ± 0.02700.3771 ± 0.016035PERM-QN703.82 ± 21.110.8998 ± 0.02700.7479 ± 0.031735PPO870.00 ± 60.000.730 ± 0.05500.4909 ± 0.050140Basic Q-learning796.98 ± 43.910.458 ± 0.01380.3103 ± 0.019540AntBot-EX1582.97 ± 47.490.846 ± 0.02540.2886 ± 0.012240PERM-QN660.76 ± 19.820.823 ± 0.02470.6726 ± 0.028540PPO820.00 ± 65.000.680 ± 0.06000.4478 ± 0.0531 Comparison of different methods in terms of path length, coverage, and search efficiency under a 3030 grid map. This enhanced Q-learning algorithm aims to balance multiple objectives dynamically based on an agent’s battery level. It initializes a Q-table and learning parameters. For each episode, the environment resets, and the agent selects actions using an epsilon-greedy policy. After taking an action, it observes reward components (coverage, energy, path length), which are weighted based on the current battery level to calculate a total reward. The Q-table is then updated using this reward and the estimated future rewards. The agent’s state is updated, and the exploration rate gradually decreases over episodes, allowing the agent to learn an efficient and energy-aware policy. Enhanced Q-Learning with Dynamic Multi-Objective Rewards Priority experience replay and memory network utilization Priority experience replay In the proposed algorithm, the robot operates within an initially unknown and dynamic grid environment, where a comprehensive exploration is required for effective decision-making. However, inherent complexity, environmental stochasticity, and sparse feedback often lead to instability during training and a tendency to converge prematurely to suboptimal policies. Furthermore, the non-stationary nature of the environment exacerbated by the agent continually updating its policy based on data sampled from its current strategy, causing the target distribution to drift over time, undermining learning stability. To mitigate these challenges and support the multi-objective nature of the task (e.g., maximizing coverage, minimizing path length, and maintaining sufficient energy), we incorporate a prioritized experience replay (PER) mechanism into the learning framework. Unlike conventional uniform sampling, PER ranks transitions based on their temporal difference error and exploration utility (e.g., coverage efficiency, path reward), ensuring that high-impact experiences are sampled more frequently. This prioritization facilitates more efficient policy refinement, accelerates adaptation to novel environmental configurations, and enhances learning robustness-especially in the presence of sparse or delayed rewards. The basic formula is shown in Eq. (11). where E is the exploration efficiency while is the multi-step rewards. where C is the explored area of map, is the immediate reward in step t, is the discount factor that controls the weight of future rewards and N is the number of future steps considered. The priority selection probability formula is shown in Eq. (13). The probabilistic selection mechanism ensures that experiences exhibiting higher exploration efficiency are assigned a greater probability of being sampled from the priority replay buffer, prioritizing the repeated learning of key experiences. Simultaneously, the formulation inherently prevents the complete exclusion of low-priority experiences, thereby maintaining a degree of diversity within the experience pool. Therefore, the formula for updating the Q value after combined with replay of the experience is given in Eq. (14). Priority Replay Buffer Integrating the prioritized experience assignment with the multi-step reward framework, the proposed algorithm dynamically balances the imperatives of immediate exploration efficiency with the pursuit of long-term global optimization. This synergy enables the achievement of a rapid and efficient full coverage search strategy. The procedural outline of the algorithm is presented in Algorithm 2. The Priority Replay Buffer algorithm enhances experience replay by prioritizing significant transitions. It initializes an empty buffer and, for each new experience, calculates its priority based on exploration efficiency and multi-step reward. This experience, containing state, action, reward, next state, done flag, and priority, is then added to the buffer. If the buffer is full, the lowest-priority experience is replaced if the new one has a higher priority, ensuring the buffer retains more valuable data. When sampling experiences for training, the algorithm calculates sampling probabilities proportional to each experience’s priority. This non-uniform sampling strategy ensures that higher-priority experiences are sampled more frequently, allowing the learning process to focus on the most informative transitions. By prioritizing the replay of important experiences, the algorithm aims to accelerate learning and improve the stability of reinforcement learning agents. Memory network utilization In addition to the inherent complexity of the environment and the challenges posed by multi-objective optimization, empirical observations across multiple experimental runs indicate that structural similarities often emerge among exploration trajectories. However, the robot initially lacks any knowledge of its surroundings in each episode, necessitating repeated exploratory efforts to gradually acquire an effective policy. This leads to a substantial amount of redundant exploration in complex and partially observable environments, especially during early-stage learning. To address this inefficiency, we introduce a memory network module to complement the prioritized experience replay mechanism. This module retains high-utility historical exploration trajectories from previous episodes and enables the agent to perform memory-guided recall during online decision-making. By retrieving structurally similar past paths and attempting local state-level matching, the agent can reuse previously successful behaviors in analogous substructures. This strategy improves sample efficiency, especially during early-stage exploration, and promotes generalization across common layout patterns. A schematic overview of this mechanism is shown in Fig. 4. Schematic illustration of the trajectory similarity-based retrieval mechanism used to identify and utilize structurally similar paths from memory. Figure 4 illustrates how the memory network module operates alongside the prioritized experience replay. The central grid represents the agent’s memory (M), storing several historical exploration trajectories (Trajectory 1, Trajectory 2, Trajectory 3) within different encountered environmental configurations. These trajectories, depicted by solid and dashed lines, represent sequences of states and actions that have previously yielded high utility, such as efficient exploration or reaching a goal. When the agent encounters a new environmental configuration (implied by the current state), the memory network employs a trajectory similarity-based retrieval mechanism. This process compares the current situation with the stored trajectories to identify those that closely match the present environmental structure, potentially based on the arrangement of obstacles or other relevant features. The arrows pointing from the central memory grid to “Example 1,” “Example 2,” and “Example 3” illustrate this retrieval process. Each example shows a recalled segment of a past trajectory (blue dashed line), overlaid on a slightly different but structurally similar current configuration. Once a high-similarity trajectory is retrieved, the agent attempts to match its current state to locally relevant segments of that trajectory. If such a state is found and its successor is both valid and unexplored, the agent executes a short one-step reference action. This matching process is designed to be lightweight and opportunistic, allowing the agent to reuse partial strategies without requiring full trajectory reproduction or exact environmental alignment. Trajectory similarity is assessed using a Euclidean distance-based metric (Eq. 15), which quantifies global structural similarity between stored paths and the current state context. This enables coarse retrieval of relevant trajectories without requiring exact environment identity. Execution, however, relies on a local match condition—only if a state exactly matches a state from a retrieved trajectory , and the successor state is both valid and unexplored, does the agent adopt a short-term one-step guidance action. This layered retrieval-and-matching mechanism provides a lightweight form of episodic recall that remains compatible with the Q-learning policy. In this equation, denotes the agent’s current state vector, while refers to a previously visited state stored in the memory network. The term n represents the dimensionality of the state space. To ensure high-quality memory contents, we also implement a dynamic storage policy based on weighted utility scoring. New trajectories are assigned a path weight based on three criteria: success frequency, energy efficiency, and trajectory compactness, as defined in Eq. 16. When memory capacity is reached, the lowest-weighted trajectory is replaced, ensuring retention of only the most transferable experiences. where is the success rate of the path and are adjustable parameters used to balance the weights of different factors, and their sum is equal to 1, ensuring balanced contributions of success rate, efficiency, and path length. Memory-Guided Recall Strategy Algorithm 3 succinctly summarizes our proposed memory recall logic, a mechanism crucial for enhancing exploration efficiency. This process begins with the retrieval of high-utility trajectories from the agent’s memory module. Specifically, although the Euclidean distance (Eq. 15) does not directly reflect topological features, it is employed as a rough but computationally efficient metric to retrieve historical trajectories whose spatial distribution is broadly similar to the current environment. This initial retrieval acts as a first-stage filter, aiming to identify candidate objects that are structurally similar in terms of spatial layout. On this basis, the reuse decision further introduces the matching of local states, where the agent iteratively searches for a precise “match” between its current state and a segment within these stored, high-weight trajectories. Upon finding such a match, the agent identifies the immediate successor state along that same high-utility trajectory. Provided this successor state is both valid (i.e., within the known free space) and currently unvisited, it is adopted as a one-step guidance goal. This two-layer approach enables the agent to benefit from relevant previous experience without relying on strict topological equivalence. This opportunistic recall is intentionally designed to activate only when the agent experiences indicators of local stagnation, such as a plateau in coverage improvement or the occurrence of repeated transitions in its exploration. This helps the agent effectively escape local loops or dead ends. Crucially, this guidance step does not override the agent’s learned Q-policy; the agent seamlessly reverts to normal learning immediately after executing this single recall action. This modular design facilitates the effective reuse of valuable prior knowledge for targeted short-term guidance while preserving the inherent flexibility and adaptability of the ongoing online learning process. To ensure that only high-quality paths are retained in memory, we incorporated a multi-factor path evaluation scheme (Eq. 16), which judiciously balances prior success rate, energy efficiency, and path compactness. The exploration strategy in our framework extends beyond a rudimentary -greedy approach, incorporating several sophisticated mechanisms to effectively address the challenges of redundant exploration and to facilitate a progressive design from local to global exploration in unknown environments, thereby ensuring both coverage completeness and efficiency. Firstly, our reward function is meticulously shaped with dynamically weighted components (Eq. 7), where the coefficients governing different reward terms are adaptively varied based on the agent’s real-time energy state and its overall exploration progress. This sophisticated weighting scheme enables the agent to prioritize comprehensive coverage in the early stages of an episode, gradually shifting its focus towards energy-aware efficiency and path optimization as exploration advances. This dynamic adjustment effectively orchestrates a staged progression, guiding the agent from intensive local exploration to a broader spatial coverage. Secondly, the memory network provides an additional, crucial layer of exploration enhancement. It enables the episodic recall of structurally relevant, high-utility paths encountered in prior experiences. This intelligent reuse of trajectories in similar sub-regions significantly reduces the need for redundant re-exploration, optimizing the search process without necessitating global map knowledge, which is typically unavailable in our assumed unknown environments. Finally, a dedicated local stagnation detection mechanism is implemented, which monitors for indicators such as a decay in the coverage rate or the occurrence of repetitive state transitions. Upon detecting such stagnation, this mechanism actively triggers short-term, memory-guided deviations. These dynamic responses are designed to immediately break local loops and effectively guide the agent out of local minima, prompting it to seek out and explore previously unvisited regions. Collectively, these integrated mechanisms form a hybrid exploration strategy that transcends the limitations of a purely -greedy policy. This comprehensive structure intelligently balances random exploration, reward-driven motivation, and memory-guided adaptation, thereby ensuring robust coverage completeness and maximizing exploration efficiency in complex, unknown environments. Result analysis and discussions Table 1 serves as a comprehensive reference for the key parameters utilized within this study’s methodology. This table outlines the recommended value ranges for these crucial coefficients, which are integral components of the unified reward function. By clearly detailing these empirically validated intervals, Table 1 significantly enhances the reproducibility and transparency of our research. The numerical values adopted for specific parameters, namely , , and are not arbitrarily selected but meticulously derived through a series of controlled simulations. This iterative process was designed to achieve a critical balance among three distinct objectives: fostering exploration within the model, maintaining a robust awareness of energy dynamics, and ensuring the overall stability of the learned policy. Our extensive empirical evaluations consistently demonstrated that the method exhibits robust convergence behavior when these parameters are set within their identified and validated intervals. To further improve clarity and facilitate the replication of our findings, we have now explicitly reformulated these coefficients to highlight their role as direct components of the unified reward function, and have consolidated all key parameter information into the newly provided Table 1. The experimental evaluation presented in this paper comprises four controlled comparative studies, benchmarking our proposed algorithm (PERM-QN) against the AntBot-EX algorithm33, the PPO-based DRL coverage agent21 and the baseline Q-learning algorithm. It is worth pointing out that the baseline Q-learning approach used here does not incorporate energy awareness, prioritized experience replay, or a memory network. The relative performance of these methods is evaluated by comparative analysis of the explored area, exploration efficiency, and exploration time. To ensure a rigorous and unbiased evaluation, key experimental parameters, including map area, environmental complexity, and initial energy level, were maintained constant under all experimental conditions. All reported results are averaged over 500 independent runs to ensure statistical robustness and reduce the impact of stochastic variability. Figure 5 provides a visual example of the experimental outcomes. Figure 5a shows the training result for a map with 10 x 10 grids, contrasting the initial state of complete environmental uncertainty with the final, fully mapped representation achieved through the robot’s exploration process. While Fig. 5b depicts the exploration results of the robot in a new unknown map with 20 x 20 grids after the robot completes the exploration. Based on the experimental results (not limited to the specific map in Fig. 5), robots equipped with the proposed PERM-QN demonstrate a robust capability to navigate and construct detailed environmental maps. This effectiveness, which extends across diverse scenarios characterized by varying map complexity, obstacle density, dynamic environmental conditions, and map dimensions, will be substantiated through the subsequent in-depth analysis. Robot exploration and mapping. a Left figure: Training result of PERM-QN. b Right figure: Test result on a new map. Coverage and efficiency analysis of PERM-QN Based on the data presented in Table 2 and Fig. 6, a comparative analysis of the basic Q-learning method, the AntBot-EX method, the PPO-based DRL baseline, and our proposed PERM-QN method reveals the following insights. Comparison of search efficiency for Basic Q-learning, AntBot-EX, PPO and PERM-QN under varying environmental complexity. a Map size 20 20. b Map size 30 30. c Map size 40 40. Table 2 presents an in-depth quantitative comparison of path length, coverage, and search efficiency for Basic Q-learning, AntBot-EX, PPO, and the proposed PERM-QN method across varying levels of map complexity (10%–40%). These three metrics collectively capture not only how extensively the environment is explored (coverage) but also how efficiently and intelligently the agent navigates through it (path length and efficiency). Across all complexity levels, PERM-QN consistently demonstrates superior performance, achieving the shortest path length in every scenario. This advantage becomes increasingly pronounced as map complexity rises, most notably at 40% complexity, where PERM-QN requires only 660.76 ± 19.82 units to complete exploration, compared to 796.98 ± 43.91 units for Basic Q-learning, 820.00 ± 65.00 units for PPO, and a much longer 1582.97 ± 47.49 units for AntBot-EX. This trend highlights PERM-QN’s ability to make more informed navigation choices, minimizing redundant exploration even in cluttered, obstacle-dense environments. In terms of coverage, PERM-QN and AntBot-EX both maintain values above 80% across all complexity levels, showing robustness in ensuring the environment is thoroughly explored. However, PERM-QN achieves this with significantly shorter paths, demonstrating its balance between completeness and efficiency. Basic Q-learning and PPO, on the other hand, exhibit declining coverage as complexity increases, with coverage dropping to 45.8% and 68.0%, respectively, at 40% complexity, indicating their diminished ability to handle more challenging maps. Finally, the efficiency score, defined as the ratio of coverage to path length, further emphasize PERM-QN’s strength. Even at higher complexities, PERM-QN maintains higher efficiency scores (e.g., 0.6726 ± 0.0285 at 40% complexity) than all other baselines, signifying its capacity to maintain purposeful, goal-directed exploration. In contrast, AntBot-EX, despite its strong coverage, suffers from very low efficiency (dropping to 0.2886 ± 0.0122 at 40%) due to excessive wandering and long paths. Collectively, these results confirm that PERM-QN not only explores more intelligently but also generalizes better under increasingly demanding exploration scenarios, making it a robust and scalable solution for complex environments. Figure 6 provides a detailed comparison of search efficiency across Basic Q-learning, AntBot-EX, PPO, and the proposed PERM-QN method under increasing levels of map complexity for three map sizes (20 20, 30 30, and 40 40). A clear performance hierarchy emerges: PERM-QN consistently outperforms all baselines, maintaining the highest search efficiency across all map sizes and complexity levels. This advantage becomes more pronounced as map complexity rises, highlighting PERM-QN’s exceptional adaptability and strong generalization to challenging, cluttered environments. Basic Q-learning, in contrast, demonstrates the steepest decline in efficiency, reflecting its limited capacity to adapt to increasingly complex terrains. AntBot-EX, while initially competitive and showing moderate resilience at lower complexities, suffers from substantial efficiency degradation as map density increases. PPO, representing a more modern deep reinforcement learning baseline, delivers stronger performance than traditional methods and exhibits moderate robustness; however, it still trails PERM-QN significantly, particularly once complexity exceeds 25%. This divergence highlights the scalability limitations of PPO’s policy optimization approach when facing highly obstructed exploration tasks. An important insight across all subplots is the widening efficiency gap between PERM-QN and the other methods as map complexity intensifies. While all methods experience some efficiency loss as environmental difficulty escalates, PERM-QN’s decline is far less severe, suggesting that its hybrid reasoning and navigation framework allows it to make more informed exploration decisions and avoid redundant movements. In the largest maps (40 40) at the highest complexity levels, this advantage becomes even more critical, as competing algorithms struggle to sustain meaningful exploration. Overall, Fig. 6 reinforces that PERM-QN is not only the most efficient exploration strategy but also the most resilient under environmental stress, maintaining high search efficiency and effective coverage where other algorithms falter. This robustness positions PERM-QN as a highly promising solution for real-world unknown environment mapping and coverage tasks, where adaptability to complex, unpredictable conditions is essential. Energy awareness of PERM-QN In program design for robotic exploration, energy optimization plays a crucial role in ensuring the robot can reliably return to its initial starting point after completing its task, rather than unpredictably ceasing operation at an arbitrary location within the environment. This focus on energy efficiency directly supports the robot’s ability to conclude its mission effectively and autonomously. Therefore, this part of the experiment mainly compares the program using energy optimization (PERM-QN) with the program without energy optimization (Q-learning method equipped with priority experience replay and memory network), and demonstrates the ability of the program by comparing the coverage rate and the probability of returning to the initial position. The robot’s ability to return to its initial position will be demonstrated by comparing the remaining energy to the shortest distance between the stop position and the starting position. In this experiment, we used a map with 20 x 20 grids, with random obstacles occupying 20% of the total map area. The maximum energy of the robot was preset to be enough for the robot to move 400 grids. Figure 7a investigates the average map coverage under different energy levels between these two methods. From the figure, we can find that without energy optimization, the map coverage rate remains higher across all energy levels, indicating a comprehensive map and a refined search. After adding energy optimization, the map coverage rate shows a certain amount of attenuation and shows some shortcomings in map coverage and performance. However, the difference in map coverage rate between the two methods is basically no more than 3%. Performance comparison with PERM-QN and Q-learning method with priority experience replay and memory network across varying energy levels. a Left figure: Average explored area vs. energy level. b Right figure: Search efficiency vs. energy level. Figure 7b shows the probability of returning to the initial position with the remaining energy of the two methods at different energy levels. From the figure, it can be found that when the robot has a low energy level, both methods could maintain a high rate of finding the way back to the starting point. However, as energy increases, especially when energy is around 80% of the preset maximum energy, the gap between the two methods reaches its maximum, and our PERM-QN method is about 20% better than the method without energy optimization, while the method with energy optimization can ensure stability of return over 90%. A joint analysis of Fig. 7a and b reveals that when the initial energy level is low, the map coverage achieved by both methods is almost equivalent and both robots are able to retain sufficient energy to return to the starting position. This phenomenon can be attributed to the dominance of nearby unexplored regions in the exploration strategy under low-energy conditions, combined with the fact that the limited initial energy prevents extensive exploration of distant areas. However, as the initial energy level increases, robots without energy optimization continue to prioritize distant unexplored regions even when the energy becomes low, which often results in insufficient remaining energy for a safe return. In contrast, robots equipped with energy optimization mechanisms tend to sacrifice some exploration of distant areas when energy levels drop, ensuring a higher probability of successfully returning, even at the cost of slightly reduced coverage. As the initial energy level further increases, providing sufficient energy for a complete exploration of the environment, the return probability of robots without energy optimization gradually recovers. Simultaneously, robots with energy optimization also show a rebound in map coverage, narrowing the performance gap between the two strategies. The impacts of priority experience and memory network within PERM-QN Figure 8 presents a comparative analysis of the average reward achieved by different reinforcement learning methodologies as the complexity of the environment increases. The average reward is defined as the cumulative reward per episode averaged over multiple trials. The trends depicted in the graph clearly illustrate the impact of these enhancements on learning performance. The basic Q-learning method consistently yields the lowest average reward across all levels of map complexity. Furthermore, its performance deteriorates most rapidly as the environment becomes more complex, indicating a significant struggle in navigating and optimizing paths in the presence of increased obstacles and intricate layouts. This suggests that basic Q-learning is less robust and efficient in handling challenging environments. Performance comparison of different exploration methods under varying map complexity, evaluated by average reward. In contrast, the other three methods, which incorporate mechanisms to improve learning efficiency and exploration, demonstrate notably higher average rewards. The method utilizing experience replay shows an improvement over basic Q-learning, indicating that replaying past experiences helps in learning more effectively. Similarly, the integration of a memory network further enhances performance, suggesting that recalling and reusing successful historical trajectories contributes positively to the average reward. Notably, the proposed PERM-QN which combines prioritized experience replay and the memory network, achieves the highest average reward across all complexity levels. Moreover, its performance degrades the least as the map complexity increases, highlighting its superior ability to handle obstacles and optimize paths even in highly complex environments. We also observe that PERM-QN exhibits the lowest standard deviation among all methods, further reflecting its strong robustness and consistency under varying environmental conditions. This stark difference in performance, especially the significant gap between basic Q-learning and the PERM-QN at higher complexity levels, demonstrates the crucial role of prioritized experience replay and memory networks in developing robust and efficient autonomous navigation and exploration strategies. The figure effectively demonstrates that these advanced techniques significantly mitigate the challenges posed by increasing environmental complexity, leading to substantially better learning outcomes. Conclusion This paper investigates the application of reinforcement learning methodologies to address the challenges of autonomous navigation in intricate and uncharted post-disaster scenarios. Building upon the Q-learning algorithmic framework, we introduce enhancements to its reward structure, employing a piecewise reward function to facilitate comprehensive coverage path planning in completely unknown environments. Concurrently, to mitigate the risks of robot disorientation or energy depletion within such uncertain settings, an energy-optimized reward function is integrated. This function dynamically adjusts the influence of the robot’s energy status on the exploration process, ensuring a sufficient capacity for return to the initial position while maintaining efficient environmental exploration. Furthermore, to enhance training efficacy and reduce computational time, we incorporate prioritized experience replay and memory network mechanisms. The experimental results demonstrate that the proposed PERM-QN outperforms the traditional Q-learning algorithm and the AntBot-EX algorithm, achieving superior exploration efficiency while ensuring effective coverage of unknown environments. Despite these advancements, the current algorithm exhibits limitations, notably a protracted computational time for offline training. While measures have been implemented to accelerate algorithm convergence, the training duration remains substantial. Future research efforts will therefore focus on optimizing this aspect to further improve training efficiency. Additionally, we intend to explore the potential of multi-robot parallel exploration to further enhance overall exploration efficiency. Author contributions Y. Xue was responsible for the conceptualization and design of the study, the development of the methodology, and the preparation of the original draft manuscript. C. K. Tan contributed to funding acquisition, provided overall supervision, and was involved in critical review and revision of the manuscript. W. P. Wong provided supervisory support and contributed to the review and editing of the manuscript. All authors reviewed and approved the final version of the manuscript. Funding This work was supported in part by the Ministry of Higher Education (MoHE) Malaysia under Fundamental Research Grant Scheme (FRGS) with grant number FRGS/1/2023/ICT08/MUSM/02/1. Data availability The authors declare that the data supporting the findings of this study are available within the paper and the Zenodo repository: https://doi.org/10.5281/zenodo.15541708 These data are provided to ensure transparency and reproducibility of the key experimental results. The full simulation environment and results are proprietary and remain under active development. The additional technical details may be made available by the corresponding author upon reasonable request. Declarations Competing interests The authors declare no competing interests. Footnotes Publisher’s note Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations. These authors are contributed equally to this work: Chee Keong Tan and Wai Peng Wong. References Associated Data Data Availability Statement The authors declare that the data supporting the findings of this study are available within the paper and the Zenodo repository: https://doi.org/10.5281/zenodo.15541708 These data are provided to ensure transparency and reproducibility of the key experimental results. The full simulation environment and results are proprietary and remain under active development. The additional technical details may be made available by the corresponding author upon reasonable request.