Enhanced twin delayed DDPG with prioritized experience replay and Noisy Nets for regional economic dispatch Authors: Chang Xu, Naoki Hayashi, Masahiro Inuiguchi, Wong Jee Keen Raymond, Hazlie Mokhlis, Hazlee Azil Illias Published: 2025 ABSTRACT AbstractIntegrating renewable energy into power systems introduces significant challenges in balancing generation costs and grid stability, necessitating advanced solutions for the Economic Dispatch Problem (EDP). While classical mathematical and meta-heuristic methods face scalability and computational efficiency limitations, reinforcement learning (RL) offers a promising alternative due to its adaptability to high-dimensional and dynamic environments. This study employs Twin Delayed DDPG (TD3), an enhanced version of Deep Deterministic Policy Gradient (DDPG). TD3 integrates Prioritized Experience Replay (PER) and Noisy Networks (Noisy Nets) for the EDP in a regional microgrid with photovoltaic (PV) generation. PER improves sample efficiency by prioritizing high-error transitions, while Noisy Nets enhance exploration through adaptive parameter noise. Experiments demonstrate that combining these techniques with TD3 achieves a 54.6% reduction in testing operation cost and a 95.3% decrease in cumulative power unbalance compared to the baseline TD3. The improvements are validated across various deterministic and stochastic RL models, with TD3+PER+Noisy Nets outperforming others in cost efficiency and stability. The findings demonstrate the proposed approach’s capability to optimize microgrid dispatch while providing a scalable and practical framework for power system control.Subject terms: Energy science and technology, Engineering, Mathematics and computing Introduction The widespread adoption of sensors has enabled the development of smart grids that offer grid operators detailed insights into power generation and transmission. At the same time, higher penetration of renewable energy on the generation side introduces new challenges. Output from renewable sources is intermittent and variable because it depends on natural conditions. A primary operational objective in smart-grid energy management is to dispatch fuel-based generators efficiently to maximize renewable-energy utilization. Consequently, these issues are commonly framed as the EDP in power systems. Solutions to the EDP can be categorized into three main approaches: classical mathematical methods, meta-heuristic techniques, and AI-based algorithms1,2. Classical mathematical methods are the earliest approaches developed for solving EDPs. These methods are generally applied to simplified problems, where the objective focuses on generation costs, typically modeled by quadratic functions, and the system features relatively few constraints1. A representative classical mathematical method based on quadratic programming, which considers line flow and emission constraints, was proposed in 1998 and tested on a power system with five generation units and ten buses3. However, classical mathematical methods generally perform poorly when applied to large-scale and complex power systems. Significant system simplifications are often required to ensure computational feasibility4, particularly when the system involves many constraints5. As modern power systems continue to grow in complexity, the computational time required by classical mathematical methods increases exponentially, rendering them unsuitable for addressing contemporary large-scale optimization problems. In contrast, meta-heuristic techniques6,7 offer a promising alternative for optimizing modern complex power systems8. These methods provide approximate yet acceptable solutions within reasonable computational time and are applicable across various optimization problems. Among them, population-based meta-heuristics excel at global exploration and at escaping local optima. The trade-off is higher computational cost and additional parameter tuning9. For solving EDPs, trajectory-based meta-heuristics have been explored in some studies10, but population-based meta-heuristics are more widely adopted1. In particular, numerous hybrid algorithms have been proposed to address various dispatch-related challenges. Awadallah et al.11 introduced the Memetic Salp Swarm Algorithm (MSSA), which integrates the global exploration capability of the Salp Swarm Algorithm with an adaptive local search strategy to enhance solution accuracy in EDPs. Shaban et al.12 proposed the Growth Optimizer (GO), a novel metaheuristic inspired by human learning and introspection processes, which demonstrated superior robustness and outperformed various contemporary algorithms across multiple benchmark dispatch systems. Similarly, Fahim et al.13 developed a hybrid Jaya and Teaching–Learning-Based Optimization (TLBO) algorithm that integrates complementary mechanisms from different metaheuristics to handle non-smooth cost functions and complex operational constraints, thereby improving solution reliability. Although meta-heuristic methods outperform classical mathematical approaches, the increasing scale of modern power systems leads to a substantial expansion of the search space, resulting in prolonged computational times required to obtain satisfactory solutions14. This limitation becomes even more pronounced when the dispatch horizon is divided into finer time intervals, as the number of decision variables grows proportionally with the time step resolution. Similarly, larger system scales, with more distributed generators, storage units, and network constraints, expand the search space. In addition, meta-heuristic methods often require extensive parameter tuning and multiple iterations to prevent premature convergence, which increases the computational burden and limits their practicality for real-time or near-real-time applications. With advancements in deep learning, RL-based approaches15,16 have been introduced into power system control17,18. Compared to meta-heuristic methods, RL-based approaches are more capable of handling high-dimensional data and are well-suited for online execution due to their rapid decision-making ability based on the current state19. Chiş et al. 20 developed an RL-based strategy for electric vehicle (EV) charging. Linear programming (LP) was first employed to determine the optimal daily charging actions under various scenarios based on historical datasets, and the Fitted Q-Iteration algorithm21 was subsequently applied to learn the optimal policy. This approach effectively reduced users’ long-term EV charging costs while simplifying daily charging decisions by eliminating the need to solve complex linear programming problems each day. Hou et al. 22 proposed a hybrid approach combining mixed-integer linear programming (MIP) and deep Q-learning (DQN). After training the DQN model, online execution was enhanced using commercial MIP solvers to ensure that all operational constraints were strictly enforced, which standard DQN policies alone could not guarantee. Additionally, Claessens et al. 23 integrated a convolutional neural network (CNN) into the Fitted Q-Iteration framework to approximate the Q-function, thereby extracting state-time features from thermostatically controlled loads and reducing electricity costs while achieving near-optimal performance in simulations. A summary of the above-reviewed EDP research works is presented in Table 1. Table 1Summary of representative methods for solving economic dispatch problems (EDPs).CategoryModelKey featuresLimitationsReferenceClassical mathematical methodsQuadratic programmingSuitable for small-scale systems; handles generation cost with polynomial approximationPoor scalability; not suitable for complex modern systems3Meta- heuristic techniquesMemetic salp swarm algorithm (MSSA)Combines global exploration of SSA with adaptive local search; improves convergence and solution accuracyAdditional computational effort due to hybridization; requires parameter tuning11Growth optimizer (GO)Inspired by human learning and introspection; robust performance across multi-scale EDP benchmarksAlgorithm still new; requires further validation in diverse scenarios12Jaya and teaching–learning-based optimization (TLBO)Integrates complementary mechanisms from different metaheuristics; effective for non-smooth cost functions and operational constraintsComplexity of coordination may increase implementation difficulty13AI-based / RL methodsFitted Q-Iteration + LPOffline training via LP; online decision-making for EV chargingRequires historical data; less generalizable to unseen scenarios20DQN + MIPCombines DQN policy learning with strict operational constraint enforcement via MIPOnline execution depends on commercial solvers; more computationally intensive22CNN + Fitted Q-IterationLearns state-time features for control of thermostatically controlled loadsTraining requires large data; tailored to specific applications23 Summary of representative methods for solving economic dispatch problems (EDPs). Beyond the scope of economic dispatch, reinforcement learning has also been combined with graph-based and federated frameworks to address challenges in distribution systems. For instance, an unrolled spatiotemporal graph convolutional network (USGCN) was applied to state estimation and forecasting by capturing renewable-related spatiotemporal correlations 24. A full-model-free adaptive graph deep deterministic policy gradient (FAG-DDPG) approach was proposed for multi-terminal soft open point voltage control, leveraging graph attention to enhance feature extraction and policy learning 25. Furthermore, a prototype federated reinforcement learning framework with a physics-aware spatiotemporal transformer (STT-PFRL) was introduced for privacy-preserving and robust voltage regulation under renewable uncertainties 26. These studies demonstrate the broader applicability of reinforcement learning and graph-based techniques beyond dispatch problems. In RL-based power system control, many studies have focused on integrating RL with other algorithms to improve performance rather than enhancing the RL algorithm structure itself. However, RL algorithms can be structurally modified to improve performance in control applications. To incorporate RL improvement techniques into power system control, this study proposes a TD3 controller enhanced with PER and Noisy Nets to address the EDP problem in regional power grids. PER enhances the sample efficiency of the standard TD3 model by prioritizing transitions that contribute more significantly to learning, thereby accelerating convergence and improving overall performance. In parallel, Noisy Nets introduce adaptive exploration by injecting parameterized noise into the network, enabling the agent to explore the action space more effectively and avoid premature convergence to suboptimal policies. The primary contributions of this study are as follows: A hybrid reinforcement learning controller is developed by integrating PER and Noisy Nets within TD3. This structural enhancement improves sample efficiency and adaptive exploration without relying on external hybridization frameworks or optimization solvers, simplifying deployment. A hybrid reinforcement learning controller is developed by integrating PER and Noisy Nets within TD3. This structural enhancement improves sample efficiency and adaptive exploration without relying on external hybridization frameworks or optimization solvers, simplifying deployment. The enhancement modules (PER and Noisy Nets) are modular and transferable across different reinforcement-learning controllers, including DDPG, SAC, and MPO, demonstrating generalizability across deterministic and stochastic policy models in power system control. The enhancement modules (PER and Noisy Nets) are modular and transferable across different reinforcement-learning controllers, including DDPG, SAC, and MPO, demonstrating generalizability across deterministic and stochastic policy models in power system control. In a regional power grid environment with realistic pricing, renewable generation, and battery storage, the proposed controller outperforms multiple baselines in operational cost minimization and power balance, indicating suitability for near real-time dispatch. In a regional power grid environment with realistic pricing, renewable generation, and battery storage, the proposed controller outperforms multiple baselines in operational cost minimization and power balance, indicating suitability for near real-time dispatch. The remainder of this paper is structured as follows: Section II introduces the background of improvement techniques and control models, including DDPG, SAC, and MPO. Section III presents the structure of the proposed PER+Noisy Nets TD3 controller. Section IV describes the modeling of the regional grid used as an environment for the RL controller and details the proposed model’s hyperparameters. Section V discusses numerical results and comparative analysis. Finally, Section VI provides the conclusions. RL improvement techniques and controlling models This section introduces several well-known RL improvement techniques and widely adopted RL-based control models. Some of these methods are also selected for comparison with the proposed approach in subsequent sections. With the advancement of deep learning, RL has gained significant attention, with documented cases of exceeding human baselines on specific tasks. Unlike supervised learning, which requires manually labeled datasets and is limited to performing at most as well as the training data, RL enables models to achieve superhuman performance27. Hessel et al.28 tested six different DQN variants on 57 Atari 2600 games, with three of them (DQN+PER, Dueling DQN, and Distributional DQN) achieving higher scores than human experts. Beyond classic video games, RL has also demonstrated superior performance in more complex scenarios. Vinyals et al.29 employed a multi-agent RL algorithm with an actor-critic structure to play StarCraft II, achieving a ranking above 99.8% of officially ranked human players after reaching the highest Grandmaster level. These results underscore the strong control performance and stability of RL algorithms. RL improvement techniques To enhance RL performance, numerous improvement techniques have been proposed. Some of the most notable include Double Q-learning, PER, Dueling Network Architecture, Multi-step Learning, Distributional RL, and Noisy Nets. Compared to the standard DQN architecture, these techniques improved performance in Atari benchmarks. When these enhancements were integrated into a single agent, Rainbow28, it achieved the highest scores, exceeding human performance by more than a factor of two. This demonstrates that combining multiple improvements can substantially boost model performance. The problems addressed in these game-related studies are highly complex. The RL agent must process state representations composed of pixel-based images, which are influenced by game mechanics and randomness. Furthermore, the action space in such environments is often high-dimensional, requiring the agent to select from a vast range of possible actions30. In contrast, RL-based controllers for power systems operate in a significantly less complex environment. Given this difference in complexity, applying these RL improvement techniques to power system control problems should be feasible and often sufficient for power-system control problems. Mathematical formulation of the economic dispatch problem In the environment adopted in this study, generator fuel costs are represented as convex quadratic functions, while grid transactions are modeled linearly with sales valued at a discounted coefficient. Penalties on power unbalance beyond the grid exchange capacity are modeled linearly, and basic operational constraints such as generator ramping and battery state-of-charge updates are also included. However, several features reflecting physical operating limits make the formulation of the EDP problem non-convex. Distributed generators can either be off or operate only within a bounded range, the battery dynamics involve clipping at the minimum and maximum SOC, and the grid exchange is asymmetric due to capped transactions, different buy and sell prices, and penalties for excess or shortage. These characteristics introduce discontinuities and non-smooth dynamics. As a result, the environment represents the EDP as a non-convex optimization problem with quadratic costs and piecewise-linear constraints, rather than as a simple convex quadratic program. This complexity motivates the use of reinforcement learning, which can learn effective dispatch policies offline and then apply them in real time without repeatedly solving a difficult optimization problem. RL-based controlling models DQNs31 introduced neural networks to approximate Q-values in reinforcement learning, enabling effective control in discrete action spaces. However, DQNs face limitations in handling continuous action problems. To address this, the actor-critic framework32 has become a foundational architecture in RL. In this framework, the actor learns a policy to select actions, while the critic estimates the value function Q(s, a) to evaluate and guide the actor’s updates. Traditional actor-critic methods adopt on-policy learning with stochastic policies, where the actor outputs a probability distribution over actions. Based on the type of policy used, actor-critic models are typically categorized into deterministic and stochastic approaches. Deterministic policy models Deep Deterministic Policy Gradient (DDPG)33 is a widely used RL algorithm designed for continuous action spaces. As its name suggests, DDPG employs a deterministic policy, meaning the actor directly outputs a specific action without sampling. To mitigate training instability in actor-critic architectures, DDPG introduces target networks to stabilize the critic updates. Additionally, it adopts an experience replay buffer to enable off-policy learning, improving data efficiency and reducing training costs. However, DDPG suffers from an overestimation bias in Q-values. Since the critic updates by maximizing the Q-value, network approximation errors or noise can lead to overestimated Q-values. To address this issue, TD3 was proposed as an improvement over DDPG. Stochastic policy models SAC34 is a stochastic policy algorithm based on the maximum entropy framework. Unlike conventional RL algorithms that solely aim to maximize the expected cumulative reward, SAC seeks to optimize both reward maximization and policy entropy simultaneously. The standard RL objective is to learn a policy that maximizes the expected cumulative reward: In contrast, SAC optimizes for an augmented objective function that includes an entropy regularization term: where the entropy term is defined as In the above formulations, represents the policy and denotes the optimal policy. The expectation operator ensures the optimization focuses on the long-term average reward rather than any individual high-reward instance. is the discount factor, which balances short-term and long-term rewards. represents the reward function. The entropy term measures the action variety of the policy, with a larger entropy indicating greater exploration. The temperature coefficient is a tunable hyperparameter that controls the importance of the entropy term in the objective function. A larger results in more stochastic policies, whereas a smaller leads to more deterministic behavior. During training, is automatically adjusted to balance exploration and exploitation. Compared to TD3, SAC differs primarily in its actor structure. While TD3 employs a deterministic policy network that directly outputs actions, SAC utilizes a stochastic policy network that outputs an action distribution, from which actions are sampled using the reparameterization trick. Despite this difference, SAC and TD3 share a common feature in their critic design. Similar to TD3, SAC employs two independent critic networks to mitigate the problem of Q-value overestimation. By incorporating the entropy term into the objective function, SAC enhances exploration without relying on external noise to introduce action variability. This leads to a more diverse set of actions, thereby improving overall performance. Additionally, this modification enhances the model’s robustness, as the policy is less likely to become overly dependent on specific actions. MPO35 is another stochastic policy algorithm inspired by Expectation-Maximization (EM) techniques and probabilistic inference. Its key feature is constraining policy updates to remain close to the previous policy, ensuring training stability. MPO decomposes policy optimization into two alternating steps: E-Step: Given state s, sample N actions using the current policy, then compute Q-values using the critic. The probabilities of these actions are then adjusted using a SoftMax transformation: 4 where the temperature parameter controls the update intensity. A higher produces smoother updates, while a lower leads to more aggressive updates. E-Step: Given state s, sample N actions using the current policy, then compute Q-values using the critic. The probabilities of these actions are then adjusted using a SoftMax transformation: 4 where the temperature parameter controls the update intensity. A higher produces smoother updates, while a lower leads to more aggressive updates. M-Step: The policy is updated via weighted maximum likelihood to favor high-Q actions. M-Step: The policy is updated via weighted maximum likelihood to favor high-Q actions. By constraining updates through temperature regulation, MPO ensures stable training. However, while this temperature-based regulation offers improved stability, it also introduces trade-offs. The constraint on policy updates results in increased computational complexity. This may limit the practical applicability of MPO in environments with high-dimensional action spaces. Moreover, the reduced exploration due to temperature constraints makes the algorithm more susceptible to getting stuck at a local minimum. In such scenarios, the policy might converge prematurely to suboptimal solutions. This limits the model’s ability to explore a diverse set of actions and learn a more globally optimal policy. For EDP in power systems, the complexity is significantly lower than in high-dimensional RL applications. Moreover, real-world constraints limit the available data for training. Using excessively complex control models may not necessarily yield better performance. It could lead to unstable or inefficient training. Considering the complexity of the environment, this study proposes an improved TD3 controller for optimizing the operation of a regional power grid. The previously discussed DDPG, SAC, and MPO algorithms are employed as benchmarks to evaluate and compare the performance of the proposed improved TD3 model. Research methodology In this study, the combined improvement techniques are introduced into the power system control model to enhance its performance when addressing the EDP. Two improvement techniques are selected for integration, namely PER and Noisy Nets. The overall framework of the proposed TD3+PER+Noisy Nets method is illustrated in Fig. 1. The following subsections provide detailed explanations of each component and the training process. The structure of the proposed TD3+PER+Noisy Nets controller. TD3 TD336 is an enhancement over DDPG and forms the core of the controller design in this study. Compared to DDPG, TD3 introduces several key improvements: Dual Critic Networks TD3 maintains two critic networks simultaneously and computes two Q-values for each state-action pair. During the update process, the smaller Q-value is selected to mitigate the overestimation bias that is commonly observed in single-critic architectures. This conservative approach helps ensure that the estimated Q-values are not overly optimistic. Dual Critic Networks TD3 maintains two critic networks simultaneously and computes two Q-values for each state-action pair. During the update process, the smaller Q-value is selected to mitigate the overestimation bias that is commonly observed in single-critic architectures. This conservative approach helps ensure that the estimated Q-values are not overly optimistic. Delayed Policy Updates The actor network is updated less frequently than the critic networks. This strategy guarantees that the critic networks can converge to more accurate Q-value estimates before the actor is updated, thereby promoting training stability. Delayed Policy Updates The actor network is updated less frequently than the critic networks. This strategy guarantees that the critic networks can converge to more accurate Q-value estimates before the actor is updated, thereby promoting training stability. Target Policy Smoothing To further stabilize the target Q-value estimation, target policy smoothing is employed. When calculating the target Q-value, noise is added to the action generated by the target policy network. Specifically, the perturbed action is computed as follows: 5 where denotes the action produced by the target policy network for the next state , and n is noise sampled from a Gaussian distribution with mean 0 and standard deviation , which is then clipped to the interval . Importantly, this noise is not introduced for the purpose of exploration; rather, it encourages the critic to consider Q-values over a local neighborhood of actions, thereby promoting a smoother Q function that is less prone to overestimation. Target Policy Smoothing To further stabilize the target Q-value estimation, target policy smoothing is employed. When calculating the target Q-value, noise is added to the action generated by the target policy network. Specifically, the perturbed action is computed as follows: 5 where denotes the action produced by the target policy network for the next state , and n is noise sampled from a Gaussian distribution with mean 0 and standard deviation , which is then clipped to the interval . Importantly, this noise is not introduced for the purpose of exploration; rather, it encourages the critic to consider Q-values over a local neighborhood of actions, thereby promoting a smoother Q function that is less prone to overestimation. The selection of TD3 as the primary controller in this study is motivated by its demonstrated stability and robustness in continuous control tasks. The additional structural improvements in TD3 (especially in stabilizing the critic network) are critical when combining the controller with other techniques. In particular, the subsequent integration of PER and Noisy Nets necessitates a robust baseline to ensure that a reliable policy can be obtained. PER Prioritized Experience Replay (PER)37 is an extension of the conventional Experience Replay mechanism. In traditional Experience Replay, a replay buffer is used to store transitions, i.e., the state, action, reward, and next state obtained from interactions with the environment. This allows the RL algorithm to perform off-policy updates by sampling experiences from the buffer, thereby increasing sample efficiency by reusing past experiences multiple times without requiring additional environment interactions. Typically, experiences in the buffer are sampled uniformly with the probability: where N is the capacity of the replay buffer. In contrast, PER assigns a priority to each transition based on its temporal-difference (TD) error. The sampling probability for a given transition i is defined as: where , denotes the TD error of transition i, is a small positive constant that ensures every transition has a non-zero probability of being sampled (even when ), and a is a hyperparameter that determines the degree of prioritization. When , the sampling becomes uniform, larger values of a bias the sampling process towards transitions with larger TD errors. Since non-uniform sampling introduces bias (i.e., transitions with high TD errors are sampled more frequently, potentially reducing sample diversity), importance sampling weights are incorporated into the loss function to counteract this bias. The importance sampling weight for a transition i is given by: where is another hyperparameter that adjusts the extent of the correction. The weighted loss function is then expressed as: with m being the minibatch size, the target value for transition i, and the current Q-value estimate. PER effectively accelerates learning by focusing on transitions that are more informative for the learning process. This process is particularly advantageous in settings with high interaction costs. A typical example is a power-system environment, where operating costs and power-unbalance metrics must be computed repeatedly. Furthermore, PER is generalizable to any off-policy algorithm utilizing a replay buffer, and its code implementation is relatively simple. Noisy Nets Enhancing exploration in RL is essential for improving the quality of the solutions obtained. A common strategy is to inject external noise into the policy, for instance, by adding Gaussian noise directly to the deterministic action output: where a means the output action and n denotes the added noise, and is the original action determined by the policy based on state s. While this method is straightforward and can facilitate exploration, the fixed noise intensity may be insufficient in complex environments. To overcome this limitation, Noisy Nets38 are introduced as a more adaptive approach. As can be seen from Fig. 2, instead of adding noise directly to the action output, Noisy Nets incorporate noise into the parameters (weights and biases) of the neural network itself. This approach allows the magnitude of the noise to be learnable, thereby endowing the network with a more powerful and context-sensitive exploration capability. Consider a standard linear layer represented as: where w and b denote the weight matrix and bias vector, respectively, and x and y represent the input and output of the layer. When Noisy Nets are employed, the layer is modified as follows: In this expression, and are the deterministic parameters, while and represent the learnable noise scaling factors. The variables and are random noise samples drawn from a predefined distribution (typically a Gaussian) and denotes element-wise multiplication. By integrating noise directly into the network parameters, Noisy Nets achieve adaptive exploration that is directly tied to the agent’s interactions with the environment. In the proposed method, only the actor network is replaced with a Noisy Net architecture, as applying Noisy Nets to the critic networks was found to introduce training instability. The comparison between Noisy Nets and conventional noise injection. Training process Training Framework of this study This study evaluates four models: TD3, DDPG, SAC, and MPO. All models are based on an actor–critic architecture, and the fundamental training framework is maintained consistently across experiments, as outlined in Algorithm 1. The training process comprises several stages: Initial Data Collection Initially, a stochastic policy augmented with extra exploration noise is employed to interact with the environment. A total of 10,000 steps are collected and stored in the replay buffer. Initial Data Collection Initially, a stochastic policy augmented with extra exploration noise is employed to interact with the environment. A total of 10,000 steps are collected and stored in the replay buffer. Off-Policy Training Following the initial data collection, off-policy training is conducted over 450 episodes. At each training iteration, a minibatch of 256 transitions is sampled from the PER buffer using priority sampling. The sampling probability for each transition is determined by its TD error, so that transitions with larger errors are more likely to be selected. For each sampled transition, the following update steps are performed: Critic Update The target actor network generates a new action for the next state , incorporating additional policy noise for robustness. Subsequently, two target critic networks compute the Q-values for the generated action. The smaller of the two Q-values is selected to reduce the overestimation bias and is used to calculate the TD target (also known as target Q-value). The current critic networks then compute their respective Q-values for the sampled transitions, and the TD errors are computed accordingly. These TD errors inform the update of the critics’ parameters via the loss function. Additionally, the computed TD errors are used to update the priorities of the corresponding transitions in the PER buffer.Actor Update The actor network, implemented as a Noisy Net, produces an action for the current state . A single critic network then evaluates the Q-value corresponding to this action. The actor’s parameters are updated in the direction that maximizes the Q-value, following a policy gradient method.Target Network Update After a fixed number of iterations, soft updates are applied to synchronize the target networks with their corresponding online networks. This step is critical for maintaining the stability of the training process. Off-Policy Training Following the initial data collection, off-policy training is conducted over 450 episodes. At each training iteration, a minibatch of 256 transitions is sampled from the PER buffer using priority sampling. The sampling probability for each transition is determined by its TD error, so that transitions with larger errors are more likely to be selected. For each sampled transition, the following update steps are performed: Critic Update The target actor network generates a new action for the next state , incorporating additional policy noise for robustness. Subsequently, two target critic networks compute the Q-values for the generated action. The smaller of the two Q-values is selected to reduce the overestimation bias and is used to calculate the TD target (also known as target Q-value). The current critic networks then compute their respective Q-values for the sampled transitions, and the TD errors are computed accordingly. These TD errors inform the update of the critics’ parameters via the loss function. Additionally, the computed TD errors are used to update the priorities of the corresponding transitions in the PER buffer.Actor Update The actor network, implemented as a Noisy Net, produces an action for the current state . A single critic network then evaluates the Q-value corresponding to this action. The actor’s parameters are updated in the direction that maximizes the Q-value, following a policy gradient method.Target Network Update After a fixed number of iterations, soft updates are applied to synchronize the target networks with their corresponding online networks. This step is critical for maintaining the stability of the training process. Critic Update The target actor network generates a new action for the next state , incorporating additional policy noise for robustness. Subsequently, two target critic networks compute the Q-values for the generated action. The smaller of the two Q-values is selected to reduce the overestimation bias and is used to calculate the TD target (also known as target Q-value). The current critic networks then compute their respective Q-values for the sampled transitions, and the TD errors are computed accordingly. These TD errors inform the update of the critics’ parameters via the loss function. Additionally, the computed TD errors are used to update the priorities of the corresponding transitions in the PER buffer. Critic Update The target actor network generates a new action for the next state , incorporating additional policy noise for robustness. Subsequently, two target critic networks compute the Q-values for the generated action. The smaller of the two Q-values is selected to reduce the overestimation bias and is used to calculate the TD target (also known as target Q-value). The current critic networks then compute their respective Q-values for the sampled transitions, and the TD errors are computed accordingly. These TD errors inform the update of the critics’ parameters via the loss function. Additionally, the computed TD errors are used to update the priorities of the corresponding transitions in the PER buffer. Actor Update The actor network, implemented as a Noisy Net, produces an action for the current state . A single critic network then evaluates the Q-value corresponding to this action. The actor’s parameters are updated in the direction that maximizes the Q-value, following a policy gradient method. Actor Update The actor network, implemented as a Noisy Net, produces an action for the current state . A single critic network then evaluates the Q-value corresponding to this action. The actor’s parameters are updated in the direction that maximizes the Q-value, following a policy gradient method. Target Network Update After a fixed number of iterations, soft updates are applied to synchronize the target networks with their corresponding online networks. This step is critical for maintaining the stability of the training process. Target Network Update After a fixed number of iterations, soft updates are applied to synchronize the target networks with their corresponding online networks. This step is critical for maintaining the stability of the training process. Continuous Data Collection and Model Saving As training progresses and the performance of the actor improves, new samples are continuously collected from interactions with the environment and added to the replay buffer. This process ensures that the buffer always contains up-to-date experiences, which is crucial for the continual refinement of the model. During each episode, the critic loss, actor loss, and various environmental evaluation metrics are recorded. The parameters of the actor and critic networks are saved when the operational cost reaches its optimal performance, thereby designating the best model for subsequent testing. Continuous Data Collection and Model Saving As training progresses and the performance of the actor improves, new samples are continuously collected from interactions with the environment and added to the replay buffer. This process ensures that the buffer always contains up-to-date experiences, which is crucial for the continual refinement of the model. During each episode, the critic loss, actor loss, and various environmental evaluation metrics are recorded. The parameters of the actor and critic networks are saved when the operational cost reaches its optimal performance, thereby designating the best model for subsequent testing. Post-Training Evaluation After the completion of training, the best model is evaluated in an independent testing environment using unseen data. Key performance metrics such as overall operational cost and power unbalance are calculated to validate the model’s effectiveness in practical applications. Post-Training Evaluation After the completion of training, the best model is evaluated in an independent testing environment using unseen data. Key performance metrics such as overall operational cost and power unbalance are calculated to validate the model’s effectiveness in practical applications. In summary, each component of the proposed method contributes a distinct advantage: TD3 ensures the stability of the critic during training; PER enhances sample efficiency and accelerates convergence; and Noisy Nets provide an adaptive mechanism for improved exploration. The synergistic integration of these techniques results in a high-performance controller that is robust and reliable for continuous control tasks in power system operations. Experiment settings This section details the modeling of the regional grid used as the environment in this study and outlines the hyperparameters for the TD3 controller and the integrated improvement techniques. The experimental setup is designed to rigorously evaluate the performance of the RL controllers under realistic operational conditions. Modeling of regional grid The environment employed in this study originates from a publicly available framework introduced by Shengren et al.22. The original study formulated the problem using a MIP solver to guarantee strict satisfaction of operational constraints. Although the initial purpose of that framework was constraint-oriented optimization, it also provides a well-structured setting for evaluating RL controllers under realistic economic dispatch conditions. Accordingly, the environment was employed in this study with an extended training period of 11 months and a testing period of 1 month. As can be seen from Fig. 3, the environment comprises two main components: a regional grid and a main grid. The regional grid consists of a solar farm, three distributed generators with distinct generation parameters, a battery energy storage system (BESS), and various loads. The BESS is modeled by considering its energy capacity, state-of-charge (SOC) limits, maximum charging and discharging rates, round-trip efficiency, and a degradation factor. At each time step, the SOC is updated based on the charging or discharging action while constrained within the allowable range. This simplified formulation captures the essential operational characteristics of the BESS while remaining computationally efficient for integration into the reinforcement learning framework. For the solar farm and loads, simulation data and collected real-world data are used rather than detailed numerical models. In contrast, the generation cost of the fuel-based generators and the charging/discharging process of the BESS are represented using mathematical models. The main grid functions as a backup supplier. When the regional grid’s generation is insufficient, electricity is purchased from the main grid; conversely, if the regional grid produces excess power, it is sold back to the main grid to maintain power balance. Real-time electricity prices are defined based on historical data. In addition to operational costs, the analysis accounts for power unbalance. The transmission line linking the main grid and the regional grid has a specified power limit. When a generation shortfall requires imports beyond this limit, an unbalance penalty is incurred. This penalty is added to the total operational cost to reflect the higher expense associated with disrupted power balance. The components of the environment used in the experiments. The overall objective in this environment is to minimize the total operational cost of the regional grid. Achieving this objective requires an optimal strategy for utilizing the BESS to counteract the intermittent nature of solar power output and for efficiently scheduling power generation among the three fuel generators. The reward function for the environment is defined as follows: where denotes the operational cost of the BESS (set to 0 in this experiment), represents the generation cost of fuel-based generator at time with cost coefficients , , and , and accounts for the revenue from electricity transactions at time (with as the real-time electricity price and set to 0.5 to reflect the price differential between buying and selling electricity). is the power transmitted between the main grid and the regional grid. is the penalty cost associated with power unbalance, with set to 20 (substantially higher than the standard electricity price to emphasize the importance of maintaining balance). For the BESS, the parameters include the total capacity C, SOC limits and , maximum charging and discharging power , and charging and discharging efficiencies . The divisor of 2000 serves as a reward-scaling constant that limits the numerical magnitude of the reward signal. This value approximates the upper bound of the total daily operating cost in the studied environment and is used solely to stabilize optimization. Since the reward is defined as the negative total cost, the scaled reward becomes a negative quantity, typically around . The intent of this scaling is not to normalize rewards to the range [0, 1], but rather to prevent excessively large gradients and to facilitate stable and comparable learning dynamics across experiments. For further details regarding the operational constraints of the target environment, please refer to the original work 22. The test environment used in the experiments has the same structure and parameter settings as the training environment. The key difference lies in the dataset used: the original study22 utilizes a one-year dataset containing load demand, electricity price, and PV generation data. In the training environment, only the first 11 months of this dataset are used for model training, while the test environment is evaluated on the data from the final month. This separation ensures that the model is assessed on unseen data, providing a robust evaluation of its generalization ability and real-world applicability. The dispatch problem considered in this study corresponds to a day-ahead scheduling framework. Each episode covers a 24-hour horizon with an hourly resolution, i.e., the time interval between decision steps is set to 1 hour. Hyperparameters selection The hyperparameters for the RL training process and the PER mechanism are summarized in Table 2a. These settings are critical to ensuring the rapid convergence and stability of the models under investigation. Table 2Hyperparameters for the training process and PER.ParameterValueDescription(a) Hyperparameters for the training processEpisode number300, 450The total number of training episodes was set to 450 for TD3 and 300 for the remaining models.Learning rate1e-4Learning rate used by the optimizer.OptimizerAdamThe Adam optimizer is selected for its robustness, eliminating the need for a more complex alternative.Mini batch size256Number of samples drawn from the replay buffer per update.Layer number4–5Depth of the RL network.Neuron number per hidden layer64Chosen based on a trade-off between computational efficiency and model performance.Discount factor ()0.995Balances immediate and future rewards.Soft update factor ()1e-2Controls the smoothness of target network updates to prevent abrupt parameter changes and stabilize the policy network’s output.(b) Hyperparameters of PERMax capacity50,000Maximum size of the experience replay buffer; a larger buffer ensures that high-priority samples do not dominate the updates.Alpha0.5Prioritization exponent () that controls the influence of the TD error on the sampling probability.Beta start value0.4Initial value of the importance sampling weight (), which corrects for the bias introduced by prioritized sampling.Beta frames2e6The number of frames over which is linearly increased from the start value to 1.Target step1000Number of samples collected before updating the replay buffer. Hyperparameters for the training process and PER. This study evaluates multiple models, including the baseline TD3, TD3 augmented with Noisy Nets, TD3 integrated with PER, and TD3 combining both PER and Noisy Nets. Additionally, the study considers DDPG, SAC, and MPO models enhanced with both PER and Noisy Nets. All models share similar architectural frameworks and hyperparameter configurations, with key parameters such as Target Step, Max Capacity, Discount Factor, and Soft Update Factor kept consistent with those reported in the original work22. The baseline model, a plain TD3 without any enhancement techniques, serves as the benchmark for performance comparison. The episode numbers of 300 and 450 were selected based on preliminary experiments, which demonstrated that the models converged within this range even without the use of PER. Furthermore, incorporating PER further accelerates convergence. The choice of the number of neurons per hidden layer and the number of layers was constrained by the computational overhead introduced by Noisy Nets. Since each neuron in a Noisy Net requires individual noise generation, the network size was limited to maintain reasonable training times while preserving adequate model capacity for the environment. The learning rate, batch size, and optimizer were selected following standard practices and demonstrated sufficient effectiveness in experiments, negating the need for more complex alternatives. The PER hyperparameters used in this experiment are summarized in Table 2b. These parameters are configured to maintain a balance between prioritizing high-error transitions and ensuring sufficient sample diversity. The values for Alpha and the Beta start value (0.5 and 0.4, respectively) are adopted from the recommended settings in the Rainbow network28, which have been demonstrated to yield robust outcomes. During training, is increased gradually from 0.4 to 1 over 2 million frames (as controlled by the Beta Frames parameter), thereby ensuring that the importance sampling corrections become increasingly effective and contribute to stabilizing the learning process. Preliminary tests explored alternative combinations of Alpha and initial Beta value, but none outperformed the default configuration; therefore, the recommended settings were retained. For Noisy Nets, the sole hyperparameter is the noise parameter , which governs the magnitude of the noise added to the network parameters. In this study, the initial standard deviation for the noise in the Noisy Linear layers is set to 0.1. This relatively small noise magnitude was selected based on experimental observations indicating that larger noise levels can adversely affect performance. Specifically, the environment used in this study is relatively simple. It contains limited inherent randomness and involves fine-grained control of the BESS’s charging and discharging states. In such an environment, large exploration noise can be detrimental because it may disrupt the precise control actions required for optimal performance. For the DDPG model employed in this study, Gaussian exploration noise with a standard deviation of 0.5 is used. To ensure sustained exploration during training, an exploration rate parameter is implemented; initially, the exploration rate is set to 1 (ensuring that noise is always added to the action), and it gradually decreases to 0.3 by the end of training, meaning that 30% of the actions are perturbed by noise. To maintain fairness in comparison, the TD3 model also incorporates the same Gaussian noise mechanism. In contrast to DDPG, TD3 further integrates policy noise (set to 0.2) and adopts a delayed policy update schedule (with the actor updated after every two critic updates) to enhance training stability. For the SAC model, the target entropy is the most critical hyperparameter. While a common default is (where is the action dimensionality), this study sets the target entropy to . Preliminary experiments revealed that setting a smaller target entropy (e.g., or ) promotes excessive exploration, which is counterproductive in power system control tasks that require precise adjustments. The temperature parameter for SAC is initially set to 1 and is updated automatically during training. Finally, for MPO, the hyperparameters include the temperature parameter () and the number of action samples. To conserve computational resources, the number of action samples is set to 16, which is acceptable given the relative simplicity of the environment. The temperature parameter is set to 0.2, representing an aggressive configuration intended to encourage the reuse of high Q-value samples to learn superior policies. However, experimental results indicate that MPO underperforms relative to algorithms with stronger exploration mechanisms. This outcome likely reflects MPO’s emphasis on stable updates in high-dimensional settings rather than on extensive exploration. Overall, the hyperparameters in this study were selected to facilitate rapid and stable convergence to near-optimal solutions. Given the complexity of the models and the integration of multiple improvement techniques, the hyperparameter space is vast, and an exhaustive grid search was impractical under the available computational resources. The experiments were implemented in Python and conducted on a system equipped with an Intel i9-10900K processor, 128 GB of RAM, and an NVIDIA RTX 3070 GPU. Numerical analysis This section presents a comprehensive numerical analysis of the performance of various RL controllers. Both the baseline forms and enhanced versions are analyzed, where the enhanced versions refer to RL controllers integrated with improvement techniques. The results demonstrate the efficacy of integrating RL improvement techniques into the EDP in power systems and their broad applicability to different off-policy training models. Fairness and reproducibility were ensured by employing a fixed set of 15 random seeds per model, which mitigated training stochasticity. Comparison of TD3-based controllers The performance of the baseline TD3 model is compared with its variants incorporating Noisy Nets, PER, or both. Table 3 summarizes the key performance metrics, including training operation cost, training unbalance, testing operation cost, testing unbalance, and the number of episodes required for convergence. The reduction of actor loss during the training process is illustrated in Fig. 4. A lower actor loss indicates that the policy network generates actions that align more closely with the critic’s value estimation, thus reducing the discrepancy between predicted and optimal actions. Consequently, the reduction of actor loss reflects the progressive stabilization of the policy and serves as evidence of model convergence. Table 3Result comparison between TD3 controllers and its variants.ModelTraining operation cost ($)Training unbalance (kW)Testing operation cost ($)Testing unbalance (kW)Convergence needed episodesTD3+Both993.6909559.1635.49TD3+PER1075.201.8613040.55151.64TD3+Noisy1227.61017553.63164.69TD31660.57275.0521074.13755.32 Result comparison between TD3 controllers and its variants. Actor loss comparison of TD3 variants. It should be noted that the Training Operation Cost and Training Unbalance are measured at the end of a series of training episodes, while the Testing Operation Cost and Testing Unbalance represent cumulative values over the entire test period. In the original study, the total dataset spans one year22. In this study, the training phase utilizes data from the first 11 months, while the testing phase is conducted on data from the final month. Evaluating cumulative operational costs over the full 11-month training period would be computationally prohibitive; therefore, such assessments are deferred to the testing phase to ensure computational efficiency while maintaining the integrity of performance evaluation. The reported “Convergence Needed Episodes” is provided as an approximate range. This is due to occasional instability in the critic’s training process, which can transiently affect the actor’s performance, sometimes resulting in a sudden increase in the actor’s loss. In some cases, particularly when PER is employed, convergence can occur as early as 80 episodes; in other instances, the behavior is comparable to that of the plain TD3 model. Overall, however, the incorporation of PER consistently accelerates convergence. A detailed comparison of the data in Table 3 reveals that both improvement techniques, when applied individually, yield significant enhancements in performance relative to the baseline TD3 model. Specifically, both training and testing operation costs and unbalance are substantially reduced. Notably, PER is particularly effective in accelerating convergence and improving performance, an outcome that aligns with findings reported in the Rainbow literature28. When PER and Noisy Nets are combined in the TD3 framework, the improvements are most pronounced: the training operation cost is reduced by 40.2% and the testing operation cost by 54.6% relative to the baseline. Moreover, the training set achieves a zero unbalance, and the cumulative testing unbalance is reduced by 95.3%. In this configuration, Noisy Nets enhance the agent’s exploration capabilities, while PER increases sample utilization through prioritized sampling of high temporal-difference error transitions. Their combined effect fosters an effective balance between exploration and exploitation, enabling the model to converge more rapidly and to identify near-global optima in solving the complex EDP. As shown in Fig. 4, after experiencing an initial period of varying degrees of fluctuation, all TD3 variants eventually converge during training. Notably, the TD3+PER+Noisy Nets variant, which achieves the best overall performance, also exhibits the highest actor loss among all models. While this may seem counterintuitive at first glance, it is a reasonable outcome. A lower actor loss typically indicates that the actor’s selected actions are assigned higher Q-values by the critic network. However, when PER is introduced, the critic is trained on samples with higher TD errors more frequently, making the learning target more complex and potentially less stable. As a result, the critic provides more challenging feedback to the actor, leading to an increase in actor loss. This does not imply poorer actor performance; rather, it reflects the increased difficulty of learning under a more dynamic and informative critic, which can ultimately result in a more effective control policy. Figure 5 presents the dispatch results of three distributed generation units (DG1–DG3) and the BESS over the first three days of the testing set. It can be observed that the TD3+PER+Noisy model outperforms the plain TD3 baseline in balancing load demand and coordinating storage operations. Dispatch results of DG units and BESS under different models. Illustration of DG1–DG3 and BESS scheduling in the first three days of the testing set. First, regarding the DG unit outputs, the plain TD3 model exhibits rigid or unstable scheduling patterns. For instance, DG2 and DG3 remain idle during most periods, causing the system to rely heavily on a single generator or the BESS, which increases both operational cost and unbalance. By contrast, the TD3+PER+Noisy model achieves a more flexible allocation, with DG1–DG3 alternately supplying power at different times. This results in a more balanced dispatch that reduces both cost and system unbalance. Second, the BESS power and SOC trajectories highlight further differences. Under the plain TD3 model, charging and discharging events are limited, and SOC remains at relatively low levels, indicating underutilization of the storage capacity. In contrast, the TD3+PER+Noisy model dynamically adjusts BESS operation in response to load and PV fluctuations: charging during low-demand periods and discharging during peak demand periods. The resulting SOC follows a regular cyclical pattern, demonstrating improved peak shaving and valley filling. This behavior enhances the stabilizing role of storage and ensures reliable operation under load variability. Overall, the dispatch results confirm that the TD3+PER+Noisy model not only achieves lower operational cost and reduced unbalance in numerical terms, but also produces more realistic and practically interpretable scheduling strategies. These findings underscore the superiority and application potential of the proposed approach in solving the economic dispatch problem. Comparative analysis of RL controllers with enhancement techniques This study evaluates the generality of the proposed improvement techniques by conducting experiments on three distinct RL controllers: DDPG, SAC, and MPO. Each controller was evaluated in its baseline configuration as well as after incorporating Noisy Nets and PER. Table 4 summarizes the performance metrics, including training and testing operation costs and system unbalance. The convergence of the actor loss for DDPG, SAC, and MPO is illustrated in Figs. 6, 7, and 8, respectively. Table 4Result comparison between RL controllers.ModelTraining operation cost ($)Training unbalance (kW)Testing operation cost ($)Testing unbalance (kW)SAC+Noisy+PER1180.56213.8014370.42339.54SAC1538.841.8716413.88399.66Improvements23.28%–12.45%15.04%DDPG+Noisy+PER1907.14020773.47184.52DDPG2147.64021575.91454.90Improvements11.20%–3.72%59.44%MPO+Noisy+PER3823.32166.4531500.511488.39MPO4056.81116.2138124.181740.82Improvements5.76%–17.37%14.50% Result comparison between RL controllers. Actor loss convergence of DDPG variants. Actor loss convergence of SAC variants. Actor loss convergence of MPO variants. A detailed analysis of the results for each RL controller is presented in the following subsections. DDPG with enhancement techniques DDPG serves as the foundational model for TD3 and is inherently deterministic. As shown in Fig. 6, similar to TD3, the model converged successfully after experiencing some initial fluctuations during the early stages of training. The integration of Noisy Nets and PER into DDPG led to a reduction in training operation cost by 11.20% and testing operation cost by 3.72% compared to the baseline DDPG. Notably, the testing unbalance metric showed a substantial improvement, decreasing by 59.44%. It is also worth emphasizing that both the original and enhanced DDPG models achieved a training unbalance of 0 kW. This outcome can be attributed to the deterministic nature of the policy, which enables precise action selection and facilitates effective management of system unbalance. In general, deterministic models are well-suited for handling such constraints, provided that the model has sufficient complexity. However, when model capacity is inadequate, even deterministic policies may face difficulties. In contrast, stochastic policies, which generate action distributions rather than fixed outputs, often struggle to achieve highly precise control. Although the improvement in operation cost during testing was less pronounced than in the training phase, the enhanced model benefited from a more comprehensive state exploration during training. This led to a marked improvement in the testing unbalance performance, underscoring the importance of robust exploration in achieving long-term stability. SAC with enhancement techniques As a stochastic policy algorithm, SAC demonstrates competitive operational cost performance comparable to TD3. As shown in Fig. 7, the model converged after passing through an initial dip in actor loss. The negative values of the actor loss are a result of its definition in SAC and are considered normal. In its baseline configuration, SAC achieved a lower operation cost than TD3. However, because its policy outputs a probability distribution over actions, its ability to perform fine-grained control in specific states is limited. This limitation is reflected in the higher testing unbalance observed with SAC. The integration of Noisy Nets and PER with SAC resulted in a reduction of the training operation cost by 23.28% and the testing operation cost by 12.45%. Despite the stochastic policy, the incorporation of these techniques yielded significant improvements in economic efficiency. It is worth noting that the training unbalance for the improved SAC model increased relative to the baseline. A possible explanation is that, during certain episodes, the actor’s stochastic policy might have incidentally produced actions that coincided with favorable grid conditions, thereby minimizing unbalance. However, such favorable instances are not consistently reproducible across different episodes or unseen test datasets. Nevertheless, the overall effect of incorporating Noisy Nets and PER was positive, as evidenced by the reduction in the cumulative unbalance during the testing period. These findings support the conclusion that the improvement techniques enhance the performance of stochastic models in terms of both cost reduction and system balance. Although SAC achieves favorable operational cost performance, its relatively high grid unbalance during operation limits its suitability as a reliable solver for regional power grid EDP. MPO with enhancement techniques As shown in Fig. 8, the actor loss of MPO converged rapidly during training on this EDP environment. MPO is known for its training stability, especially in high-dimensional action spaces, and consistently converged to an operation cost of approximately 4000 dollars during training. However, in the context of power system dispatch, its overall performance in terms of cost efficiency and system unbalance was less competitive compared to other models. This can be attributed to the design focus of MPO, which emphasizes stable policy updates in complex environments rather than direct optimization of global performance. Furthermore, as a stochastic policy method, MPO produces probabilistic action outputs, which can limit its ability to make fine-grained control adjustments in scenarios where high precision is required. In the generation dispatch problem for electric power systems, the integration of Noisy Nets and PER into RL controllers consistently improved performance. Both deterministic and stochastic policy methods experienced reductions in overall operating costs and enhancements in system balance during testing. Although each algorithm exhibited different levels of improvement across various performance metrics, the collective evidence strongly indicates that these improvement techniques contribute positively to the robustness and generalization capability of RL models when applied to economic dispatch problems. Among all the methods examined in this study, the deterministic approach, specifically TD3 enhanced with Noisy Nets and PER, demonstrated the best performance, achieving the lowest operation costs and power unbalance values in both training and testing environments. This outcome highlights the effectiveness of combining deterministic policy frameworks with advanced exploration and sampling strategies, particularly in applications requiring high precision and reliability. Computational time analysis Table 5 compares the training time of baseline models with their counterparts enhanced by PER and Noisy Nets. The results indicate that while these enhancements consistently improve model performance, they also introduce a noticeable increase in training time. The magnitude of this increase varies across algorithms: DDPG exhibits the largest increase of approximately 159%, whereas SAC records the smallest at around 32%. Table 5Comparison of Training time for baseline models and models with PER + Noisy Nets.ModelBaseline time (min)With PER + Noisy Nets (min)Time increase (%)Number of episodesTD361.0085.0039.34450DDPG26.2568.00159.05300SAC50.6767.0032.16300MPO29.3355.5089.22300 Comparison of Training time for baseline models and models with PER + Noisy Nets. The variation in time overhead can be attributed to differences in algorithmic architecture, sensitivity to stochastic perturbations, and network update frequency. DDPG, with its relatively simple structure and high susceptibility to noisy inputs, incurs substantial computational costs when both PER and Noisy Nets are applied. Conversely, TD3 and SAC incorporate stabilization mechanisms, including target policy smoothing and delayed policy updates. These mechanisms mitigate the impact of the enhancements, leading to more moderate increases in training time. The additional computational burden of PER, including priority value computation, sampling distribution maintenance, and replay buffer updates, affects algorithms to different extents depending on the efficiency of their implementation and the structure of their replay mechanisms. These factors collectively explain the observed variation in training time across models. It should be noted that the reported values reflect single training runs and do not include the extra time required for hyperparameter tuning. The introduction of PER and Noisy Nets expands the hyperparameter search space substantially, potentially prolonging the tuning process, especially under constrained computational resources. Despite these increases, the proposed method remains practical for power system applications. The additional training time is within acceptable limits and is justified by the performance gains achieved, making the approach suitable for scenarios where a balance between solution quality and computational cost is critical. Furthermore, the inference latency (i.e., the time required for the model to produce a decision, also known as response time) exhibits no significant variation across different models and environments39 and consistently remains at the millisecond level, which is sufficiently fast. Therefore, the extended training phase does not hinder the applicability of the method in real-time operational contexts. Discussion and future research The numerical results presented above indicate that incorporating RL improvement techniques can improve the performance of RL controllers for economic dispatch in power systems. Although the environment utilized in this study is relatively simple (where even the plain TD3 model yields satisfactory solutions), the application of PER and Noisy Nets, along with careful hyperparameter tuning, brings performance close to global optimality22. In this setting, more complex controllers appear to offer limited additional benefits. They also tend to complicate hyperparameter tuning and increase training time. A key challenge in this study is the expansion of hyperparameters introduced by the enhancement techniques. The resulting parameter space is large, making exhaustive grid search impractical. In practice, both academic and industrial work often fixes most settings to well-established defaults28 and tunes only a small subset that is most sensitive to performance. Although a systematic tuning framework was not the focus, the chosen settings follow established RL practice, supporting fairness and reproducibility. This consideration explains the decision not to include more advanced algorithms such as Rainbow28 and DreamerV340 in the present experiments. These methods are substantially more complex and are typically designed for higher-dimensional or perception-heavy tasks, which differ from the EDP formulation studied here. Their training is also computationally demanding. In practice, effective use often relies on adaptive hyperparameter tuning or meta-learning procedures; without such support, manual tuning can be slow and unreliable. Evaluation of these high-complexity models is deferred to future work when additional computational resources become available. Commercial solvers such as CPLEX41 have been widely applied to EDPs through branch-and-bound or mixed-integer programming formulations. They are capable of producing near-exact solutions in relatively small-scale or simplified settings. However, their computational burden grows rapidly with the number of generators, constraints, and time intervals. This makes them less practical for large-scale or real-time applications. In contrast, the reinforcement learning framework employed in this study avoids repeatedly solving large optimization problems. Once trained, the controller generates decisions within milliseconds. While solver-based approaches may deliver slightly higher accuracy in offline conditions, the RL-based method provides superior adaptability, faster inference speed, and robustness under dynamic operating conditions. These properties are essential for online power system operation. Another challenge lies in the fundamental differences between RL environments used in gaming and those in power system control. In gaming applications, the state space is typically represented by high-dimensional image frames. In power system scenarios, the state consists of key system parameters computed in real time. As the scale of power system models increases, the computational cost associated with agent–environment interactions grows substantially. This is also the reason why additional power system–related parameters, including CO emissions and frequency deviation, were not included in the current experiments. Incorporating these parameters would significantly increase the computational time required for model–environment interactions. A practical mitigation is to parallelize the RL environment on multi-core CPU architectures. This approach reduces the computational burden. At present, the proposed TD3 model enhanced with PER and Noisy Nets demonstrates satisfactory performance on the EDP problem within existing regional power grids. In more complex scenarios, such as regional grids characterized by dynamic pricing and stochastic topologies, retraining the controller for the new environment is expected to produce effective results. This highlights the adaptability and generalization potential of the proposed approach across diverse power-system configurations. When the controller is neural network-based, its capacity should be matched to task complexity. Insufficient capacity risks underfitting, whereas excessive capacity and weak regularization can lead to overfitting. Beyond the EDP, future research might extend RL-based controllers to tackle even more challenging problems, such as stability control in complex power systems, blackout prevention, and management under extreme conditions. In such scenarios, advanced RL frameworks (e.g., those based on the Rainbow architecture) may be required to accommodate the diverse constraints and operational conditions present in large-scale power systems. Conclusions This study proposes a novel approach, TD3+PER+Noisy Nets, which integrates RL improvement techniques to address the EDP in power systems. Unlike methods from past research that combine RL models with external approaches, such as MIP22, to enhance performance, the proposed pipeline directly refines the RL model’s workflow through internal architectural modifications. By incorporating Noisy Nets, the agent’s exploration capability is significantly enhanced, enabling more effective search within the action space. Simultaneously, the utilization of PER improves sample efficiency and accelerates the training process. Experimental results demonstrate that the proposed method substantially reduces operational costs while enhancing power balance during both training and testing phases. When compared with the baseline plain TD3 model, the TD3+PER+Noisy Nets model achieves a 54.6% reduction in testing operation cost and a 95.3% reduction in cumulative testing unbalance. Comparative analyses across various RL models, including both deterministic and stochastic policy methods, further confirm the general effectiveness of the improvement techniques. Among all evaluated methods, the deterministic TD3+PER+Noisy Nets model exhibited the best overall performance, achieving the lowest operation cost and power unbalance in both training and testing environments. In summary, the proposed method provides an alternative to hybrid approaches by leveraging targeted improvement techniques to enhance controller performance for specific power system problems. This study not only establishes a solid theoretical foundation but also offers practical evidence for the successful integration of RL improvement techniques in solving complex optimization problems in power systems. Future research should focus on adaptive hyperparameter tuning, parallelized environment simulations, and extending these techniques to more challenging control tasks, such as stability management and blackout prevention, thereby paving the way for more robust and efficient RL-based controllers in the energy sector. Author contributions C.X. conceived the experiments, conducted the experiments, analysed the results, and wrote the manuscript. N.H. and M.I. contributed to the experimental design and reviewed the manuscript. W.J.K.R., H.B.M., and H.A.B.I. reviewed and revised the manuscript. All authors approved the final version of the manuscript. Funding This research did not receive any specific grant from funding agencies in the public, commercial, or not-for-profit sectors. Data availability The datasets and environment analysed during the current study are available in the GitHub repository: https://github.com/EnergyQuantResearch/Optimal-Energy-System-Scheduling-Combining-Mixed-Integer-Programming-and-Deep-Reinforcement-Learning. Declarations Competing interests The authors declare no competing interests. Footnotes Publisher’s note Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations. References Associated Data Data Availability Statement The datasets and environment analysed during the current study are available in the GitHub repository: https://github.com/EnergyQuantResearch/Optimal-Energy-System-Scheduling-Combining-Mixed-Integer-Programming-and-Deep-Reinforcement-Learning.