Multi-Objective Reinforcement Learning for Power Grid Topology Control

Thomas Lautenbacher, Ali Rajaei, Davide Barbieri, Jan Viebahn, Jochen L. Cremer · 2025

Read Multi-Objective Reinforcement Learning for Power Grid Topology Control by Thomas Lautenbacher, Ali Rajaei, Davide Barbieri, Jan Viebahn, Jochen L. Cremer (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2502-00040.txt.

Introduction

Thomas Lautenbacher, Ali Rajaei, Davide Barbieri, Jan Viebahn, and Jochen Cremer train grid controllers that juggle several goals at once. Their 2025 paper targets power grid congestion, which grows as more sectors electrify. Switching substation layouts can ease congestion, yet operators barely use the lever.

Single-goal learning agents force one trade-off up front, which may not match operator needs. The team instead builds agents that return a set of Pareto-best policies. Those policies balance line loading, layout change, and switching rate against each other. Early case studies beat random search at mapping the trade-off front. The multi-goal policies also prevent grid failure more often under faults. They hold up better when the training budget shrinks, too.

Catalog record

CREATORThomas Lautenbacher, Ali Rajaei, Davide Barbieri, Jan Viebahn, Jochen L. Cremer
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE31 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2502-00040

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Multi-Objective Reinforcement Learning for Power Grid Topology Control Thanks: This research was carried out for an MSc thesis project in Delft University of Technology, supported by the ERASMUS+ program. Thomas Lautenbacher, Ali Rajaei, Davide Barbieri, Jan Viebahn, Jochen L. Cremer Abstract Transmission grid congestion increases as the electrification of various sectors requires transmitting more power. Topology control, through substation reconfiguration, can reduce congestion but its potential remains under-exploited in operations. A challenge is modeling the topology control problem to align well with the objectives and constraints of operators. Addressing this challenge, this paper investigates the application of multi-objective reinforcement learning (MORL) to integrate multiple conflicting objectives for power grid topology control. We develop a MORL approach using deep optimistic linear support (DOL) and multi-objective proximal policy optimization (MOPPO) to generate a set of Pareto-optimal policies that balance objectives such as minimizing line loading, topological deviation, and switching frequency. Initial case studies show that the MORL approach can provide valuable insights into objective trade-offs and improve Pareto front approximation compared to a random search baseline. The generated multi-objective RL policies are 30 30 % more successful in preventing grid failure under contingencies and 20 20 % more effective when training budget is reduced - compared to the common single objective RL policy. I Introduction The energy transition and the shift toward renewable energy sources are crucial steps for mitigating climate change and ensuring a sustainable energy future. However, this transition poses significant operational challenges for system operators, including congestion management. Transmission network topology control is an under-utilized and non-costly source of flexibility. Adjusting the network topology, such as line switching or modifying busbar connections within substations, can reroute power flows to prevent line overloads and mitigate cascading outages [ 1 , 2 , 3 ] . In addition to maintaining continuous electricity supply, power systems must address other objectives, such as minimizing asset wear, reducing operational cost, and mitigating environmental impacts. Achieving these objectives requires a multi-objective approach to decision-making that maintains grid security while addressing other operational objectives [ 4 ] . Transmission network topology control problem can be modeled as a mixed-integer non-linear optimization problem which is computationally challenging to approach. The so-called combinatorial explosion of possible topologies and the complex nonlinear nature of power systems [ 1 ] makes this problem challenging. To address these challenges, heuristic and expert rule-based approaches, such as in [ 5 , 6 , 7 , 8 ] are developed to determine corrective topological actions to relieve congestion. However, these approaches do not provide a sequence of control actions and may lead to sub-optimal solutions. To provide sequences of actions, recently researchers explored the use of reinforcement learning (RL) and Artificial Intelligence (AI) more broadly for topological control [ 9 , 3 , 10 , 11 ] . Studies such as [ 12 , 13 , 14 , 15 ] explore RL-based approaches, including the deep duelling Q-network (DDQN) initialized with imitation learning [ 12 ] , the Semi-Markov actor-critic algorithm [ 13 ] , the cross-entropy method with importance sampling [ 14 ] , and the proximal policy optimization (PPO) [ 15 ] . Additionally, [ 16 ] develops an AlphaZero-based approach using Monte-Carlo tree search to simulate future outcomes, and guide the agent toward long-term strategies, while [ 17 ] presents a curriculum-based approach to improve learning efficiency and stability. Building on these ideas, [ 18 ] combines curriculum learning with tree search to benefit from long-term strategies as well as the efficiency and stability. Some studies focus on addressing the combinatorial explosion of the topology control problem through hierarchical RL [ 19 ] and multi-agent RL [ 20 ] . Furthermore, [ 21 ] proposes a reward design using multiple metrics to reduce overloads. However, the developed approaches in [ 6 , 7 , 5 , 8 , 12 , 13 , 14 , 15 , 16 , 17 , 18 , 19 , 20 , 21 ] focus on single operational objectives and providing only a single policy. This limits their application to address the trade-offs inherent in the multi-objective nature of power systems and to provide a set of policies for operators to select from. This paper proposes a multi-objective RL (MORL) approach to address the network topology control problem. Despite previous studies on RL approaches, to the best of the authors’ knowledge, a multi-policy MORL approach for topology control has not been investigated before. To this end, we implement deep optimistic linear support (DOL) and multi-objective PPO (MOPPO) to generate a set of Pareto-optimal policies. Additionally, we develop custom reward functions for different operational objectives, including line loading, topological deviation, and switching frequency. The proposed MORL approach not only shows the trade-offs between these objectives but also provides a set of policies that balance these trade-offs, offering a decision-support approach for system operators. By considering multiple rewards, the approach effectively relieves grid congestion while addressing other operational objectives. The rest of the paper is organized as follows. Section II presents the proposed MORL approach and the design of the reward functions. Section III presents the case studies, investigating the efficiency and robustness of the proposed approach. Section IV provides discussions and concludes the paper. II Methodology This paper aims to provide a decision-support approach for transmission system operators to perform topological control considering multiple objectives. The proposed approach integrates a multi-objective adaptation of the PPO algorithm [ 22 , 23 ] (MOPPO) with deep optimistic linear support [ 24 ] . To capture different operational objectives, we design custom reward functions that address line loading, topological deviation and switching frequency. The proposed approach considers a multi-policy MORL [ 25 ] , which results in a set of optimal solutions rather than a single solution. This allows system operators to better understand the trade-offs among objectives and select the most appropriate policy. Fig. 1 depicts the proposed approach. By using DOL as an outer loop method within the MORL approach, the convex coverage set of solutions is constructed iteratively. In each iteration, the DOL generates a new set of weight vectors w and gives one weight vector with the highest priority to MOPPO, that is then trained in the multi-objective environment. MOPPO uses w to account for the multiple rewards \textbf{R}_{t} . After training, the MOPPO is evaluated. If the average value vector over the evaluation episodes \mathbfcal{V} , found by the MOPPO, is Pareto optimal, it is added to the convex coverage set (CCS). The detailed methodology is explained in the following. Fig. 1 : Schematic of the proposed MORL approach with deep optimistic linear support. II-A Single-Policy Multi-Objective PPO In order to learn on multiple rewards, the agent needs to receive a reward signal for each of the objectives. To this end, we extend the original RL grid environment into a multi-objective environment, allowing the agent to receive a d-dimensional reward vector \mathbf{r}_{t}\in\mathbb{R}^{d} , where d is the number of objectives. In this paper, the rewards reflect the operational objectives of reducing the line loading, decreasing the topological deviation, and reducing the switching frequency, explained in detail in section II-B . At the core of the proposed approach is the MOPPO Algorithm 1 , which is

Read the complete text