RL2Grid: Benchmarking Reinforcement Learning in Power Grid Operations
Enrico Marchesini, Benjamin Donnot, Constance Crozier, Ian Dytham, Christian Merz, Lars Schewe, Nico Westerbeck, Cathy Wu, Antoine Marot, Priya L. Donti · 2025
Read RL2Grid: Benchmarking Reinforcement Learning in Power Grid Operations by Enrico Marchesini, Benjamin Donnot, Constance Crozier, Ian Dytham, Christian Merz, Lars Schewe, Nico Westerbeck, Cathy Wu, Antoine Marot, Priya L. Donti (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2503-23101.txt.
Introduction
Enrico Marchesini and co-authors launch a benchmark for teaching AI to run power grids. Learning by trial and error suits grid control, since grids shift fast and must stay within hard physical limits. Yet progress stalls: grid tasks vary from lab to lab, so results cannot be compared. The team built a shared test bed with grid operators.
The benchmark, RL2Grid, fixes the tasks, the data an agent sees, its allowed moves, and its rewards. It bakes in operator know how as safety rails. Classic learning methods were run to set reference scores. Their gaps show how far the field sits from real grid needs. The authors map next steps for agents that respect physics by design.
Catalog record
| CREATOR | Enrico Marchesini, Benjamin Donnot, Constance Crozier, Ian Dytham, Christian Merz, Lars Schewe, Nico Westerbeck, Cathy Wu, Antoine Marot, Priya L. Donti |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 58 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2503-23101
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
RL2Grid: Benchmarking Reinforcement Learning in Power Grid Operations Enrico Marchesini, Benjamin Donnot, Constance Crozier, Ian Dytham, Christian Merz, Lars Schewe, Nico Westerbeck, Cathy Wu, Antoine Marot, Priya L. Donti Abstract Reinforcement learning (RL) can provide adaptive and scalable controllers essential for power grid decarbonization. However, RL methods struggle with power grids’ complex dynamics, long-horizon goals, and hard physical constraints. For these reasons, we present RL2Grid, a benchmark designed in collaboration with power system operators to accelerate progress in grid control and foster RL maturity. Built on RTE France’s power simulation framework, RL2Grid standardizes tasks, state and action spaces, and reward structures for a systematic evaluation and comparison of RL algorithms. Moreover, we integrate operational heuristics and design safety constraints based on human expertise to ensure alignment with physical requirements. By establishing reference performance metrics for classic RL baselines on RL2Grid’s tasks, we highlight the need for novel methods capable of handling real systems and discuss future directions for RL-based grid control. 1 1 1 Code is available at https://github.com/emarche/RL2Grid 1 Introduction Power grids require a rapid transition to low-carbon energy and improved robustness against climate-induced extremes in order to combat climate change. This requires operating under increasing speed, scale, and uncertainty, due in large part to evolving supply and demand profiles resulting from distributed devices and variable renewable energy sources (VREs) ( Li et al.,, 2023 ) . This integration creates significant challenges for human operators and traditional power system solvers ( Marot et al., 2022b, ) . To clarify what a power grid is, Figure 1 shows a simplified scenario with four substations (dots) interconnected by transmission lines (edges), two power generators, and two loads connected to buses within each substation. Generators produce power that flows through transmission lines to meet demands (loads). Transmission leads to power losses due to resistive heat on the lines, and substations (which may contain multiple buses) can act as “switches” to direct power flows to an extent. All these electrical components have physical constraints that must be satisfied (e.g., generators have ramping limits preventing arbitrary instantaneous changes in power output, and transmission lines have maximum capacities, with prolonged overloads causing disconnections and permanent damage). Deep reinforcement learning (RL) is a promising approach for power grid operations, having demonstrated impressive control performance over the last decade ( Mnih et al.,, 2013 ; Silver et al.,, 2016 ; Wurman et al.,, 2022 ) . However, power grids encompass many open research questions in RL , including dealing with complex dynamics, aleatoric uncertainty, learning long-horizon goals, and satisfying hard physical constraints. Investigating realistic power grid tasks from an RL perspective could thus yield substantial benefits for both society and the RL research community. Nonetheless, progress in relevant RL methodologies is hindered by a lack of standardized benchmarks that can help promote and monitor progress, identify bottlenecks, and develop insights to address real-world challenges. We fill this gap by introducing RL2Grid, an RL benchmark for realistic power grid operations designed in collaboration with major transmission system operators (TSOs). RL2Grid aims to accelerate progress in grid control and advance RL methods tailored for real-world problems by modeling a diverse, standardized set of increasingly complex power grid environments for RL research. These tasks build upon RTE France’s Grid2Op ( RTE France,, 2020 ) , a realistic power grid simulation framework, and are presented within a standard Gymnasium-based interface, alongside common state and action spaces, and rewards, to provide a shared base for comparison. We additionally perform an in-depth analysis of RL2Grid’s design choices by investigating the quality of the actions available in different task settings. To incorporate power grids’ real operation practices and hard physical requirements, we also introduce a heuristic module incorporating common line reconnection and idle practices in the grid dynamics ( RTE France,, 2020 ) , as well as constrained task formalizations for safe RL ( Gu et al.,, 2024 ) . Finally, by extending the well-known CleanRL library ( Huang et al.,, 2022 ) to include flexible configurations for algorithm implementation details, we conduct a comprehensive empirical comparison of classic RL algorithms that are frequently used in the literature as baselines or building blocks for more complex approaches. Our codebase is available as supplementary material. Figure 1: A high-level overview of a power grid action (bus split) to address an overloaded line (red). Finally, our collaboration with TSOs allows us to extensively discuss ( Section 6 ): (i) directions to further improve the realism of power grid simulators to enable last-mile development and deployment of the methodological advances we hope RL2Grid will promote; and (ii) the relationship of power grid operations to open challenges in RL. Through RL2Grid, we aim to foster the maturity of RL methods for real-world power grid domains and provide a standardized basis for comparative analysis. 2 Preliminaries A power grid task can be modeled as a Markov decision process (MDP)—a tuple (\mathcal{S},\mathcal{A},\mathcal{P},\rho,R,\gamma) , where \mathcal{S} and \mathcal{A} are the finite sets of states and actions, respectively, \mathcal{P}:\mathcal{S\times\mathcal{A}\times\mathcal{S}}\to[0,1] is the state transition probability distribution, \rho:\mathcal{S}\to[0,1] is the initial uniform state distribution, R:\mathcal{S}\times\mathcal{A}\to\mathbb{R} is a reward function, and \gamma\in[0,1) is the discount factor. In policy optimization algorithms, agents learn a parameterized policy \pi:\mathcal{S}\times\mathcal{A}\to[0,1] , modeling the probability of taking an action a_{t}\in\mathcal{A} in a state s_{t}\in\mathcal{S} at a certain step t . In value-based algorithms, agents learn state and/or action value functions V_{\pi} and Q_{\pi} , representing the expected discounted return when starting from a state s (and action a for Q_{\pi} ) and following the policy \pi thereafter. In these contexts, agents typically use a greedy policy taking the action corresponding to argmax over Q_{\pi} . The goal is to find a policy that maximizes the expected discounted return \mathbb{E}_{\pi}[\sum_{t=0}^{\infty}\gamma^{t}R(s_{t},a_{t})] . To promote safety, we also model grid tasks as constrained MDPs (CMDPs) ( Altman,, 1998 ) , by adding a set of constraints \mathcal{C}:=\{C_{i}\}_{i=1,\ldots,n} defined over unsafe state and action pairs identified by indicator cost functions. These can be used to describe both instantaneous constraints which must be satisfied at every point in time, and cumulative constraints specifying limits on the accumulation of cost over a specified horizon. For instance, policy optimization approaches typically transform the CMDP into an equivalent unconstrained Lagrangian optimization problem \mathcal{L} over the policy parameters using dual variables as \mathcal{L}^{\pi}(\bm{\lambda})=J_{R}+\mathcal{L}_{\mathcal{C}}(\bm{\lambda}) , where \mathcal{L}_{\mathcal{C}}(\bm{\lambda})=-\sum_{i=1}^{n}\lambda_{i}(V^{\pi}_{C_{i}}-\tau_{i}) , J_{R} is the return objective to maximize, \bm{\lambda}=\{\lambda_{i}\}_{i=1,\dots,n} act as penalties on J_{R} for each constraint, \bm{\tau}=\{\tau_{i}\}_{i=1,\dots,n} are the constraint thresholds, and V^{\pi}_{\mathcal{C}}=\{V_{C_{i}}^{\pi}\}_{i=1,\dots,n} are the expected cost returns. 3 RL2Grid Benchmark RL2Grid considers the general setting of operating a power grid via topology optimization, as well as redispatch and curtailment actions (wrapped within a traditional Gymnasium interface), in order to keep the grid operational over a long horizon—a month of operations divided into 5 minute steps: (i) Topology optimization involves identifying substations where a bus-split action can mitigate the overload by adjusting the grid topology (i.e., how elements are interconnected in the grid). This approach is cost-effective for grid operators as it typically involves simple switch activation. 2 2 2 There is some uncertainty (and debate) regarding how frequently each component can be switched safely in practice, without degrading the underlying equipment. However, determining the “optimal” topology from the exponential number of