Hierarchical Reinforcement Learning with Runtime Safety Shielding for Power Grid Operation
Gitesh Malik · 2026
Read Hierarchical Reinforcement Learning with Runtime Safety Shielding for Power Grid Operation by Gitesh Malik (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2604-14032.txt.
Introduction
Malik pairs a high-level reinforcement learning policy with a deterministic runtime safety shield for power-grid operation. The policy proposes abstract actions while the shield filters unsafe ones through fast forward simulation, enforcing safety as a runtime invariant regardless of policy quality. On Grid2Op stress tests and zero-shot deployment on a large 2021 transmission grid, the hierarchy survives longer with lower peak loading than flat or safety-only methods.
Catalog record
| CREATOR | Gitesh Malik |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 44561 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2604.14032
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Hierarchical Reinforcement Learning with Runtime Safety Shielding for Power Grid Operation Gitesh Malik Abstract Reinforcement learning has shown promise for automating power-grid operation tasks such as topology control and congestion management. However, its deployment in real-world power systems remains limited by strict safety requirements, brittleness under rare disturbances, and poor generalization to unseen grid topologies. In safety-critical infrastructure, catastrophic failures cannot be tolerated, and learning-based controllers must operate within hard physical constraints. This paper proposes a safety-constrained hierarchical control framework for power-grid operation that explicitly decouples long-horizon decision-making from real-time feasibility enforcement. A high-level reinforcement learning policy proposes abstract control actions, while a deterministic runtime safety shield filters unsafe actions using fast forward simulation. Safety is enforced as a runtime invariant, independent of policy quality or training distribution. The proposed framework is evaluated on the Grid2Op benchmark suite under nominal conditions, forced line-outage stress tests, and zero-shot deployment on the ICAPS 2021 large-scale transmission grid without retraining. Results show that flat reinforcement learning policies are brittle under stress, while safety-only methods are overly conservative. In contrast, the proposed hierarchical and safety-aware approach achieves longer episode survival, lower peak line loading, and robust zero-shot generalization to unseen grids. These results indicate that safety and generalization in power-grid control are best achieved through architectural design rather than increasingly complex reward engineering, providing a practical path toward deployable learning-based controllers for real-world energy systems. I Introduction Modern power grids are undergoing rapid structural transformation driven by large-scale renewable integration, increasing electrification of demand, and growing network interconnection complexity. These trends significantly increase operational uncertainty and impose substantial cognitive and computational burden on human operators, who must continuously manage congestion, maintain stability, and respond to contingencies under strict physical constraints. Reinforcement learning (RL) has emerged as a promising paradigm for sequential decision-making in power-system operation, including topology control, redispatch, and corrective action selection. Benchmark platforms such as Grid2Op [ 5 ] and the Learning to Run a Power Network (L2RPN) competitions [ 8 , 9 ] have demonstrated that learning-based agents can achieve strong performance under nominal operating conditions. However, despite these advances, the deployment of RL-based controllers in real-world power systems remains limited. A fundamental barrier is safety. Power grids are safety-critical infrastructures in which violations of physical constraints such as thermal line limits, voltage bounds, or network islanding can trigger cascading failures and large-scale blackouts. Standard reinforcement learning algorithms optimize expected cumulative reward and provide no hard guarantees against catastrophic actions, especially under rare disturbances or distributional shift [ 1 , 7 ] . Even when large penalties are incorporated into the reward function, unsafe actions may still be selected during exploration or in previously unseen operating regimes. A second major challenge is generalization. Power grids vary widely in size, topology, generation mix, and operating conditions. Training learning-based controllers directly on large, realistic grids is computationally expensive and often infeasible. Policies trained on one environment frequently fail when deployed on unseen grids, raising serious concerns about robustness and reliability [ 6 ] . These limitations suggest that the core difficulty is not the choice of reinforcement learning algorithm alone, but the system architecture within which learning is embedded. In real-world grid operation, decision-making is inherently layered: operators reason at a strategic level, while protection systems and feasibility checks enforce physical constraints in real time. Unlike prior Grid2Op and L2RPN agents, our approach enforces safety as a runtime invariant rather than a learned objective, and demonstrates zero-shot transfer to a large-scale transmission grid never seen during training. Motivated by this observation, we propose a hierarchical reinforcement learning framework with explicit runtime safety shielding for power-grid operation. A high-level policy learns abstract, long-horizon control strategies, while a deterministic safety layer evaluates and filters actions at execution time using fast forward simulation. This architectural separation decouples decision-making from feasibility enforcement, ensuring that hard operational constraints are satisfied independently of policy quality or training distribution. II Hierarchical and Safety-Aware Control Framework In this section, we formalize the control problem underlying the proposed power-grid control system and introduce a hierarchical decision-making framework augmented with runtime safety shielding. II-A Problem Formulation We model the power grid as a discrete-time dynamical system interacting with an agent through control actions. At each time step t , the environment is characterized by a state s_{t}\in\mathcal{S} derived from system observations, including line loading ratios, generator outputs, load demands, and network topology. The system evolves according to s_{t+1}=f(s_{t},a_{t},w_{t}), (1) where w_{t} represents exogenous disturbances such as load fluctuations, renewable variability, or forced line outages. Operational safety is captured through hard constraints on line thermal limits: \rho_{\ell}(s_{t})\leq 1,\quad\forall\ell\in\mathcal{L}. (2) II-B Hierarchical Control Architecture Hierarchical reinforcement learning has long been proposed as a mechanism for scaling decision-making in complex domains [ 10 , 4 ] . We adopt a two-layer architecture: • High-level policy: A learned policy \pi_{\theta}(a_{t}\mid s_{t}) proposing abstract control actions. • Runtime safety shield: A deterministic module that evaluates feasibility before execution. At each time step, the policy proposes \tilde{a}_{t} , and the executed action is given by a_{t}=S(s_{t},\tilde{a}_{t}), (3) where S(\cdot) enforces safety constraints. II-C Runtime Safety Shield The safety shield performs a one-step forward simulation and rejects actions that violate constraints: \max_{\ell\in\mathcal{L}}\rho_{\ell}(f(s_{t},\tilde{a}_{t}))>1. (4) This mechanism is conceptually related to control barrier functions [ 3 ] and action shielding [ 2 ] , but adapted to discrete, combinatorial grid control. III Algorithmic Realization and Execution Dynamics This section presents the concrete algorithmic realization of the proposed hierarchical and safety-aware control framework. We describe the policy parameterization, execution loop, safety intervention logic, and the variants evaluated in our experiments. The goal is to make explicit how abstract design principles translate into a deployable decision-making system. III-A Policy Parameterization Let s_{t}\in\mathcal{S} denote the observation at time step t . We consider a parametric policy \pi_{\theta} implemented as a neural network mapping observations to action logits: \pi_{\theta}:\mathcal{S}\rightarrow\mathbb{R}^{|\mathcal{A}|}. (5) The logits are transformed into a categorical distribution over actions: p_{\theta}(a\mid s_{t})=\mathrm{softmax}(\pi_{\theta}(s_{t})). (6) The policy is trained using reinforcement learning to maximize the expected discounted return: J(\theta)=\mathbb{E}_{\pi_{\theta}}\left[\sum_{t=0}^{T}\gamma^{t}r_{t}\right], (7) where \gamma\in(0,1) is the discount factor. Importantly, the policy is agnostic to safety constraints. It does not receive explicit supervision about constraint violations and is trained solely on reward signals. This deliberate separation ensures that learning focuses on strategic behavior rather than fragile constraint memorization. III-B Hierarchical Decomposition To improve scalability and generalization, we decompose control into two conceptual levels: 1. High-level decision layer: The high-level policy proposes a candidate action \tilde{a}_{t} based on global grid features. This layer captures long-horizon structure, such as congestion relief patterns and topology adaptation strategies. 2. Low-level execution layer: The low-level layer evaluates \tilde{a}_{t} against instantaneous feasibility constraints and either accepts or modifies it. This layer does not optimize reward; it only enforces safety. Formally, the executed action is given by a_{t}=\begin{cases}\tilde{a}_{t},&\text{if }\tilde{a}_{t}\in\mathcal{A}_{\mathrm{safe}}(s_{t}),\\ P(s_{t},\tilde{a}_{t}),&\text{otherwise},\end{cases} (8) where P denotes the projection operator implemented by the safety shield. This decomposition ensures that policy learning and safety enforcement operate on different time scales and with different objectives. III-C Safety Shield and Constraint Evaluation The safety shield evaluates candidate actions using a forward simulation of grid dynamics. Given a proposed action \tilde{a}_{t} , the shield predicts the resulting line loadings: \hat{\rho}_{\ell}=\rho_{\ell}(f(s_{t},\tilde{a}_{t})),\quad\forall\ell\in\mathcal{L}. (9) An action is deemed