Comparing Traditional and Reinforcement-Learning Methods for Energy Storage Control
Elinor Ginzburg, Itay Segev, Yoash Levron, Sarah Keren · 2025
Read Comparing Traditional and Reinforcement-Learning Methods for Energy Storage Control by Elinor Ginzburg, Itay Segev, Yoash Levron, Sarah Keren (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2506-00459.txt.
Introduction
Elinor Ginzburg, Itay Segev, Yoash Levron, and Sarah Keren compared classic and learning-based control of energy storage in twenty twenty-five. Batteries and other storage units need a policy that says when to charge and when to discharge. Classic methods compute the best policy for a set case, while reinforcement learning trains a general policy from trial and error.
Their test bed is a simple micro-grid with a load, a solar panel source, and one storage unit. Three cases grow harder in turn: perfect storage, storage with losses, and storage with losses on the lines too. The authors spell out each case and its math hurdles, then compare both methods head to head. They map where each method wins and suggest open questions for careful use of learning in power systems.
Catalog record
| CREATOR | Elinor Ginzburg, Itay Segev, Yoash Levron, Sarah Keren |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 37 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2506-00459
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Comparing Traditional and Reinforcement-Learning Methods for Energy Storage Control Elinor Ginzburg, Itay Segev, Yoash Levron, Sarah Keren Abstract We aim to better understand the tradeoffs between traditional and reinforcement learning (RL) approaches for energy storage management. More specifically, we wish to better understand the performance loss incurred when using a generative RL policy instead of using a traditional approach to find optimal control policies for specific instances. Our comparison is based on a simplified micro-grid model, that includes a load component, a photovoltaic source, and a storage device. Based on this model, we examine three use cases of increasing complexity: ideal storage with convex cost functions, lossy storage devices, and lossy storage devices with convex transmission losses. With the aim of promoting the principled use RL based methods in this challenging and important domain, we provide a detailed formulation of each use case and a detailed description of the optimization challenges. We then compare the performance of traditional and RL methods, discuss settings in which it is beneficial to use each method, and suggest avenues for future investigation. Introduction Energy storage devices are vital in modern power systems especially given the increasing penetration of renewable energy sources. Power production of renewable sources such as solar and wind cannot be controlled, thus making their behaviour intermittent and unreliable. Thus, to effectively incorporate these technologies into the existing grid, storage devices must be used. In this manner, surplus energy from renewable sources during low-demand hours can be stored, and released during peak times, ensuring the stability and efficiency of the modern grid operation as well as reducing carbon emission ( Hannan et al. 2017 ; Comello, Reichelstein, and Sahoo 2018 ) . Moreover, the surge in electric vehicle production further amplifies the need for efficient storage control to manage the charging process intelligently to minimize strain on the grid and optimize consumption patterns ( Karden et al. 2007 ; Verma et al. 2021 ) . Consider the simplified micro-grid represented in Figure 1 , with a load (consumption) component, a PV with generation capabilities, a storage device, and a connection to a generator, which may represent local fuel-based generation or the electrical grid from which power can be purchased. A storage control policy will account for the predicated load and set the current state of charge (SOC) of the battery to minimize the amount of electricity acquired from the generator. Figure 1: A simplified microgrid, comprising of a generator, a photovoltaic source (PV), a load, and a storage device. Traditional methods for storage control typically seek an optimal value for power to charge or discharge at every time step over a fixed horizon. Thus, the number of decision variables increases rapidly and the problem quickly becomes high-dimensional. Moreover, the optimization problems are often non-convex either because the objective function is non-convex or because the decision variables are defined over a non-convex set. As a result, gradient-based methods are often inefficient since they tend to converge to local minima. Another main limitation lies in that traditional methods heavily rely on a model of the environment and are not designed to adapt to changes in the dynamics of the environment. Finally, traditional methods are designed to find an optimal solution for a given instance and have no generalization capabilities. Thus, each system (e.g., household) needs to be optimized separately. This may be sufficient if the objective is to optimize a single micro-grid but is insufficient if the objective is to find a general policy that will yield good performance for different micro-grids. To address this challenge, various machine learning methods, and reinforcement learning (RL) methods in particular have been introduced. Their strength lies in their ability to use a data-driven approach to produce high accuracy and efficient solutions, despite the high dimensional feature space. A major advantage of such approaches is that full knowledge of the system’s dynamics, its input signal, and its statistics is often not required. Instead, deep-learning-based RL methods are designed to learn from data collected from the system and improve performance by exploring unknown aspects of the system. A key weakness of these approaches is that they often require a vast amount of interaction data to learn effectively. This high sample complexity makes training expensive and time-consuming, especially in real-world environments, like the one considered here, where data collection is costly or slow. In light of this emerging trend in this crucial application, we aim to better understand the tradeoffs between the traditional and RL approaches to storage management. More specifically, we wish to better understand the performance loss incurred when the physical model of the storage component is not known, and a data-driven approach is used instead. Our comparison is based on the simplified micro-grid represented in Figure 1 , with a load component (representing consumption), a PV with generation capabilities, a storage device, and a connection to a generator, e.g., the grid. We examine three use cases of increasing complexity: ideal storage with convex cost functions, lossy storage devices, and lossy storage devices with non-convex cost functions. A key characteristic of most existing approaches to storage control is that they have been developed by energy systems experts who use off-the-shelf RL methods which may not be the most suitable. This is mostly because energy systems are highly complex and require extensive domain knowledge that is not accessible to general AI researchers and because creating effective collaboration between researchers in these very different fields is challenging. Thus, in light of the emerging trend of using RL in energy systems and the potential societal and environmental benefits of optimizing storage control, our objective is to facilitate this bridge by formulating the storage optimization in a way that is comprehensible to AI researchers and by better understanding the effectiveness of using RL in this critical application. Our examination is based on three variations of increasing complexity of the setting depicted in Figure 1 . Background and Related Work There are several aspects to maximizing the efficiency, reliability, and safety of methods for storage control. These include monitoring the charge and discharge policy, assessment of the overall performance of the storage device over time, temperature control, and more. Our focus here is on the former challenge and on comparing traditional approaches to novel RL methods for finding policies that minimize the expected overall cost of power acquired from the generator. The problem of managing grid-connected storage device is well-known and has with various approaches based on traditional control methods ( Jiang et al. 2014 ) , including shortest-path methods ( Levron and Shmilovitz 2010 ) , Pontryagin’s minimum principle ( Chowdhury et al. 2021 ) , Dynamic Programming (DP) ( Zargari et al. 2023 ) , and Model-Predictive Control (MPC) ( Zhang et al. 2020 ) . Such methods are designed to optimize the storage policy over a fixed time frame or episode , given a model of the load, storage, and generation dynamics. While the complexity of these solutions is typically polynomial in the state-space size, however this size is exponential in the number of environment features and granularity of discretization. In addition, these methods heavily rely on prior knowledge of the system dynamics and input signals. For instance, classic use