A Reinforcement Learning Approach for Optimal Control in Microgrids

1 st Davide Salaorni, 2 nd Federico Bianchi, 3 rd Marcello Restelli, 4 st Francesco Trovò · 2025

Read A Reinforcement Learning Approach for Optimal Control in Microgrids by 1 st Davide Salaorni, 2 nd Federico Bianchi, 3 rd Marcello Restelli, 4 st Francesco Trovò (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2506-22995.txt.

Introduction

Davide Salaorni and co-authors use reinforcement learning to run small local power grids, in a paper from 2025. Solar and wind now feed many such microgrids, which juggle making, storing, and selling power. Set rules for that juggling often waste chances. A learning agent can find better habits from past data on output, use, and prices.

Their agent learns when to trade power and when to charge or drain the battery. A digital twin of the storage system mimics wear, so the lessons stay realistic. Tests used real data from a grid in Italy. The learned policy beat rule-based control and older learning methods. The authors call it a robust tool for smart grid care. It could cut costs for towns that run their own power.

Catalog record

CREATOR1 st Davide Salaorni, 2 nd Federico Bianchi, 3 rd Marcello Restelli, 4 st Francesco Trovò
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE40 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2506-22995

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

A Reinforcement Learning Approach for Optimal Control in Microgrids 1 st Davide Salaorni, 2 nd Federico Bianchi, 3 rd Marcello Restelli, 4 st Francesco Trovò Abstract The increasing integration of renewable energy sources (RESs) is transforming traditional power grid networks, which require new approaches for managing decentralized energy production and consumption. Microgrids (MGs) provide a promising solution by enabling localized control over energy generation, storage, and distribution. This paper presents a novel reinforcement learning (RL)-based methodology for optimizing microgrid energy management. Specifically, we propose an RL agent that learns optimal energy trading and storage policies by leveraging historical data on energy production, consumption, and market prices. A digital twin (DT) is used to simulate the energy storage system dynamics, incorporating degradation factors to ensure a realistic emulation of the analysed setting. Our approach is validated through an experimental campaign using real-world data from a power grid located in the Italian territory. The results indicate that the proposed RL-based strategy outperforms rule-based methods and existing RL benchmarks, offering a robust solution for intelligent microgrid management. This paper has been accepted for presentation at IEEE IJCNN 2025. © 2025 IEEE. The final version will be published by IEEE. I Introduction The process of producing, distributing, and consuming energy in Europe and in the US has undergone a significant paradigm change in the past years due to the rapid diffusion of renewable energy sources (RESs). The classical approach included a few large-scale producers supplying power to private individuals and companies, distributors that could effectively connect entities across the grid, and consumers whose role was limited to stipulating contracts for the energy supply. However, the decentralized nature of RESs challenges the traditional role of centralized power plants in maintaining grid reliability. In response, a policy shift has emerged, prioritizing the integration of distributed energy resources (DERs) within the so-called smart grid structure, i.e., networks in which each node is allowed to produce, consume, store, and sell energy. This shift reflects a recognition of DERs’ potential to enhance grid resilience, promote energy independence, and seamlessly incorporate RESs into the system. However, to realize these benefits, it is crucial to develop optimized configurations and architectures for DERs, such as microgrids (MGs), which enable effective management and control of decentralized energy systems. An MG is a local power grid that can operate synchronously with the main electrical network and switch to a stand-alone ( islanded ) mode without repercussions. Commonly, they are composed of different energy sources, e.g., microturbines, fuel cells, and photovoltaic panels, supplied with battery energy storage systems (BESSs), and designed to handle flexible loads [ 1 ] . Even if MGs introduce a new level of flexibility, they also require tools for managing new elements to exploit the opportunities this setting provides. The need for intelligent management and control systems represents a significant challenge. Indeed, given the intrinsic combinatorial nature of the problem, designing a centralized controller located at the grid level to handle the production and consumption needs coming from each node, i.e., each MG, is no longer a viable option. Conversely, an alternative solution entails a distributed approach, where each MG defines its energy storage strategy. According to the literature, a widely adopted MG control scheme is the hierarchical, multi-layered framework, which effectively manages operations by dividing tasks across three levels. Primary control ensures local stability and real-time power sharing; secondary control addresses voltage and frequency deviations while refining system performance; tertiary control focuses on optimizing the economic efficiency of the BESS and coordinating power exchange with the main grid [ 2 ] . While the first two levels can be handled employing classical automation control strategies, for the third level, data-driven methods derived from the field of machine learning (ML), in particular, relying on deep reinforcement learning (DRL) techniques, have been proposed [ 3 ] . These methods have demonstrated the potential for learning optimal decision-making policies for MG control under conditions of uncertainty. Nonetheless, common available solutions optimize energy management based solely on costs and profits from MG interactions with the energy market, while neglecting the BESS degradation costs. Indeed, BESS replacement represents a substantial portion of MG’s operating expenses. Therefore, literature works that disregard such a cost might provide significantly suboptimal policies in a real-world scenario. Following this direction, our work aims to design a controller that optimizes the MG tertiary control level, i.e., to manage the BESS usage, avoid its excessive degradation, and reduce the expenses due to energy exchanges with the main grid. In the current work, we propose: • an RL-based [ 4 ] methodology to learn the optimal tertiary control strategy that uses historical information about energy consumption, power generation, market prices, and a digital twin (DT) to simulate the BESS; • an experimental campaign to validate the proposed approach, which uses real-world data from the Italian energy market, real appliances consumption, photovoltaic production, and temperature profiles, to build effective data-driven control policies through RL approaches. II Related Works In the past few years, a number of methods for controlling the tertiary level of MGs have been developed, especially in the field of energy storage management, spanning from RL techniques [ 3 ] to control theory methods [ 5 ] . Among the most relevant RL approaches, [ 6 ] employed the Deep Q-Network (DQN) algorithm [ 7 ] to manage the load of an MG under energy generation uncertainties. The authors used RL to learn the strategy of a distributor agent, which has to set the energy price of the main grid and the policies of each independent prosumer belonging to the MG. Authors of [ 8 ] employ DQN with real-world data of two simulation periods, i.e., a winter week and a summer week, corroborated by an exhaustive experimental campaign and hyperparameters tuning. DQN is also adopted in [ 9 ] , where the objective is the optimization of isolated MG management without considering market dynamics, and in [ 10 ] , where authors focus on finding the best strategy with a realistic peer-to-peer energy trading model. Other researchers adopted policy-based methods [ 11 ] , which allow for continuous actions and the development of safer, more robust strategies. In [ 12 ] , authors employ the Deep Deterministic Policy Gradient (DDPG) algorithm to tackle the mixed-integer linear programming of energy dispatch. Such a work considers multiple generation sources with the dual objective of satisfying power flow constraints and minimizing operational costs. In [ 13 ] , authors employ the Proximal Policy Optimization (PPO) algorithm to solve the optimal management of an MG comprising real-world data of forecasted generation sources, load consumptions, and energy prices. Regardless of the adopted RL methods, a common limitation in these approaches is related to the simplistic modeling of the BESS, which is often restricted solely to their electrical behavior. However, the BESS plays a main role within the MG due to its significant replacement cost. Therefore, not considering an accurate and realistic model for its calendar and usage degradation in the tertiary control strategies is a crucial weakness point of the above-mentioned methods. A partial solution to these shortcomings is offered by [ 14 ] , which addresses the problem of scheduling charge, discharge, and resting periods

Read the complete text