Multi-Agent Deep Reinforcement Learning for Zonal Ancillary Market Coupling

Francesco Morri, Hélène Le Cadre, Pierre Gruet, Luce Brotcorne · 2025

Read Multi-Agent Deep Reinforcement Learning for Zonal Ancillary Market Coupling by Francesco Morri, Hélène Le Cadre, Pierre Gruet, Luce Brotcorne (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2505-03288.txt.

Introduction

Francesco Morri and three co-authors studied in 2025 how power grid zones should link their backup-service markets. The hosted text titles the work around game theory and multi-agent learning, which the catalog title shortens. They model the market as a game where zone leaders move first and one follower responds. Conditions for a stable outcome are derived, and the game turns out to have a potential structure that helps computation.

Two exact solvers are compared with a multi-agent learning method on real data from Germany and Austria. The learning method converged fastest but needed pretraining, and it gave lower market costs with more variable profit splits. Stronger links between zones cut costs for larger zones. Best-response solving was slowest of the three.

Catalog record

CREATORFrancesco Morri, Hélène Le Cadre, Pierre Gruet, Luce Brotcorne
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE55 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2505-03288

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Multi-Agent Deep Reinforcement Learning for Zonal Ancillary Market Coupling Francesco Morri, Hélène Le Cadre, Pierre Gruet, Luce Brotcorne Abstract We characterize zonal ancillary market coupling relying on noncooperative game theory. To that purpose, we formulate the ancillary market as a multi-leader single follower bilevel problem, that we subsequently cast as a generalized Nash game with side constraints and nonconvex feasibility sets. We determine conditions for equilibrium existence and show that the game has a generalized potential game structure. To compute market equilibrium, we rely on two exact approaches: an integrated optimization approach and Gauss-Seidel best-response, that we compare against multi-agent deep reinforcement learning. On real data from Germany and Austria, simulations indicate that multi-agent deep reinforcement learning achieves the smallest convergence rate but requires pretraining, while best-response is the slowest. On the economics side, multi-agent deep reinforcement learning results in smaller market costs compared to the exact methods, but at the cost of higher variability in the profit allocation among stakeholders. Further, stronger coupling between zones tends to reduce costs for larger zones. I Introduction A characteristic feature of electricity is the impossibility to store it in large proportions. Hence, it must be produced exactly when it is consumed, which raises a need for a real-time monitoring of the balance between production and consumption to guarantee the stability of the power grid. When imbalances occur, mechanisms are automatically triggered to restore the nominal frequency of the electric system: a change in power production is made by some production units, which are then said to provide ancillary services , in exchange of a monetary contribution. In Western Europe, many market zones are progressively switching from an automatic and mandatory involvement of qualified production units with an amount that is fixed by the system operator to a market mechanism based on daily auctions in which the market operator publishes a demand and then the producers submit offer curves that are aggregated until they meet the demand. From the point of view of a market actor, submitting a bid in such an auction leaves some space for strategic bidding, which requires some information about the market participants. This knowledge, together with the design of a relevant bidding strategy, should enable market actors to engage their production assets in the best way in ancillary services markets, which will then give them a fair idea of the revenue they can expect. Although these ancillary services markets exist in many countries, the bidding zones can be somehow associated: for instance the Austrian and German areas can share their bids, subject to some constraints and to the availability of the transmission network across both countries. Market participants might be willing to investigate the impact of this shared market structure and to see if it stabilizes their revenues over time. The main challenges to address are the characterization of the market equilibria and the understanding of their evolution when market zones get interconnected: from the perspective of the market operator, it is important to gain insight into the behaviors of the stakeholders and to gauge the opportunity to enforce market integration over multiple countries if this can, e.g., mitigate price peak events in one country thanks to interconnection. In this article, we use noncooperative game theory to model ancillary services markets. Market equilibria are computed relying on reformulations of the game and multi-agent deep reinforcement learning (MARL). Our flexible framework allows us to assess the consequences of the interconnection of market zones by dynamically changing the coupling factors. The approach can be generalized to any number of zones. Our main contributions are summarized as follows: • we formulate a multi-zone ancillary market model as a bilevel problem, that we subsequently reformulate as a generalized Nash game with generalized potential structure. We obtain theoretical guarantees for the existence of market equilibria; • we propose three algorithmic solutions for the market model: two exact approaches to compute an equilibrium, integrated optimization and Gauss-Seidel best-response, and a multi-agent deep reinforcement learning (MARL) simulation. The latter allows us to run a decentralized simulation of the energy market with strategic learning and non-strategic agents; • we compare the three algorithm performance on real data from the Germany-Austria market, computing social cost and fairness indicators. Finally, we test multiple coupling settings for the market, observing a decrease in the market operators’ costs with higher couplings. This paper is organized as follows: in Section I-A , we introduce the relevant literature; in Section II , we formulate the ancillary service market as a noncooperative game. In Section III , we derive conditions for existence of market equilibria, that can be subsequently computed relying on the algorithms introduced in Section IV . Finally, in Section V , we implement the algorithms on real data from Austria and Germany, that we compare numerically using performance measures and economic metrics, concluding in Section VI . I-A Related Work There are two main directions for research on ancillary (and zonal) market design: working on the design of the market itself, studying different formulations and their features [ 1 ] ; or working on the strategic bidding aspect [ 2 ] , focusing more on the point of view of the energy producers. In [ 3 ] , the author discusses possible ways of clearing the electricity market and selecting bids, proposing and analyzing different options for the market operator’s objective function, the settlement rules and the pricing of the products. Previous works are mostly theoretical, comparing the different possible market designs relying on noncooperative game theory. The authors of [ 4 ] instead, propose a market designed to account for the presence of wind power producers. On the same line is the two parts work of [ 5 , 6 ] , where the authors first discuss the design of an ancillary service focused on primary frequency response, and then simulate this design on different test cases. Another example exploring the impact of renewable based generators, is [ 7 ] , discussing bidding strategies for ancillary markets with participation of battery energy storage systems. We conclude the discussion on the formulation and features of ancillary markets with [ 8 ] , which explores the consequences and opportunities of implementing ancillary services at distribution grid level, considering frequency response, voltage regulation and other parameters. Regarding the computing of bidding strategies and solutions for ancillary and zonal markets, a first contribution can be found in [ 9 ] , where the authors model the market as a two-players game, analyzing the strategies of two producers going one against the other, and solving the game to obtain different market equilibria. In [ 10 ] a control based, decentralized solution is proposed to create ancillary services helping with power imbalances deriving from renewable sources. The authors in [ 11 ] formulate a zonal market as a two-stage stochastic problem, which they then recast as a mixed-integer bilinear program in order to solve it. This solution is computed from an external point of view, considering all producers at once. In [ 12 ] , the market is modeled as a Stackelberg game where the market operator is the leader and the producers are the followers; the solution is then obtained by recasting the problem

Read the complete text