A Reinforcement Learning-based Transmission Expansion Framework Considering Strategic Bidding in Electricity Markets

Tomonari Kanazawa, Hikaru Hoshino, Eiko Furutani · 2026

Read A Reinforcement Learning-based Transmission Expansion Framework Considering Strategic Bidding in Electricity Markets by Tomonari Kanazawa, Hikaru Hoshino, Eiko Furutani (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2602-19421.txt.

Introduction

Planning new transmission lines shapes, and is shaped by, how power generators bid in electricity markets. Tomonari Kanazawa and two co-authors propose a reinforcement learning framework that co-optimizes grid investment decisions and generator bidding plans in one unified training process. A multi-agent market model provides the setting, with a new design policy layer joining continuous and discrete growth choices to bidding plans.

Through repeated interaction between clearing the market and designing investment, the framework captures how each side influences the other. Case studies on the IEEE 30-bus network test the approach as a proof of concept. Readers get a design that treats grid expansion and strategic bidding as one coupled problem, not two separate ones.

Catalog record

CREATORTomonari Kanazawa, Hikaru Hoshino, Eiko Furutani
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE24 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2602-19421

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

A Reinforcement Learning-based Transmission Expansion Framework Considering Strategic Bidding in Electricity Markets Tomonari Kanazawa, Hikaru Hoshino, Eiko Furutani Abstract Transmission expansion planning in electricity markets is tightly coupled with the strategic bidding behaviors of generation companies. This paper proposes a Reinforcement Learning (RL)-based co-optimization framework that simultaneously learns transmission investment decisions and generator bidding strategies within a unified training process. Based on a multi-agent RL framework for market simulation, the proposed method newly introduces a design policy layer that jointly optimizes continuous/discrete transmission expansion decisions together with strategic bidding policies. Through iterative interaction between market clearing and investment design, the framework effectively captures their mutual influence and achieves consistent co-optimization of expansion and bidding decisions. Case studies on the IEEE 30-bus system are provided for proof-of-concept validation of the proposed co-optimization framework. I Introduction The liberalization of electricity markets has led to increasing decentralization in investment and operation decisions among generators and transmission operators [ 7 ] . Generation is controlled by private Generation Companies (GENCOs), whose decisions are driven by their aim to maximize their profits. Transmission development and system (and market) operation are planned by independent entities, System Operators (SOs) and Market Operators (MOs). Under such competitive environments, transmission system operators are required to conduct proactive planning [ 12 ] that accounts for uncertainties in future generation and market outcomes. Traditional multi-level optimization models have addressed transmission expansion planning together with market operation (see, e.g., [ 13 , 6 ] ), yet these models typically assume rational market equilibrium and have limited flexibility to fully represent strategic bidding behaviors and price volatility. Recent studies have applied deep Reinforcement Learning (RL) to planning of power systems in uncertain and decentralized environments. As reviewed in [ 11 ] , existing studies have used RL in two distinct ways: operational planning and expansion planning. The first direction uses RL to simulate operation and market behavior, where agents representing GENCOs learn strategic bidding policies that reproduce realistic price formation and market equilibria (see, e.g., [ 17 , 8 , 5 ] ). These models capture short-term interactions among agents and enable analysis of market efficiency, price volatility, and regulatory impacts. The second direction uses RL or related learning techniques for expansion planning and investment, such as transmission expansion problems [ 10 , 16 , 4 ] . In these studies, RL serves as a high-level optimizer that explores design or investment decisions over long-term horizons, often neglecting operational details into simplified cost/reward functions. However, these two research lines have largely developed in parallel. Operational optimization or market-simulation approaches using RL provide realistic operational dynamics but typically assume fixed network configurations, while RL-based expansion planning approaches ignore how strategic bidding behaviors affect investment outcomes. In practice, transmission expansion planning is inherently coupled with market behavior: network upgrades influence locational prices and dispatch patterns, and conversely, bidding strategies and congestion patterns determine the economic value of transmission expansions. To bridge this research gap, this paper proposes an RL-based co-optimization framework that incorporates strategic bidding and transmission expansion within a unified learning process. Based on the multi-agent deep RL framework for market simulations as reported in [ 5 ] , we extend it to jointly optimize transmission investment decision as design variables that co-evolve with agent policies. This type of RL-based co-optimization of system design and its operational policy has been proposed primariry in the robotics community (see, e.g., [ 14 , 3 ] ) and recently applied to energy systems applications [ 2 , 9 ] . To the best of our knowledge, this paper is the first to propose an RL-based co-optimization framework in the context of transmission expansion planning. Through case studies on the IEEE 30-bus system, we demonstrate that the proposed framework captures the interaction between market dynamics and investment decisions, yielding cost-efficient expansion plans that are consistent with strategic market behavior. The rest of this paper is organized as follows. In Sec. II , the formulation of the transmission expansion planning problem studied in this paper is introduced. In Sec. III , the proposed RL-based co-optimization framework is presented. In Sec. IV , numerical studies on the IEEE 30-bus system are described followed by conclusion in Sec. V . II Problem Formulation This study considers a two-level decision process that includes short-term market operation and long-term transmission investment. The formulation below defines the underlying market model and investment variables that form the basis of the proposed RL-based co-optimization framework. II-A Market Model We consider a day-ahead electricity market consisting of N_{g} generating units (agents) participating through strategic bids. Each agent i aims to maximize its own profit by deciding a bid price \lambda^{\mathrm{bid}}_{i}(t) at each time step t based on observed states such as local demand and past market outcomes (the detailed modeling of bidding behavior is described in Sec. III-A ). Given all submitted bids, the market operator performs DC Optimal Power Flow (DC-OPF) to clear the market under power balance and line-flow constraints at each time step t : \displaystyle\min_{\{P_{i},\theta_{n}\}}~ \displaystyle\sum_{i=1}^{N_{g}}\lambda_{i}^{\mathrm{bid}}(t)P_{i}(t) (1) s.t. \displaystyle\sum_{i\in\mathcal{G}_{n}}P_{i}(t)-D_{n}(t) \displaystyle~~=\sum_{(n,m)\in\mathcal{L}}B_{nm}(\theta_{n}(t)-\theta_{m}(t)),~~\forall n\in\mathcal{N}_{\mathrm{b}}, (2) \displaystyle|B_{nm}(\theta_{n}(t)-\theta_{m}(t))|\leq L_{nm},~~\forall(n,m)\in\mathcal{L}, (3) \displaystyle 0\leq P_{i}(t)\leq P_{i}^{\max},~~\forall i\in\{1,\ldots,N_{g}\}, (4) where \mathcal{N}_{\mathrm{b}} , \mathcal{L} , and \mathcal{G}_{n} are the sets of nodes (buses), lines, and generators connected to bus n\in\mathcal{N}_{\mathrm{b}} , respectively; D_{n} is the demand at bus n ; \theta_{n} is the voltage angle at bus n ; B_{nm} is the line susceptance; L_{nm} is the transmission capacity between buses n and m ; and P_{i}^{\max} is the maximum generation capacity of unit i . Solving ( 1 )–( 4 ) yields the cleared generation P_{i}^{\mathrm{cleared}}(t) and nodal prices \lambda_{i}^{\mathrm{cleared}}(t) , which reflect network congestion and marginal operating costs. The total operational system cost at time t is then defined as C_{\mathrm{oper}}(t)=\sum_{i=1}^{N_{g}}\lambda_{i}^{\mathrm{cleared}}(t)P_{i}^{\mathrm{cleared}}(t), (5) representing the total payment to generators in the cleared market. Although a single-block bidding structure is assumed here for simplicity, the formulation can be readily extended to multi-block or piecewise-linear bidding schemes, as commonly adopted in day-ahead markets. II-B Transmission Expansion In this study, transmission expansion planning is considered for a single target year rather than a multi-stage horizon. That is, investment decisions are evaluated under representative operating conditions of a given year, and the resulting network configuration is assumed to remain fixed during the market simulations. Let each transmission line \ell\in\mathcal{L} have a base capacity L_{\ell}^{\mathrm{base}} , while in Sec. II-A the pair (n,m) refers to the sending and receiving buses of each line. We consider two types of formulations of a continuous expansion model that adjusts line capacities and a discrete siting model that determines whether to upgrade candidate lines: \displaystyle\Delta L_{\ell} \displaystyle\in\mathbb{R}_{+},\quad \displaystyle\text{(continuous capacity expansion)}, (6) \displaystyle z_{\ell} \displaystyle\in\{0,1\},\quad \displaystyle\text{(discrete expansion decision)}. (7) The effective line capacity is then expressed as L_{\ell}=L_{\ell}^{\mathrm{base}}+z_{\ell}\,\Delta L_{\ell}, (8) where either z_{\ell} or \Delta L_{\ell} is fixed depending on the case study in Sec. IV . The investment cost for transmission expansion is modeled as: C_{\mathrm{exp}}=\sum_{\ell\in\mathcal{L}}c_{\ell}z_{\ell}\Delta L_{\ell}, (9) where c_{\ell} is the expantion cost per unit capacity of the line \ell per year. The overall optimization objective is to minimize the total cost including the expansion cost, defined as J=W_{\mathrm{anu}}\sum_{t=1}^{T}C_{\mathrm{oper}}(t)+C_{\mathrm{exp}}, (10) where W_{\mathrm{anu}} stands for the annualization factor depending on the total time steps T . Such formulations are consistent

Read the complete text