On-Policy Reinforcement-Learning Control for Optimal Energy Sharing and Temperature Regulation in District Heating Systems
Xinyi Yi, Ioannis Lestas · 2025
Read On-Policy Reinforcement-Learning Control for Optimal Energy Sharing and Temperature Regulation in District Heating Systems by Xinyi Yi, Ioannis Lestas (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2509-16083.txt.
Introduction
District heating pipes serve whole blocks, but demand shifts and plant settings are often unknown in advance. Xinyi Yi and Ioannis Lestas design a learning controller that shares energy well across a district system while holding indoor heat steady. Their method updates its rules from live data, in step with the system's own flow.
They prove the controller settles at the best balance point for the system. It also settles on the best rule of a classic control type, which keeps short-term behavior smooth. Long runs in simulation back up the claims. The approach suits districts where exact models are hard to build.
Catalog record
| CREATOR | Xinyi Yi, Ioannis Lestas |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 52 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2509-16083
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
On-Policy Reinforcement-Learning Control for Optimal Energy Sharing and Temperature Regulation in District Heating Systems Xinyi Yi, Ioannis Lestas Abstract We address the problem of temperature regulation and optimal energy sharing in district heating systems (DHSs) where the demand and system parameters are unknown. We propose a temperature regulation scheme that employs data-driven on-policy updates that achieve these objectives. In particular, we show that the proposed control scheme converges to an optimal equilibrium point of the system, while also having guaranteed convergence to an optimal LQR control policy, thus providing good transient performance. The efficiency of our approach is also demonstrated through extensive simulations. I Introduction Carbon neutrality places the heating sector at the center of emission reduction. District Heating Systems (DHSs), common in China, Russia, and Europe, can decarbonize heating by integrating low-carbon sources such as waste heat and large-scale heat pumps [ 1 ] . This transition cuts fossil fuel use but creates complex generation profiles, requiring advanced DHS controllers for efficiency. One challenge in DHSs with multiple energy sources is that the optimal equilibrium points for energy sharing are not known a priori, as they depend on the generally unknown demand profile. Traditional regulators and model predictive controllers require disturbance forecasting, risking suboptimality, while optimization-feedback methods often assume unrealistically fast dynamics not present in DHSs. Furthermore, unknown system parameters, for example due to changing occupancy levels, complicates controller design. Our aim is to design controllers that achieve both of these objectives. This is achieved by combining transformations that achieve optimal equilibirum points and transient performance without a prior knowledge of demand, which have been used thus far in model-based schemes [ 2 ] , with data-based approaches that achieve asymptotically optimal transient performance. Research in the area of data-driven Linear Quadratic Regulator (LQR) has evolved into two main approaches: indirect and direct controllers. Unlike indirect controllers, which require system identification, direct controllers circumvent this requirement. Recent advances in direct methods include reinforcement learning (RL), policy gradient optimization (PGO), and certainty-equivalence (CE). With greedy policy updates, RL often converges faster in deterministic settings. This work employs online RL for its model-free, real-time adaptability, while extensions to stochastic settings with disturbances and uncertainties are left for future work. The papers [ 3 , 4 ] introduced adaptive online RL controllers, while [ 5 ] and [ 6 ] advanced offline RL strategies, with [ 6 ] specifically addressing Gaussian disturbances. These works employ Q-function estimation to ensure policy convergence. Recent data-driven temperature controller mainly focused on optimizing building regulation [ 7 ] . Temperature regulation differs substantially between individual buildings and DHSs in two key aspects: 1) Modeling complexity—building models vary widely to capture unique structures, whereas DHSs rely on standardized components such as pipes and heat exchangers; 2) Energy sharing—building control balances cost and comfort for individuals, while DHSs emphasize efficient energy distribution across multiple consumers and generators. The contribution of this paper is the introduction of an on-policy RL temperature regulation scheme for DHSs that achieves both optimal energy sharing with minimal steady-state temperature deviations and asymptotically optimal transient performance. The proposed method adapts online with guaranteed convergence to an optimal LQR controller and equilibrium point, independent of prior knowledge of disturbances or system models. Moreover, we prove that during online adaptation, the controller ensures stability and convergence to the optimal equilibrium point. The paper is structured as follows. The steady-state optimal energy sharing and temperature regulation problem are formulated in Section II. A data-driven RL regulator with asymptotically optimal performance is presented in Section III. Simulations validating the scheme are given in Section IV, and key findings are summarized in Section V. II Preliminaries This section provides an overview of the temperature dynamics and the economic dispatch problem that needs to be solved at steady-state within a DHSs. II-A Temperature dynamics and economic dispatch in DHSs Assumption 1 . (Properties of DHSs) [ 2 ] 1. Heat conduction is not considered in the model. 2. Flow rates are assumed to be constant and are externally controlled by users. The paper does not address the control of mass flow rates. 3. The water’s density \rho and specific heat capacity c_{\text{s.h.}} are uniform and constant throughout the system. 4. The network is designed to be leak-free and experiences no hydraulic losses. 5. The supply and return networks are symmetric. II-A 1 Temperature dynamics A DHS consists of supply and return networks with HXs, where consumers extract heat and return cooled water, and producers reheat and recirculate it as shown in Fig. 1 . For HX i , the supply and return temperatures are T_{i}^{S} and T_{i}^{R} , the flow rates are q_{i} and q_{i}^{E} , and all temperatures denote deviations from steady-state values under a nominal demand profile. Fig. 1: An example of a DHS, where HX 1 supplies heat for both HX 3 and HX 4, while HX2 only supplies heat for HX 4. The table gives the entries of the corresponding adjacency matrix \boldsymbol{L_{q}} in ( 3 ). The temperature dynamics of HX i , as modelled in [ 8 ] under Assumption 1 , are given as follows, \rho c_{\text{s.h.}}V_{i}\dot{T}_{i}^{\text{out}}=\rho c_{\text{s.h.}}q_{i}(T_{i}^{\text{in}}-T_{i}^{\text{out}})+P_{i}+P_{i}^{\text{dis}}, (1) where V_{i} is the volume of HX i , P_{i} is the controllable thermal power of HX i , T_{i}^{\text{in}} and T_{i}^{\text{out}} are the inlet and outlet temperatures of HX i , respectively. If HX i functions as a producer, P_{i} denotes the heat production power, T_{i}^{\text{in}}=T_{i}^{R} and T_{i}^{\text{out}}=T_{i}^{S} . If HX i serves as a consumer, P_{i} denotes the adjustable heat load power, T_{i}^{\text{in}}=T_{i}^{S} and T_{i}^{\text{out}}=T_{i}^{R} . HX i ’s inlet T_{i}^{in} is determined by the energy balance equation given as follows: 0=\sum_{j\in\mathcal{O}^{HX}_{k}\cap\mathcal{I}^{HX}_{i}}q_{j}^{E}T_{k}^{\text{out}}-T_{i}^{in}\sum_{j\in\mathcal{O}^{HX}_{k}\cap\mathcal{I}^{HX}_{i}}q_{j}^{E}, (2) where \mathcal{O}^{HX}_{i} and \mathcal{I}^{HX}_{i} denote the sets of pipelines associated with the outlet and inlet of HX i , respectively. By calculating the inlet temperature T_{i}^{in} from the outlet temperature T_{i}^{out} via equation ( 2 ), the temperature dynamics ( 1 ) can be described by the outlet temperature vector \boldsymbol{T} as: \boldsymbol{V\dot{T}}=\boldsymbol{L_{q}T+P+P^{dis}}, (3) where \boldsymbol{P} is the vector of controllable load and production of HXs(i.e. it is the controlled variable). \boldsymbol{P^{dis}} denotes constant disturbances representing slow demand changes [ 8 ] . Both \boldsymbol{P} and \boldsymbol{P^{dis}} are weighted by \frac{1}{\rho c_{\text{s.h.}}} . \boldsymbol{V} is the diagonal matrix of HXs’ volumes. \boldsymbol{L_{q}} is the adjacency matrix of flow rates between HXs, defined by: \boldsymbol{L_{q}}(i,j)=\begin{cases}q^{E}_{k},&(i\in\mathcal{G}\text{ and }j\in\mathcal{L})or(i\in\mathcal{L}\text{ and }j\in\mathcal{G}),\\ -q_{i}&i=j\text{ and }i\in\mathcal{G}\cup\mathcal{L},\\ 0,&\text{otherwise},\end{cases} (4) where \mathcal{L} and \mathcal{G} are the sets of consumer and producer HXs, respectively. q^{E}_{k} is the flow rate through pipe k , connecting HX i and HX j . II-A 2 Economic dispatch The economic dispatch for DHSs ( 3 ) at steady-state, is modeled as two optimization problems. E1: \displaystyle\ \min_{\boldsymbol{P}\in\mathbb{R},\boldsymbol{T}\in\mathbb{R}}\Phi_{1}=\sum\frac{1}{2}f_{i}(P_{i})^{2}=\frac{1}{2}\boldsymbol{P}^{\top}\boldsymbol{F}\boldsymbol{P}, (5a) \displaystyle\ \text{s.t.}\ \boldsymbol{L_{q}T}=-(\boldsymbol{P^{dis}}+\boldsymbol{P}), (5b) where \Phi_{1} denotes the cost of production and load adjustments, with \boldsymbol{F}=\mathrm{diag}\{f_{i}\} and f_{i}>0 the cost coefficient at node i . Since \boldsymbol{F} is positive definite, \Phi_{1} is quadratic and strictly convex. The linear constraints in ( 5b ) make E1 a convex optimization problem with a unique solution \boldsymbol{P^{*}} . As \boldsymbol{L_{q}} is a Kirchhoff matrix [ 9 ] , an optimal \boldsymbol{T^{*}} satisfies \boldsymbol{T^{*}}=-\boldsymbol{L_{q}^{\dagger}}(\boldsymbol{\bar{P}^{dis}}+\boldsymbol{P^{*}})+z\boldsymbol{1},\quad z\in\mathbb{R} , where z\boldsymbol{1} lies in the nullspace of \boldsymbol{L_{q}} and \boldsymbol{L_{q}^{\dagger}} is