The Challenges of Using Reinforcement Learning for Controlling Industrial Energy Systems
Tobias Lademann, Théo Vincent, Jan Peters, Matthias Weigold · 2026
Read The Challenges of Using Reinforcement Learning for Controlling Industrial Energy Systems by Tobias Lademann, Théo Vincent, Jan Peters, Matthias Weigold (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2605-31044.txt.
Introduction
Lademann and colleagues study deploying reinforcement learning in an industrial energy system, a thermal heating network. Most prior studies stay in simulation, while real use brings distinct difficulties. They formulate the task as a Markov decision process and analyze challenges along its formal structure, including partial observability, action space design, reward design, and the simulation-to-reality gap. The challenges are grounded in an existing site, where it runs steadily but shows a large performance gap versus simulation.
Catalog record
| CREATOR | Tobias Lademann, Théo Vincent, Jan Peters, Matthias Weigold |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 30784 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2605.31044
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
The Challenges of Using Reinforcement Learning for Controlling Industrial Energy Systems Tobias Lademann, Théo Vincent, Jan Peters, Matthias Weigold Abstract Reinforcement learning has shown promising results for optimizing the control of industrial energy systems, yet most existing studies remain limited to the application in simulation environments. We investigate the challenges of deploying reinforcement learning in a real-world industrial energy system, considering a thermal heating network as a use case. We formulate the task as a Markov Decision Process and systematically analyze the associated challenges along the structure of the formal description, including partial observability, action space design, reward design, and the simulation-to-reality gap. The challenges are grounded in an existing real-world deployment, where reinforcement learning achieves operational stability but shows a significant performance gap compared to simulation. 1 Introduction Industrial energy systems convert final energy such as electricity or natural gas into useful energy for production processes and building operation ( 28 ) . Given their significant share of industrial energy demand ( 22 ) , improving the efficiency and flexibility of these systems while reducing operational costs is a key objective in industry ( 28 ; 29 ) . Controlling industrial energy systems is a complex task due to interlinked thermal networks, multiple energy converters, the integration of energy storage systems, complex energy pricing models and strict requirements regarding security of supply ( 13 ; 28 ) . Conventional control strategies are typically designed based on expert knowledge and implemented as static rule-based approaches, combining hysteresis controllers for discrete decisions with PID controllers for continuous control tasks ( 6 ) . Although these approaches are generally robust, they are limited in their ability to define optimal control actions given a complex system behavior and conflicting optimization targets ( 26 ) . To address these limitations, reinforcement learning (RL) has gained increasing attention in the energy system domain ( 19 ) , as it promises several advantages compared to conventional controllers, such as the ability to handle complex, uncertain and dynamic environments ( 26 ) . Existing studies demonstrate the potential of RL to improve the control of industrial energy systems. However, the application of RL in real-world environments remains limited: Most existing studies evaluate RL-based control strategies exclusively in simulation ( 21 ) , neglecting the real-world challenges in an industrial setting ( 5 ) . This paper addresses this gap by investigating the challenges associated with the real-world deployment of RL for controlling an industrial energy system. It can be seen as an open call for RL methods to tackle application-relevant challenges, thereby guiding RL researchers in framing their research problems. Contributions. (1) We model the control task of a representative industrial use case as an RL problem, tailored to the characteristics and constraints of real-world industrial applications. (2) We formulate challenges within the RL problem formulation. (3) We concretize the identified challenges based on a real-world deployment of an RL control strategy in the considered use case. 2 Use case The ETA Research Factory serves as a real-world testbed for the deployment and evaluation of optimized control strategies under conditions representative of industrial energy systems. The energy system of the factory consists of three thermal networks, operating at different temperature levels ( 6 ) . For this work, we consider the Heating Network High Temperature , as it captures key characteristics of industrial energy systems, including multiple energy converters, thermal storage, and coupled energy carriers ( 6 ) . At the same time, it is the only thermal network that enables reproducible real-world experiments, as ambient influences are negligible ( 14 ) . The use case offers realistic industrial complexity and safety constraints while reducing experimental costs and risks compared to productive industrial environments. The considered system consists of multiple energy converters and thermal storage components that jointly supply thermal power to production processes and building operation as shown in Figure 1 . In particular, it includes two combined heat and power units and one condensing boiler , which convert gas into thermal energy. The combined heat and power units additionally generate electricity, which is either used within the factory or fed into the grid, depending on the factory demand. Both reduce operating costs, depending on current electricity market prices and feed-in tariffs. Thermal energy can be stored in an active storage system, allowing the temporal decoupling of energy production and consumption. Depending on the operating mode, the storage acts either as a thermal producer (discharge) or consumer (charge). This flexibility enables the operation of the combined heat and power units in periods of favorable electricity prices. The active storage is equipped with vacuum insulation and a layering system to reduce thermal and exergy losses. In addition, a passive buffer storage , which cannot be directly controlled, hydraulically decouples systems for thermal energy production from thermal energy consumption. On the demand side, the system supplies both production processes and building processes within the factory. All relevant energy flows, including thermal, electrical, and gas power, are measured for each energy converter and consumer. In addition, each storage is equipped with three temperature sensors located at the lower, middle, and upper parts of the storage. Figure 1 : Simplified scheme of the use case system based on 7 . The main objective of the system is to reliably supply thermal power to consumer systems at the required temperature level. The target temperature depends on the overall operating mode, with 70\text{\,}\mathrm{\SIUnitSymbolCelsius} in production mode and 55\text{\,}\mathrm{\SIUnitSymbolCelsius} in building mode. To ensure safe operation, the middle buffer storage temperature must remain within predefined limits, ranging from 45\text{\,}\mathrm{\SIUnitSymbolCelsius} to 85\text{\,}\mathrm{\SIUnitSymbolCelsius} in production mode and from 40\text{\,}\mathrm{\SIUnitSymbolCelsius} to 85\text{\,}\mathrm{\SIUnitSymbolCelsius} in building mode. RL has been applied to industrial energy systems in simulation and real-world settings, but deployment challenges remain largely unaddressed, as discussed in Appendix A . 3 Problem formulation This section provides a general formulation of the control of the considered industrial energy system as an RL problem. The control task is described with a Markov Decision Process (MDP) ( 27 ) , where an agent interacts with the real-world environment. Given a state s\in\mathcal{S} and an action a\in\mathcal{A} , the environment transitions to a successor state s^{\prime}\in\mathcal{S} according to the transition probability p(s^{\prime}\mid s,a) . A deterministic reward function of the state, action and next state is received after each transition. The agent learns a policy \pi(a\mid s) that maps system states to action distributions. State Space: A reasonable state space should include all observable system states and exogenous inputs that influence the system behavior and optimal control decisions. It should also include forecasts of relevant exogenous inputs, such as thermal demand of consumer systems, electrical demand of other factory processes, and energy prices, to enable optimal control decisions. The state space is constructed from available sensor measurements acquired and provided by the building automation system (e.g. storage temperatures or heat flow rates) and exogenous inputs provided through external data interfaces (e.g. market prices). The state space can include both continuous (e.g. temperature) and discrete (e.g. energy converter state on/off) values. Action Space: The RL agent controls the energy converters and active storage (see Figure 1 ). Each component is controlled by a boolean activation variable and a continuous setpoint, whose