Accelerating Reinforcement Learning for Wind Farm Control via Expert Demonstrations

Marcus Binder Nilsen 1, Julian Quick 1, Tuhfe Göçmen 1, Nikolay Dimitrov 1 and Pierre-Elouan Réthoré 1 · 2026

Read Accelerating Reinforcement Learning for Wind Farm Control via Expert Demonstrations by Marcus Binder Nilsen 1, Julian Quick 1, Tuhfe Göçmen 1, Nikolay Dimitrov 1 and Pierre-Elouan Réthoré 1 (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2604-22794.txt.

Introduction

Nilsen, Quick, Göçmen, Dimitrov, and Réthoré speed reinforcement learning for wind-farm control by pretraining from expert demonstrations generated by a steady-state wake optimizer inside a dynamic simulation. Behavior cloning initializes both actor and critic of a Soft Actor-Critic agent, removing the costly early phase where an untrained agent trails a greedy baseline. Fine-tuned agents converge within a quarter of a million steps and beat a lookup-table controller.

Catalog record

CREATORMarcus Binder Nilsen 1, Julian Quick 1, Tuhfe Göçmen 1, Nikolay Dimitrov 1 and Pierre-Elouan Réthoré 1
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE24325
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2604.22794

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Accelerating Reinforcement Learning for Wind Farm Control via Expert Demonstrations Marcus Binder Nilsen 1, Julian Quick 1, Tuhfe Göçmen 1, Nikolay Dimitrov 1 and Pierre-Elouan Réthoré 1 Abstract Reinforcement learning (RL) offers a promising approach for adaptive wind farm flow control, yet its practical deployment is hindered by slow training convergence and poor initial performance, factors that could translate to years of reduced power output if an untrained agent were deployed directly. This work investigates whether domain knowledge from steady-state wake models can accelerate RL training and improve initial controller performance. We propose a pretraining methodology in which expert demonstrations are generated by deploying a PyWake-based steady-state optimizer within a dynamic wake simulation (WindGym), then used to initialize both the actor and critic networks of a Soft Actor–Critic agent via behavior cloning. Experiments on a 2×2 wind farm show that pretraining eliminates the costly initial learning phase: while an untrained agent underperforms the greedy zero-yaw baseline by approximately 12%, pretraining raises initial performance to near-baseline levels. During online fine-tuning, all configurations converge within 250,000 environment steps to achieve similar performance, ultimately exceeding that of a lookup-table controller, which reaches approximately 7% power gain after 500,000 steps. 1 Introduction The control of wind farms has been recognized as one of the “grand challenges” of modern wind energy systems [ 18 , 11 ] , particularly focusing on the mitigation of wake effects, which lead to decreased power production and negatively impact the structural integrity of downstream turbines [ 9 ] . Wind farm (flow) control (WFFC) has the potential to increase energy capture, decrease structural loads, thereby reducing operational costs and extending turbine lifetime [ 11 ] . However, WFFC remains a challenging optimization problem, primarily due to the inherent complexity of aerodynamic interactions between turbines, high-dimensional decision spaces, and significant uncertainties in environmental conditions. One possible solution is reinforcement learning (RL), which has recently emerged as a promising approach for adaptive data-driven WFFC [ 1 , 7 ] . RL methods are particularly attractive because of their capability to continually learn directly from interaction with dynamic environments and adapt accordingly, making them well-suited for managing the stochastic nature of wind conditions. Nevertheless, the practical application of RL in WFFC faces significant challenges, notably slow training convergence and high computational demands resulting from RL’s inherent sample inefficiency, which often requires millions of interactions to converge [ 5 ] . Because RL relies on experience gathered through environmental interaction, it inevitably exhibits an early learning phase, with performance below that of existing control strategies. For a real wind farm, such a phase could mean years of reduced power output and substantial financial loss, making direct deployment infeasible. Fortunately, extensive domain knowledge already exists in the form of analytical wake models and steady-state optimization methods, which provide efficient and accurate benchmarks for optimal turbine operation under simplified conditions. This motivates the central research question of this work: • Can knowledge from steady-state models improve the initial and overall performance of RL agents for wind-farm control? Several works have explored how steady-state models can integrate within the RL framework. Zhao et al. [ 19 ] proposed a knowledge-assisted ’Deep Deterministic Policy Gradient’ framework that uses analytical models for safety constraints and reward shaping during online learning, though without pretraining the policy networks. Stanfel et al. [ 16 ] demonstrated that Q-tables trained offline, based on steady-state optimization using FLORIS [ 13 ] , can transfer to quasi-dynamic simulations. Most closely related, Bizon Monroc et al. [ 3 ] derived deterministic reference policies from lookup tables to initialize actor-critic networks. In contrast, we propose a pretraining methodology based on expert demonstrations collected during a dynamic wake simulation, training both the policy and value networks of a Soft Actor-Critic (SAC) agent. Furthermore, we analyze the effect of pretraining dataset size on learning performance—an aspect that has not been previously investigated. By transferring such domain knowledge, we aim to enhance the agent’s initial performance and ultimately improve the practicality of RL-based wind-farm control. In this work, a dynamic wake simulation serves as a surrogate for a real wind farm, allowing us to evaluate pretraining benefits in a controlled setting. While field deployment would raise additional questions—such as when to update the policy—one could envision a multi-stage pipeline in which the agent is first pretrained on a steady-state model, refined in a dynamic simulator, and finally fine-tuned on the real plant. 2 Methodology RL formalizes sequential decision-making as a Markov Decision Process (MDP), defined by the tuple (\mathcal{S},\mathcal{A},P,r) : • \mathcal{S} : the state space, represented as a vector of per-turbine features including local wind speed and direction, yaw angle, and their 25-step averages. All values are normalized to [-1,1] . • \mathcal{A} : the action space specifying yaw adjustments for each turbine. An action of 0 keeps the yaw unchanged, while \pm 1 corresponds to the maximum allowable change per timestep. The action is defined as: \gamma^{\prime}_{i}=\gamma_{i}+a_{t,i}\cdot\gamma_{\max} Here \gamma_{i} and \gamma^{\prime}_{i} is the current and previous yaw angle of turbine i , and \gamma_{\max} is the maximum allowed yaw step. This work uses a maximum yaw rate of 0.5\deg/\sec • P(s_{t+1}\mid s_{t},a_{t}) : the transition model, assumed deterministic as turbines perfectly follow control commands. • r(s_{t},a_{t}) : the reward, defined as the relative power gain compared to a baseline controller that aligns all turbines with the inflow: r_{t}=\left(\frac{\sum_{i=1}^{N_{\text{turb}}}P_{i,\text{agent}}(t)}{\sum_{i=1}^{N_{\text{turb}}}P_{i,\text{baseline}}(t)}-1\right) (1) At each time step t , the agent observes a state vector s_{t}\in\mathcal{S} and selects an action vector a_{t}\in\mathcal{A} , where subscript i denotes the per-turbine component (e.g., a_{t,i} ). The RL objective is to find a stochastic policy \pi_{\psi}(a\mid s) that maximizes the expected return: J(\psi)=\mathbb{E}_{\tau\sim\pi_{\psi}}\left[\sum_{t=0}^{T}\lambda^{t}r(s_{t},a_{t})\right], (2) where \tau=(s_{0},a_{0},r_{0}\ldots,s_{T},a_{T},r_{T}) is a trajectory induced by the policy and environment, and \lambda is used to denote the discount factor, set to 0.99 in this work. Actor–critic algorithms are commonly used to solve this optimization problem, where the actor \pi_{\psi}(a\mid s) samples actions and the critic V_{\phi}(s) estimates expected returns. This framework is particularly effective for continuous, high-dimensional control problems such as WFFC. This work uses the WindGym environment [ 4 ] and the Soft Actor–Critic (SAC) algorithm [ 8 ] for the policy, as described below. 2.1 WindGym Environment WindGym [ 4 ] is a custom, open-source RL environment for wind-farm flow control. The environment utilizes DYNAMIKS [ 15 , 17 ] , which is a collection of different fidelity simulation tools. This work uses dynamic wake meandering (DWM) [ 10 ] to simulate the transient evolution of the wakes in a turbulent inflow. The simulated wind farm consists of a 2\times 2 grid of DTU 10MW turbines [ 2 ] , each represented with their performance ( C_{P} ) and thrust ( C_{T} ) coefficients using PyWake [ 14 ] . Turbines are spaced 5D apart in both streamwise and spanwise directions, where D is the rotor diameter. Each WindGym episode corresponds to a fixed inflow scenario (wind speed, direction, turbulence intensity) and runs until the flow has passed through the farm 20 times. The simulation time step is 5 s, while the agent issues control commands every 10 s. During each episode, the policy accumulates experience by observing turbine-level measurements and adjusting yaw angles accordingly. 2.2 Soft

Read the complete text