Generalising Battery Control in Net-Zero Buildings via Personalised Federated RL
Nicolas Cuadrado, Samuel Horváth, Martin Takáč · 2024
Read Generalising Battery Control in Net-Zero Buildings via Personalised Federated RL by Nicolas Cuadrado, Samuel Horváth, Martin Takáč (2024) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2412-20946.txt.
Description
A 2024 arXiv research paper by Nicolas Cuadrado, Samuel Horváth, Martin Takáč. This work studies the challenge of optimal energy management in building-based microgrids through a collaborative and privacy-preserving framework. We evaluated two common RL algorithms (PPO and TRPO) in different collaborative setups to manage distributed energy resources (DERs) efficiently. Using a customized version of the CityLearn environment and synthetically generated data, we simulate and design net-zero energy scenarios for microgrids composed of multiple buildings. Our approach emphasizes reducing energy costs and carbon emissions while ensuring privacy.
Catalog record
| CREATOR | Nicolas Cuadrado, Samuel Horváth, Martin Takáč |
|---|---|
| YEAR | 2024 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 33 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2412-20946
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Generalising Battery Control in Net-Zero Buildings via Personalised Federated RL Nicolas Cuadrado, Samuel Horváth, Martin Takáč Abstract This work studies the challenge of optimal energy management in building-based microgrids through a collaborative and privacy-preserving framework. We evaluated two common RL algorithms (PPO and TRPO) in different collaborative setups to manage distributed energy resources (DERs) efficiently. Using a customized version of the CityLearn environment and synthetically generated data, we simulate and design net-zero energy scenarios for microgrids composed of multiple buildings. Our approach emphasizes reducing energy costs and carbon emissions while ensuring privacy. Experimental results demonstrate that Federated TRPO is comparable with state-of-the-art federated RL methodologies without hyperparameter tuning. The proposed framework highlights the feasibility of collaborative learning for achieving optimal control policies in energy systems, advancing the goals of sustainable and efficient smart grids. Our code is accessible in this repo . † † affiliationnotice: Equal contribution 1 Introduction Global reviews of progress under the SDGs and the 2015 Paris Agreement still show the world drifting beyond the 1.5–2 °C ceiling, as confirmed by the latest IPCC assessment ( IPCC, 2023 ) . Because building operations consume about 30% of global final energy and produce roughly 26% of energy-related CO 2 emissions ( International Energy Agency, 2024 ) , cutting their footprint is pivotal. A recent analysis estimates that AI-enabled controls alone could trim building energy demand and emissions by 8–19% by 2050 ( Ding et al., 2024 ) . Machine learning (ML) is a technology that, when used correctly, can accelerate the actions needed to fight global warming. ML can support the transition from fossil fuels, accelerate renewable energy adoption, improve energy efficiency, and promote sustainable transportation, offering powerful tools to meet some of the most pressing climate goals ( Rolnick et al., 2019 ) . Rooftop solar and other on-site renewables introduce fast power swings that can create acute shortage events at the building level ( Zheng et al., 2024 ; Perera et al., 2020 ; Gernaat et al., 2021 ) . Modern Building Energy Management Systems (BEMS) mitigate these fluctuations by pairing local storage with predictive control and have been shown to cut energy use and costs across diverse facilities ( Michailidis et al., 2024 ) . Model-free reinforcement learning (RL) can learn such control policies directly from data, capturing the non-linear dynamics of distributed resources without a detailed physics model ( Barbalho et al., 2025 ) . Yet standard RL is sample-hungry and generalises poorly to unseen climates or occupancy patterns ( Bekal et al., 2025 ) . Privacy-preserving federated approaches overcome both issues by exchanging model updates rather than raw traces, protecting occupant data while pooling experience across sites ( Lee et al., 2021 ) . Recent demonstrations report cost and CO 2 reductions when scaling federated RL to dozens of heterogeneous buildings in multiple climates ( Sievers et al., 2025a ) . Privacy guarantees remove regulatory and organisational barriers to pooling knowledge across many buildings, making collaborative learning a practical path to scalable net-zero control ( Sievers et al., 2025b ) . This study explores building-scale net-zero microgrids (“a group of interconnected loads and distributed energy resources within clearly defined electrical boundaries that acts as a single controllable entity” ( Ton & Smith, 2012 ) ) as test-beds for scalable, privacy-preserving control. We generated scenarios where buildings are self-sustainable by design as long as the electrical storage is managed using the optimal policy. Using a custom microgrid scenario based on CityLearn ( Vázquez-Canteli et al., 2019 ) , we explored the generalization capabilities of RL agents and the challenges they faced even in simplified scenarios showing that naive policies are difficult to learn in a collaborative way. We improved the learning process using Federated Learning (FL), in this setup each building trains a local RL agent but only shares parameter updates, a practice that retains data privacy while exploiting the scalability and generalisation benefits identified in recent federated-learning surveys ( Kairouz et al., 2021 ; Lee et al., 2021 ) . Additionally, we included personalisation through split-learning ( Han et al., 2022 ) , further shrinking the generalisation gap that often appears when controllers are deployed in heterogeneous conditions or occupancy patterns ( Bekal et al., 2025 ) . We present a summary of our contributions next: • We introduce a custom environment based on CityLearn, which allows a better understanding of the learned policies and the impact of the reward function in the learning process. We also modified some characteristics that better represent the real-life dynamics. • We provide experimental assessment of traditional RL algorithms in our environment, testing generalization and optimality of the learned policies. • Our results demonstrate the generalization capabilities of FL and how reasonable considerations like personalization and split-learning ( Han et al., 2022 ) can reduce the optimality gap in the learned policies. • We provide a wrapper for the CityLearn environment that makes it compatible with TorchRL ( Bou et al., 2023 ) , improving the modularity and portability of our code. We also provided a version of the TRPO algorithm using TorchRL. 2 Related Work Federated Learning. FL trains a common model by letting each client update locally and then merging weights. The seminal FedAvg algorithm performs this aggregation on a central server ( McMahan et al., 2017 ) , whereas Gossip protocols average peer-to-peer without a server ( Koloskova et al., 2019 ) . Subsequent work addresses FedAvg’s limits under non-IID data and privacy leakage: split learning keeps part of each network private to enable personalisation ( Vepakomma et al., 2018 ) , and differential privacy injects calibrated noise into updates ( Geyer et al., 2017 ) . Studies on heterogeneous clients ( Diao et al., 2021 ) and generalisation theory ( Yuan et al., 2022 ) confirm that, with these extensions, FL can learn robust, privacy-aware controllers—making it well suited to smart-grid scenarios with diverse buildings. RL-based and Federated RL-based microgrids. Recent studies model microgrids as groups of houses with distinct loads, rooftop renewables, and storage devices, typically using the open-source CityLearn platform ( Vázquez-Canteli et al., 2019 ; Nweye et al., 2024 ) built on Gymnasium ( Towers et al., ) and real demand data ( Wilson et al., 2021 ) . Within CityLearn, ( Nweye et al., 2023 ) trained multi-agent Soft Actor-Critic controllers for battery systems to cut cost, carbon, and peak demand, while ( Nweye et al., 2022a ) showed that warm-starting RL with rule-based policies speeds convergence. Beyond public benchmarks, a model-based RL approach with constrained optimisation targets multi-energy hubs ( Ceusters et al., 2021 ) ; a federated RL scheme addresses free-riders and non-IID data ( Su et al., 2022 ) ; and hierarchical FRL embeds FedAvg in layered control architectures ( Cuadrado et al., 2023a ; Cuadrado et al., 2023b ) . Collectively, these works highlight both the promise and the outstanding challenges of RL and FRL for collaborative, net-zero microgrids. Despite steady progress, two gaps remain. First, many studies treat a monotonic rise in cumulative reward as proof of optimality, yet trivial local minima, such as always charging or doing nothing, can produce similar curves. Second, CityLearn challenge winners (2021–2023) optimised the highest-weighted terms and largely