HMPC-assisted Adversarial Inverse Reinforcement Learning for Smart Home Energy Management

Jiadong He, Liang Yu, Zhiqiang Chen, Dawei Qiu, Dong Yue, Goran Strbac, Meng Zhang, Yujian Ye, Yi Wang · 2025

Read HMPC-assisted Adversarial Inverse Reinforcement Learning for Smart Home Energy Management by Jiadong He, Liang Yu, Zhiqiang Chen, Dawei Qiu, Dong Yue, Goran Strbac, Meng Zhang, Yujian Ye, Yi Wang (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2506-00898.txt.

Introduction

Jiadong He and co-authors teach a smart home energy manager by having it imitate optimal decisions, rather than by spelling out goals, in a 2025 paper. Their pipeline first uses a hierarchical predictive controller, helped by a neural network, to generate expert choices from past data. An adversarial learning module then trains a fresh policy to match those choices.

A discriminator learns to tell expert moves from the learner's, while the learner improves until the two blur. The method needs no explicit model of home heating, no forecasts of uncertain inputs, and no hand-built reward. On real-world traces, the authors report strong performance with modest data.

Catalog record

CREATORJiadong He, Liang Yu, Zhiqiang Chen, Dawei Qiu, Dong Yue, Goran Strbac, Meng Zhang, Yujian Ye, Yi Wang
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE19 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2506-00898

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

HMPC-assisted Adversarial Inverse Reinforcement Learning for Smart Home Energy Management Jiadong He, Liang Yu, Zhiqiang Chen, Dawei Qiu, Dong Yue, Goran Strbac, Meng Zhang, Yujian Ye, Yi Wang Abstract This letter proposes an Adversarial Inverse Reinforcement Learning (AIRL)-based energy management method for a smart home, which incorporates an implicit thermal dynamics model. In the proposed method, historical optimal decisions are first generated using a neural network-assisted Hierarchical Model Predictive Control (HMPC) framework. These decisions are then used as expert demonstrations in the AIRL module, which aims to train a discriminator to distinguish expert demonstrations from transitions generated by a reinforcement learning agent policy, while simultaneously updating the agent policy that can produce transitions to confuse the discriminator. The proposed HMPC-AIRL method eliminates the need for explicit thermal dynamics models, prior or predictive knowledge of uncertain parameters, or manually designed reward functions. Simulation results based on real-world traces demonstrate the effectiveness and data efficiency of the proposed method. I Introduction Building sector consumed approximately 30% of global final energy and contributed to 26% of energy-related carbon emissions in 2022. Among these, residential buildings represent a substantial share, accounting for nearly 60% of total building energy consumption [ 1 ] . One of the most effective strategies to reduce energy use in residential buildings is the development of smart homes [ 2 ] . In a smart home, the home energy management system (HEMS) serves as a core component, collecting operational data from devices (including photovoltaic (PV) panels, energy storage systems (ESSs), heating, ventilation, and air conditioning (HVAC) systems, and various household appliances) and making intelligent decisions, leading to lower energy consumption/cost, occupant discomfort, and renewable energy waste. Thus, it is of great importance to develop advanced smart home energy management (SHEM) methods. Many SHEM approaches have been developed in existing works, which can be classified into model-based and data-driven approaches. Model-based approaches rely on explicit building thermal dynamics models and require prior or predictive information about uncertain parameters. However, due to complex factors (e.g., building structure, materials, and surrounding environment), it is challenging to obtain an accurate and explicit thermal dynamics model. Moreover, parameter prediction inevitably introduces errors, which can affect the performance of such approaches. To eliminate the above-mentioned reliance and requirements, deep reinforcement learning (DRL)-based SHEM approaches have been adopted [ 2 , 3 ] . Although such DRL-based approaches are effective, they require manually designed reward functions, which introduces complexity in selecting appropriate reward structures and parameters. To avoid such manual burden, Dey et al. employed maximum entropy inverse reinforcement learning (MaxEnt IRL) with rule-based demonstrations for building comfort control [ 4 ] . However, the MaxEnt IRL-based control method faces the issue of reward ambiguity (i.e., multiple reward functions may induce the same optimal policy), which can affect the robustness of policy learning [ 5 ] . Moreover, rule-based expert demonstrations typically suffer from low quality. To address the above limitations, this letter proposes a novel SHEM method based on historical optimization decision-driven Adversarial Inverse Reinforcement Learning (AIRL) under an implicit thermal dynamics model. Specifically, a neural network-assisted hierarchical model predictive control (HMPC) framework is employed to generate high-quality historical optimization decisions. These decisions are then used as expert demonstrations for the AIRL module, which is designed to train a discriminator to distinguish expert demonstrations from transitions of the HEMS agent policy, while simultaneously updating the agent’s policy to generate transitions that can confuse the discriminator. Simulation results demonstrate the effectiveness and data efficiency of the proposed method. To the best of our knowledge, this is the first work that applies HMPC-AIRL to SHEM. II Problem Formulation Fig. 1 : The considered smart home energy system in this letter This letter considers a smart home energy system as shown in Fig. 1 , where PV, ESS, HVAC, non-shiftable loads, and an HEMS could be identified. The purpose of the HEMS is to minimize the long-term system energy cost while maintaining thermal comfort for occupants by controlling HVAC and ESS jointly. Suppose that the HEMS operates in slotted time, i.e., t\in[1,M] , where M denotes the total number of time slots. Then, a long-term multi-objective optimization problem related to the HMES can be formulated by \displaystyle\min_{e_{t},h_{t}}\mathop{\lim\sup}\limits_{M\to\infty}\frac{1}{M}\mathbb{E}\Big\{\sum\nolimits_{t=1}^{M}(X_{1,t}+X_{2,t})\Big\} (1) \displaystyle\min_{e_{t},h_{t}}\mathop{\lim\sup}\limits_{M\to\infty}\frac{1}{M}\mathbb{E}\Big\{\sum_{t=1}^{M}X_{3,t}\Big\} (2) \displaystyle\text{s.t.}~E^{\min}\leq E_{t}\leq E^{\max}, (3) \displaystyle-d^{\max}\leq e_{t}\leq c^{\max}, (4) \displaystyle E_{t+1}=\begin{cases}E_{t}+\eta_{c}e_{t}\Delta t,&\text{if}~e_{t}>0\\ E_{t}+\frac{e_{t}}{\eta_{d}}\Delta t,&\text{else},\end{cases} (5) \displaystyle 0\leq h_{t}\leq h^{\max}, (6) \displaystyle T^{\text{in}}_{t+1}=\mathcal{F}_{T}(T^{\text{in}}_{t},T^{\text{out}}_{t},h_{t},\epsilon_{t}), (7) \displaystyle g_{t}+p_{t}=h_{t}+e_{t}+l_{t}, (8) where the expectation operator \mathbb{E} is imposed on uncertain system parameters, e.g., electricity prices u^{+}/u^{-} , PV generation p_{t} , outdoor temperature T^{\text{out}}_{t} , and non-shiftable load l_{t} . X_{1,t}=u^{+}g_{t}\Delta t|_{g_{t}>0}+u^{-}g_{t}\Delta t|_{g_{t}\leq 0} , X_{2,t}=k_{e}|e_{t}| , X_{3,t}=[T^{\text{in}}_{t}-T^{\text{upp},\text{in}}]^{+}+[T^{\text{low},\text{in}}-T^{\text{in}}_{t}]^{+} denote the energy transaction cost, ESS aging cost, and temperature deviation, respectively. Here, u^{+}_{t} , u^{-}_{t} , and g_{t} denote the buying price, the selling price, and the transaction power at slot t , respectively. \Delta t denotes the length of time slot t , k_{e} is an ESS depreciation coefficient, and e_{t} denotes the ESS power at slot t . Specifically, e_{t}>0 denotes the charging power and e_{t}<0 denotes the discharging power. The operator [\cdot]^{+}=\max\{\cdot,0\} . T^{\text{in}}_{t} is the indoor temperature at slot t ; T^{\text{low},\text{in}} and T^{\text{upp},\text{in}} are the lower and upper bounds of comfort temperature, respectively. In ( 3 ), the range of energy level in the ESS is provided, where the minimum and maximum ESS energy level is E^{\min} and E^{\max} , respectively. E_{t} denotes the ESS energy level at slot t . In ( 4 ), the allowable charging/discharging power is provided, where c^{\max} and d^{\max} are the maximum charging power and discharging power, respectively. ( 5 ) describes the ESS dynamics, where \eta_{c} signifies the charging coefficient and \eta_{d} denotes discharging coefficient. ( 6 ) describes the allowable range of HVAC input power h_{t} and its maximum value is h^{\max} . ( 7 ) describes an implicit thermal dynamics model \mathcal{F}_{T} for the smart home, which is the function of indoor temperature T^{\text{in}}_{t} , outdoor temperature T^{\text{out}}_{t} , HVAC input power h_{t} , and random thermal disturbance \epsilon_{t} . Without loss of generality, we assume that this implicit thermal dynamics model can be approximated by neural networks, similar to [ 6 ] . ( 8 ) denotes the power balance, where p_{t} is the PV generation, and l_{t} is the total non-shiftable loads. Fig. 2 : Architecture of the proposed SHEM method III The Proposed Energy Management Method In this letter, we propose a SHEM method based on historical optimization decision-driven AIRL. In order to obtain historical optimization decisions in the presence of an implicit thermal dynamics model, a neural network-assisted HMPC is adopted. Then, such decisions are taken as expert demonstrations in the AIRL module, which aims to train a discriminator to distinguish expert demonstrations from transitions of the HEMS agent policy, and to update the HEMS agent’s policy that confuses the discriminator. After finishing the above training process, a robust policy is obtained for online SHEM. The proposed HMPC-AIRL method has strong universality compared with existing model-based approaches and DRL-based approaches, since it does not require an explicit thermal dynamics model, prior or predicted information of uncertain parameters, or manually designed reward function. The architecture of the proposed method is illustrated by Fig. 2 , where three blocks can

Read the complete text