Higher-Order Action Regularization in Deep Reinforcement Learning: From Continuous Control to Building Energy Management

Faizan Ahmed, Aniket Dixit, James Brusey · 2026

Read Higher-Order Action Regularization in Deep Reinforcement Learning: From Continuous Control to Building Energy Management by Faizan Ahmed, Aniket Dixit, James Brusey (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2601-02061.txt.

Introduction

Reinforcement learning controllers often jitter, issuing rapid-fire action changes that waste energy and wear out equipment. Faizan Ahmed, Aniket Dixit, and James Brusey investigate smoothing those actions by penalizing their higher-order derivatives, from simple changes up through jerk, the third derivative.

Across four continuous control benchmarks, jerk penalties gave the best smoothness while keeping task performance competitive. The team then carried the idea to building heating and cooling control, where smooth policies cut equipment switching by sixty percent, a direct operational saving. The work argues higher-order action regularization is a practical bridge between reinforcement learning's optimization goals and the physical constraints of energy-critical deployments.

Catalog record

CREATORFaizan Ahmed, Aniket Dixit, James Brusey
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE13 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2601-02061

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Higher-Order Action Regularization in Deep Reinforcement Learning: From Continuous Control to Building Energy Management Faizan Ahmed, Aniket Dixit, James Brusey Abstract Deep reinforcement learning agents often exhibit erratic, high-frequency control behaviors that hinder real-world deployment due to excessive energy consumption and mechanical wear. We systematically investigate action smoothness regularization through higher-order derivative penalties, progressing from theoretical understanding in continuous control benchmarks to practical validation in building energy management. Our comprehensive evaluation across four continuous control environments demonstrates that third-order derivative penalties (jerk minimization) consistently achieve superior smoothness while maintaining competitive performance. We extend these findings to HVAC control systems where smooth policies reduce equipment switching by 60%, translating to significant operational benefits. Our work establishes higher-order action regularization as an effective bridge between RL optimization and operational constraints in energy-critical applications. \workshoptitle UrbanAI: Harnessing Artificial Intelligence for Smart Cities 1 Introduction Reinforcement learning has demonstrated remarkable success in complex decision-making tasks; however, its deployment in real-world systems reveals a critical limitation: learned policies frequently exhibit rapid, erratic control behaviors that optimize immediate rewards while incurring significant operational costs. This phenomenon manifests across diverse domains—from robotic manipulators experiencing mechanical stress due to abrupt movements 7 to building HVAC systems consuming excess energy through frequent equipment cycling 11 . The core challenge lies in the mismatch between RL’s objective function and real-world constraints. Standard RL formulations maximize cumulative rewards without considering the derivative structure of action sequences. While this approach succeeds in simulation environments, physical systems operate under thermodynamic, mechanical, and economic constraints that penalize high-frequency control variations. We address this challenge through three main contributions: (1) We introduce a third-order derivative penalty and compare first-order and second-order derivative penalties across control benchmarks. (2) We demonstrate that jerk minimization (third-order derivatives) provides optimal smoothness-performance trade-offs. (3) We validate our approach on the HVAC system, quantifying equipment longevity improvements and energy efficiency gains. Our work establishes that incorporating higher-order action regularization enables RL deployment in energy-critical applications where smoothness directly impacts operational efficiency and system lifetime. 2 Related Work Smooth control in RL has been addressed through regularization approaches that add penalty terms to reward functions. Early robotics work penalized first-order action differences 9 , while recent studies have investigated second-order penalties in manipulation 7 . However, systematic comparison across derivative orders remains limited, and higher-order penalties have received minimal attention despite their theoretical importance for mechanical systems. Quantifying smoothness in continuous control varies across domains. Robotics commonly uses jerk (third derivative of position) due to its relationship with mechanical stress and energy consumption 6 . Control theory emphasizes total variation and spectral properties 2 , while recent RL work has primarily focused on action variance 5 . Our work adopts jerk-based metrics as they capture both kinematic smoothness and practical concerns for physical deployment. Building energy systems present unique opportunities for smooth control validation. HVAC systems benefit from gradual adjustments that maintain occupant comfort while minimizing equipment cycling 8 . Studies demonstrate that excessive switching increases energy consumption compared to optimal smooth control 10 . Recent work has begun applying RL to building management 11 , but most implementations focus on reward engineering rather than fundamental smoothness constraints. 3 Problem formulation and methodology 3.1 Markov decision process with action history We consider continuous control tasks as Markov Decision Processes (MDPs) defined by state space \mathcal{S} , action space \mathcal{A}\subset\mathbb{R}^{d} , transition dynamics P(s^{\prime}|s,a) , and reward function r(s,a) . To maintain the Markov property while incorporating action smoothness constraints, we augment the state space with action history: \tilde{s}_{t}=[s_{t},a_{t-1},a_{t-2},a_{t-3}] (1) This augmentation enables direct computation of action derivatives. The policy \pi_{\theta}(\tilde{s}_{t}) can then access sufficient information for derivative-based regularization. 3.2 Higher-order derivative penalties We augment the reward function with penalties targeting different orders of action derivatives: First-order (velocity): \displaystyle r^{\prime}_{t}=r_{t}-\lambda_{1}\|a_{t}-a_{t-1}\|^{2} (2) Second-order (acceleration): \displaystyle r^{\prime}_{t}=r_{t}-\lambda_{2}\|a_{t}-2a_{t-1}+a_{t-2}\|^{2} (3) Third-order (jerk): \displaystyle r^{\prime}_{t}=r_{t}-\lambda_{3}\|a_{t}-3a_{t-1}+3a_{t-2}-a_{t-3}\|^{2} (4) Norms are computed as \|x\|^{2}=\sum_{i}x_{i}^{2} with equal weighting across action dimensions. Higher-order penalties target increasingly sophisticated aspects of smoothness. Jerk minimization is particularly relevant for mechanical and thermal systems, as rapid acceleration changes induce stress, vibration, and energy inefficiency. 3.3 Experimental setup Continuous control benchmarks: We evaluate across four OpenAI Gym environments 3 : HalfCheetah-v4 (high-dimensional locomotion), Hopper-v4 (unstable balancing), Reacher-v4 (precise manipulation), and LunarLanderContinuous-v2 (fuel-efficient spacecraft control). All experiments use PPO with consistent hyperparameters across methods, training for 1M timesteps with results averaged over 5 random seeds. For penalty weights in equations ( 2 )–( 4 ), we use \lambda_{1}=\lambda_{2}=\lambda_{3}=0.1 across all environments, enabling direct comparison of derivative orders without confounding from differential scaling. Building energy benchmark: We evaluate our methodology on a two-zone HVAC control environment, a custom-built open-source DollHouse, with SINDy-identified dynamics 4 from practical building data. The system controls temperature setpoints and damper positions while minimizing energy consumption and maintaining occupant comfort. 4 Results 4.1 Evaluation metrics Smoothness quantification: We measure smoothness using jerk standard deviation: \sigma_{\text{jerk}}=\text{std}(\dddot{a}_{t}) , computed via third-order finite differences. Energy efficiency: HVAC energy consumption follows standard building energy models, with particular focus on equipment switching frequency as a key efficiency indicator. 4.2 Continuous control benchmarks Table 1 : Performance and smoothness comparison across continuous control environments. Values show mean \pm std over 5 seeds following standard evaluation practices 1 . Smoothness is measured as jerk standard deviation (lower is smoother). Method HalfCheetah Hopper Reacher LunarLander Reward | Smoothness Reward | Smoothness Reward | Smoothness Reward | Smoothness Baseline 1052\pm 146 1977\pm 629 -6\pm 2 204\pm 90 6.806\pm 0.098 8.012\pm 0.150 0.158\pm 0.017 2.524\pm 0.122 First-order 990\pm 222 1711\pm 649 -4\pm 1 102\pm 125 2.811\pm 0.184 2.565\pm 0.078 0.144\pm 0.017 2.224\pm 0.122 Second-order 937\pm 36 1577\pm 947 -5\pm 2 185\pm 103 1.728\pm 0.019 2.833\pm 0.083 0.143\pm 0.013 1.498\pm 0.095 Third-order 725\pm 70 1379\pm 845 -5\pm 2 230\pm 68 1.443\pm 0.030 1.822\pm 0.077 0.097\pm 0.010 1.053\pm 0.091 Table 1 summarizes our continuous control results. Third-order derivative penalties consistently produce the smoothest policies, reducing jerk standard deviation by 78.8% in HalfCheetah, 77.3% in Hopper, 38.6% in Reacher, and 58.3% in LunarLander relative to the baseline. These gains come with only a modest performance cost in certain environments, indicating that higher-order regularization is the most effective approach to achieving smooth control. The trend across derivative orders is also clear: first-order penalties yield modest improvements, second-order penalties strengthen the effect, and third-order penalties consistently provide the largest reductions in action jerkiness across all environments. 4.3 Building energy management validation Figure 1 : HVAC control performance vs smoothness trade-off. Third-order regularization achieves optimal position in the upper-right quadrant, indicating both high performance and superior smoothness. Figure 1 demonstrates the practical value of smooth control in building energy management. Third-order regularization achieves the optimal performance-smoothness trade-off, maintaining competitive reward while delivering superior smoothness properties. HVAC short cycling significantly increases energy consumption. On-off control strategies can increase energy costs compared to continuous operation, with demand-limiting control methods achieving energy reduction ratios of 9.8% to 10.5%. Short cycling leads to increased energy consumption and higher energy bills due to systems working harder and less efficiently. Our smooth control approach reduces equipment switching events by 60%, aligning with established findings that frequent cycling imposes substantial energy penalties on HVAC systems. 5 Analysis and discussion 5.1 Why higher-order penalties excel Our results reveal that third-order

Read the complete text