When Maximum Entropy Misleads Policy Optimization

Ruipeng Zhang, Ya-Chien Chang, Sicun Gao · 2025

Read When Maximum Entropy Misleads Policy Optimization by Ruipeng Zhang, Ya-Chien Chang, Sicun Gao (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2506-05615.txt.

Introduction

Ruipeng Zhang, Ya-Chien Chang, and Sicun Gao warn about a popular learning method in twenty twenty-five. Maximum entropy reinforcement learning rewards both return and randomness in a policy. It learns fast and acts robustly on many tasks. Yet on precision-critical control problems, plain methods sometimes win while entropy methods fail.

The authors analyze the trade-off between robustness and optimality. Extra entropy aids exploration but can pull the policy away from exact actions. Tasks needing precise, low-entropy policies suffer most. Experiments across varied control problems show the misleading effect concretely. Readers get guidance on when to temper entropy bonuses. The work aims at better balance between reward design and exploration pressure.

Catalog record

CREATORRuipeng Zhang, Ya-Chien Chang, Sicun Gao
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE67 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2506-05615

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

When Maximum Entropy Misleads Policy Optimization Ruipeng Zhang, Ya-Chien Chang, Sicun Gao Abstract The Maximum Entropy Reinforcement Learning (MaxEnt RL) framework is a leading approach for achieving efficient learning and robust performance across many RL tasks. However, MaxEnt methods have also been shown to struggle with performance-critical control problems in practice, where non-MaxEnt algorithms can successfully learn. In this work, we analyze how the trade-off between robustness and optimality affects the performance of MaxEnt algorithms in complex control tasks: while entropy maximization enhances exploration and robustness, it can also mislead policy optimization, leading to failure in tasks that require precise, low-entropy policies. Through experiments on a variety of control problems, we concretely demonstrate this misleading effect. Our analysis leads to better understanding of how to balance reward design and entropy maximization in challenging control problems. 1 Introduction The Maximum Entropy Reinforcement Learning (MaxEnt RL) framework ( Ziebart et al. 2008 ; Abdolmaleki et al. 2018 ; Haarnoja et al. 2018a ; Han & Sung 2021 ) augments the standard objective of maximizing return with the additional objective of maximizing policy entropy. MaxEnt methods such as Soft-Actor Critic (SAC) ( Haarnoja et al. 2018a ) have shown superior performance than other on-policy or off-policy methods ( Schulman 2015 ; Schulman et al. 2017 ; Lillicrap 2015 ; Fujimoto et al. 2018 ) in many standard continuous control benchmarks ( Achiam 2018 ; Raffin et al. 2021 ; Weng et al. 2021 ; Huang et al. 2024 ) . Explanations of their performance include better exploration, smoothing of optimization landscape, and enhanced robustness to disturbances ( Hazan et al. 2019 ; Ahmed et al. 2019 ; Eysenbach & Levine 2021 ) . Figure 1: ( Upper ) In the quadrotor control environment, SAC learns well under simplified dynamics, but fails to learn when under realistic dynamics models. PPO can learn well despite the use of the latter. ( Lower ) Intuitive illustration of hard control problems, where critical states naturally require low-entropy policies, while MaxEnt RL can favor mediocre states with robust policies of low returns that branch out towards failure and are not recoverable. Interestingly, the well-motivated benefits of MaxEnt and SAC have not led to its dominance in RL for real-world continuous control problems in practice ( Shengren et al. 2022 ; Tan & Karaköse 2023 ; Xu et al. 2021 ; Radwan et al. 2021 ) . Most recent RL-based robotic control work ( Kaufmann et al. 2023 ; Miki et al. 2022 ; Zhuang et al. 2024 ) mostly still uses a combination of imitation learning and fine-tuning with non-MaxEnt methods such as PPO ( Schulman et al. 2017 ) . The typical factors of consideration that favor PPO over SAC in practice include computational cost, sensitivity to hyperparameters, and ease of customization. Often the performance by SAC is indeed shown to be inferior to PPO despite efforts in tuning ( Muzahid et al. 2021 ; Lee & Moon 2021 ; Nair et al. 2024 ) . In fact, it is easy to reproduce such behaviors. Figure 1 shows the comparison of SAC and PPO for learning to control a quadrotor to follow a path. When the underlying model for the quadrotor is a simplified dynamics model, SAC can quickly learn a stable controller. When a more realistic dynamics model for the quadrotor is used, then SAC always fails, while PPO can succeed under the same initialization and dynamics. In this paper, we show how the conflict between maximizing entropy and maximizing overall return can be magnified and hinder learning in performance-critical control problems that naturally require precise, low-entropy policies to solve. The example of quadcopter control in Figure 1 highlights a common structure in complex control tasks: achieving desired performance often requires executing precise actions at a sequence of critical states. At these states, only a narrow (often zero-dimensional) subset of the action space is feasible, hence the ground-truth optimal policy has inherently low entropy. (In aerodynamic terms, this is often referred to as flying on the edge of instability .) Conversely, actions that deviate from this narrow feasible set often lead to states that are not recoverable: once the system enters these states, all available actions tend to produce similarly poor outcomes but accumulate short-term “entropy benefits” that can be favored by MaxEnt. Over time, this drift can compound, ultimately pushing the system into irrecoverable failure. Consequently, MaxEnt RL may bias the agent toward suboptimal behaviors rather than the precise low-entropy optimal policies that are key to solving hard control problems. Formalizing this intuition, we give an in-depth analysis of the trade-off. Our main result is that for an arbitrary MDP, there exists an explicit way of introducing entropy traps that we define as Entropy Bifurcation Extension , such that MaxEnt methods can be misled to consider an arbitrary policy distribution as MaxEnt-optimal, while the ground-truth optimal policy is not affected by the extension. Importantly, this is not a matter of sample efficiency or exploration bias during training, but is the end result at the convergence of MaxEnt algorithms. Consequently, the misleading effect of entropy can occur easily where standard policy optimization methods are not affected. We then demonstrate that this concern is not theoretical, and can in fact explain key failures of MaxEnt algorithms in practical control problems. We analyze the behavior of SAC in several realistic control environments, including controlling wheeled vehicles at high speeds, quadrotors for trajectory tracking, quadruped robot control that directly corresponds to hardware platforms. We show that the gap between the value landscapes under MaxEnt and regular policy optimization explains the difficulty of SAC for converging to feasible control policies on these environments. Our analysis does not imply that MaxEnt is inherently unsuitable for control problems. In fact, following the same analysis, we can now concretely understand why MaxEnt leads to successful learning on certain environments that benefit from robust exploration, including common benchmarking OpenAI Gym environments where SAC generally performs well. Overall, the analysis aims to guide reward design and hyperparameter tuning of MaxEnt algorithms for complex control problems. We will first give a toy example to showcase the core misleading effect of entropy maximization in Section 4 , and then generalize the construction to the technique of entropy bifurcation extension in Section 5 . We then show experimental results of how the misleading effects affect learning in practice, and how the adaptive tuning of entropy further validates our analysis in Section 6 . 2 Related Work MaxEnt RL and Analysis. The MaxEnt RL framework incorporates an entropy term in the RL objective and performs probability matching, such that the policy distribution aligns with the soft-value landscape ( Ziebart et al. 2008 ; Toussaint 2009 ; Rawlik et al. 2013 ; Fox et al. 2015 ; O’Donoghue et al. 2016 ; Abdolmaleki et al. 2018 ; Haarnoja et al. 2018a ; Mazoure et al. 2020 ; Han & Sung 2021 ) . MaxEnt RL has strong theoretical connections to probabilistic inference ( Toussaint 2009 ; Rawlik et al. 2013 ; Levine 2018 ) and well-motivated for ensuring robustness from a stochastic inference ( Ziebart 2010 ; Eysenbach & Levine 2021 ) and game-theoretic perspective ( Grünwald & Dawid 2004

Read the complete text