Gradual Transition from Bellman Optimality Operator to Bellman Operator in Online Reinforcement Learning

Motoki Omura, Kazuki Ota, Takayuki Osa, Yusuke Mukuta, Tatsuya Harada · 2025

Read Gradual Transition from Bellman Optimality Operator to Bellman Operator in Online Reinforcement Learning by Motoki Omura, Kazuki Ota, Takayuki Osa, Yusuke Mukuta, Tatsuya Harada (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2506-05968.txt.

Introduction

Motoki Omura and co-authors speed up online reinforcement learning by blending two value targets over time in a 2025 paper. Methods for smooth, continuous actions usually learn values for the current policy only, so gains come solely from policy updates and samples are used poorly. Modeling the best possible values learns faster but inflates estimates.

Their fix starts training with the best value target and slowly shifts to the current policy target. Joined with two standard actor critic methods, the blend beat prior approaches on many walking and manipulation tasks. It also proved less touchy about settings tied to the best value target. Code is public.

Catalog record

CREATORMotoki Omura, Kazuki Ota, Takayuki Osa, Yusuke Mukuta, Tatsuya Harada
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE55 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2506-05968

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Gradual Transition from Bellman Optimality Operator to Bellman Operator in Online Reinforcement Learning Motoki Omura, Kazuki Ota, Takayuki Osa, Yusuke Mukuta, Tatsuya Harada Abstract For continuous action spaces, actor-critic methods are widely used in online reinforcement learning (RL). However, unlike RL algorithms for discrete actions, which generally model the optimal value function using the Bellman optimality operator, RL algorithms for continuous actions typically model Q-values for the current policy using the Bellman operator. These algorithms for continuous actions rely exclusively on policy updates for improvement, which often results in low sample efficiency. This study examines the effectiveness of incorporating the Bellman optimality operator into actor-critic frameworks. Experiments in a simple environment show that modeling optimal values accelerates learning but leads to overestimation bias. To address this, we propose an annealing approach that gradually transitions from the Bellman optimality operator to the Bellman operator, thereby accelerating learning while mitigating bias. Our method, combined with TD3 and SAC, significantly outperforms existing approaches across various locomotion and manipulation tasks, demonstrating improved performance and robustness to hyperparameters related to optimality. 1 Introduction In recent years, online reinforcement learning (RL) has been widely applied to robotics ( Schulman et al. 2015 ; Haarnoja et al. 2018 ) and gaming ( Mnih et al. 2013 ; Mnih et al. 2015 ; Badia et al. 2020 ; Tessler et al. 2017 ) , demonstrating remarkable performance, particularly in tasks with discrete action spaces like games. Q-learning-based algorithms, commonly used for discrete action tasks, estimate the optimal Q-value via the Bellman optimality operator. However, in tasks with continuous action spaces, computing \max_{a^{\prime}}Q(s^{\prime},a^{\prime}) for an infinite number of actions is challenging. Actor-critic-based algorithms address this by estimating the Q-value for the current policy using the Bellman operator. In these cases, policy improvement is achieved solely through policy updates, leading to slower performance improvement and reduced sample efficiency ( Ji et al. 2024 ) . In tasks with continuous action spaces, such as robotic control, sample collection is costly, which makes low sample efficiency a critical challenge. Although some prior studies have explored modeling (soft) optimal values instead of the values of the current policy, they faced challenges such as increased computational cost and instability ( Haarnoja et al. 2017 ; Garg et al. 2023 ; Kalashnikov et al. 2018 ) . This study first examines the effectiveness of modeling optimal values in actor-critic methods for online RL. Preliminary experiments were conducted with a low-dimensional environment using a tabular actor-critic method. These experiments compared SARSA-based updates, which used the Bellman operator, to Q-learning-based updates, which used the Bellman optimality operator, for critic learning. The results showed that in Q-learning-based methods, the Q-value directly approached the optimal value, whereas in SARSA-based updates, the Q-value improved only after the policy improved. This delay in policy learning caused the Q-value to converge to its optimal value more slowly in the SARSA-based approach. This finding motivates the application of the Bellman optimality operator in actor-critic methods. However, using the Bellman optimality operator with function approximators to model Q-values introduces overestimation bias due to the stochasticity of Q-values and the max operator in target values ( Thrun & Schwartz 1993 ; Lan et al. 2020 ; Chen et al. 2021 ) . Simulating this stochasticity by injecting noise into Q-values in our preliminary experiments showed that Q-learning-based methods converged to values larger than the optimal Q-value due to overestimation bias. Conversely, SARSA-based methods, which do not involve a max operator, exhibited lower sensitivity to noise. In summary, while the Bellman optimality operator accelerates learning, it also increases overestimation bias, leading to convergence to inflated Q-values. Conversely, the Bellman operator suppresses overestimation bias but results in slower learning. To address this, we propose a gradual transition from the Bellman optimality operator to the Bellman operator. This approach accelerates learning while eventually estimating values with reduced bias. In the preliminary experiments, linearly interpolating target values calculated with each operator, combined with annealing the degree of optimality, resulted in faster learning and less biased estimation. This method has additional benefits. While overestimation bias may occur in the early stages of learning depending on the annealing schedule, it can promote exploration, which is advantageous at this stage ( Lan et al. 2020 ) . In the later stages of learning, the current policy is expected to converge, making it undesirable to introduce bias for further policy improvement. Therefore, applying updates based on the Bellman operator, which inherently reduces bias, becomes a reasonable approach. In tasks with continuous action spaces, which we focus on in this study, the Bellman optimality operator is computationally intractable due to the max operation. To address this, we use the expectile loss ( Kostrikov et al. 2022 ) , which enables operations analogous to the max operator in continuous action spaces and a smooth interpolation between the Bellman optimality operator and the Bellman operator. By gradually reducing a parameter \tau representing the estimated expectile from a value close to 1 to 0.5, our operator transitions smoothly from the Bellman optimality operator to the Bellman operator. This approach can be implemented by replacing the L2 loss used for critic learning in existing actor-critic methods with the expectile loss and annealing the parameter \tau related to optimality. Experiments on various locomotion and manipulation tasks in continuous action spaces tested the proposed Annealed Q-learning (AQ-L) method combined with TD3 and SAC. The proposed methods significantly outperformed widely used algorithms such as TD3 and SAC. Additionally, annealing improved robustness to hyperparameters related to optimality and enhanced performance compared to static optimality estimates. The contributions of this study are as follows: • Preliminary experiments provided the insight that in actor-critic methods, using the Bellman optimality operator accelerates learning but introduces overestimation bias, while the Bellman operator reduces bias at the cost of slower learning. • We proposed a method for annealing from the Bellman optimality operator to the Bellman operator, enabling faster learning with reduced bias. This method can be easily implemented in existing actor-critic methods using the expectile loss. • We demonstrated that Annealed Q-learning combined with TD3 and SAC achieves significantly better performance than widely used algorithms in continuous action tasks, while annealing improves performance and robustness to optimality-related hyperparameters. The code for this study is available at https://github.com/motokiomura/annealed-q-learning . 2 Preliminaries 2.1 Reinforcement Learning In RL, the problem is defined within a Markov decision process (MDP) framework presented by the tuple (\mathcal{S},\mathcal{A},\mathcal{P},r,\gamma,d) . Here, \mathcal{S} denotes the set of all possible states, \mathcal{A} denotes the set of all possible actions, \mathcal{P}(s_{t+1}|s_{t},a_{t}) is the transition probability from one state to another given a specific action, r(s,a) represents the reward function assigning values to each state-action pair, \gamma is the discount factor that diminishes the value of future rewards, and d(s_{0}) is the probability distribution of initial states. The policy \pi(a\mid s) is the probability of taking a specific action in a given state. The objective of RL is to discover a policy that maximizes the expected sum of discounted rewards, denoted as \mathbb{E}[R_{0}\mid\pi] , where R_{t} is the return calculated as R_{t}=\sum^{T}_{k=t}\gamma^{k-t}r(s_{k},a_{k}) and T is a task horizon. 2.2 Bellman Operator and Bellman Optimality Operator In RL, the Bellman (expectation) operator and Bellman optimality operator ( Sutton 1988

Read the complete text