Learning to Explore in Diverse Reward Settings via Temporal-Difference-Error Maximization

Sebastian Griesbach, Carlo D’Eramo · 2025

Read Learning to Explore in Diverse Reward Settings via Temporal-Difference-Error Maximization by Sebastian Griesbach, Carlo D’Eramo (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2506-13345.txt.

Introduction

Sebastian Griesbach and Carlo D'Eramo propose in 2025 a new exploration method for reinforcement learning agents. Exploration is hard when rewards are sparse, dense, or even punish movement. Common noise or bonus methods need much tuning to switch between those settings.

Their method, called SEE, revives the idea of seeking states where prediction error is high. Three design choices tame the instability that plagued older versions of that idea. It can plug into standard off-policy algorithms without changing their training flow. Tests show one agent design performing robustly across all three reward settings with no tuning changes.

Catalog record

CREATORSebastian Griesbach, Carlo D’Eramo
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE42 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2506-13345

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Learning to Explore in Diverse Reward Settings via Temporal-Difference-Error Maximization Sebastian Griesbach, Carlo D’Eramo Abstract Numerous heuristics and advanced approaches have been proposed for exploration in different settings for deep reinforcement learning. Noise-based exploration generally fares well with dense-shaped rewards and bonus-based exploration with sparse rewards. However, these methods usually require additional tuning to deal with undesirable reward settings by adjusting hyperparameters and noise distributions. Rewards that actively discourage exploration, i.e., with an action cost and no other dense signal to follow, can pose a major challenge. We propose a novel exploration method, Stable Error-seeking Exploration (SEE) 1 1 1 The code to reproduce all experiments is available at: https://github.com/Sebastian-Griesbach/SEE , that is robust across dense, sparse, and exploration-adverse reward settings. To this endeavor, we revisit the idea of maximizing the TD-error as a separate objective. Our method introduces three design choices to mitigate instability caused by far-off-policy learning, the conflict of interest of maximizing the cumulative TD-error in an episodic setting, and the non-stationary nature of TD-errors. SEE can be combined with off-policy algorithms without modifying the optimization pipeline of the original objective. In our experimental analysis, we show that a Soft-Actor Critic agent with the addition of SEE performs robustly across three diverse reward settings in a variety of tasks without hyperparameter adjustments. 1 Introduction Exploration is widely recognized as a crucial and distinctive aspect of reinforcement learning (RL). Being such a fundamental problem, a plethora of theoretical studies and empirical works have been produced over the years. Practical solutions to tackle exploration are numerous, with a prevalence of methods based on the injection of stochastic noise into policies either directly ( Mnih et al. 2013 ; Lillicrap et al. 2016 ; Scott Fujimoto et al. 2018 ; Fortunato et al. 2018 ) or via entropy maximization ( Tuomas Haarnoja et al. 2018 ; Schulman et al. 2017 ) . Noise-based exploration has proven to be highly effective in settings where the reward function is informative and well-shaped. For less favorable reward settings, e.g., sparse rewards, typically more targeted methods are employed. For example, bonus-based methods, also referred to as intrinsic motivation, are a class of reward-shaping mechanisms that aim to substitute sparse rewards such that the reward signal is transformed back to a shaped reward scheme again ( Bellemare et al. 2016 ; Burda et al. 2019 ; Yiming Wang et al. 2023 ) . Such methods often substitute the reward but still rely entirely on random noise for the actual action selection for exploration. The reshaped rewards can potentially dilute the original reward function of the MDP, changing the optimal policy or causing over-exploration of irrelevant state regions, which may harm performance in other reward settings ( Taıga et al. 2020 ) . If in a sparse reward setting one tries to enforce efficient behavior, it is reasonable to apply action costs. The resulting reward may actively discourage exploration as every action that does not immediately reach the goal is punished, adding complication to this setting. Adapting the exploration of a given RL algorithm from one setting to another often demands a costly tuning procedure, which might be infeasible in complex problems. Therefore, making RL algorithms more robustly applicable is an ongoing effort of the research community. To this endeavor, we propose Stable Error-seeking Exploration (SEE), a novel approach for exploration that robustly handles diverse reward settings. SEE is a directed action selection mechanism that can be combined with any off-policy RL algorithm and does not introduce relevant additional hyperparameters. Our approach separates exploration from exploitation by optimizing for two disentangled objectives and training two individual policies. On the one hand, the exploitation objective remains the unchanged RL objective of maximizing the (discounted) cumulative return, and, thus, it can be combined with any existing RL optimization pipeline, including reward shaping mechanisms. On the other hand, our exploration objective maximizes the absolute temporal-difference-error (TD-error) encountered during training rollouts. The intuition behind this choice is that a high TD-error, especially in deterministic environments, indicates the presence of potentially relevant information not yet learned. We point out that decoupling exploration and exploitation into two separate objectives is not a new idea, as shown in prior works ( Whitney et al. 2021 ; Simmons-Edler et al. 2020 ; Schäfer et al. 2022 ) . However, till now, it has been challenging to devise effective and stable learning procedures when maximizing the TD-error. In this paper, we identify three distinct causes of instability affecting this setting, namely (i.) far off-policy learning, when a behavior policy is too out of distribution w.r.t. the target policy; (ii.) the conflict of interest of terminating an episode while the exploration objective follows an always positive reward signal; and (iii.) the non-stationary nature of the TD-error as a target. The contribution of this work is the formulation of a methodological solution to tackle each of these issues, i.e., (1.) combining the exploration and exploitation policies into a single behavior policy that bridges both distributions; (2.) using a maximum reward update, which is agnostic towards the length of an episode; (3.) conditioning the exploration value function on current estimates of the exploitation value function to inform it about the cause of change in the TD-error target. We integrate these solutions into one algorithmic approach, resulting in SEE. We empirically analyze SEE by combining it with Soft Actor-Critic (SAC) ( Tuomas Haarnoja et al. 2018 ; Haarnoja et al. 2019 ) and TD3 ( Scott Fujimoto et al. 2018 ) . We show that incorporating SEE enhances robustness across diverse reward settings, without the need for hyperparameter tuning. Especially SAC+SEE shows strong performance across all tested environments. Furthermore, we conduct ablation studies to analytically evince the positive effect of our proposed design choices. 2 Preliminaries We consider a reinforcement learning (RL) setting ( Sutton & Barto 2020 ) in which the environment is modeled as a Markov decision process (MDP), defined by the tuple \mathcal{M}=(\mathcal{S},\mathcal{A},\mathcal{P},\mathcal{R},\gamma) , where \mathcal{S} is the state space, \mathcal{A} the action space, \mathcal{P}(s^{\prime}|s,a) the state transition function, \mathcal{R}(s,a,s^{\prime}) the reward function, and \gamma\in[0,1) the discount factor. A stochastic policy \pi_{\theta}:\mathcal{S}\to\mathcal{P}(\mathcal{A}) defines \pi_{\theta}(a|s) , the probability of taking action a in state s . The objective is to maximize the expected return J(\pi_{\theta})=\mathbb{E}_{\pi_{\theta}}[\sum_{t=0}^{\infty}\gamma^{t}r_{t}] , where the expectation is over trajectories induced by \pi_{\theta} . In the actor-critic framework, the actor represents the policy, while the critic estimates either the state-value function V^{\pi}(s)=\mathbb{E}_{\pi}[\sum_{t=0}^{\infty}\gamma^{t}r_{t}|s_{0}=s] or the action-value function Q^{\pi}(s,a)=\mathbb{E}_{\pi}[r(s,a)+\gamma Q^{\pi}(s^{\prime},a^{\prime})|s,a] . The critic is typically updated by minimizing the temporal-difference (TD) error \delta=r+\gamma Q^{\pi}(s^{\prime},a^{\prime})-Q^{\pi}(s,a) , while the actor is updated via the policy gradient \nabla_{\theta}J(\pi_{\theta})\approx\mathbb{E}_{\pi_{\theta}}[\nabla_{\theta}\log\pi_{\theta}(a|s)Q^{\pi}(s,a)] . This framework leads to deep RL algorithms such as TD3 ( Scott Fujimoto et al. 2018 ) and SAC ( Tuomas Haarnoja et al. 2018 ) , which we are using as the basis of our work. 3 Stable Error-seeking Exploration We propose an off-policy exploration method that performs robustly in different reward scenarios without hyperparameter adjustments. To this end, we use separate objectives for exploitation and exploration . On the one hand, the exploitation objective remains the regular cumulative discounted return. On the other hand, the exploration objective aims to maximize the absolute value of the encountered TD-error of the

Read the complete text