Zero-Incentive Dynamics: a look at reward sparsity through the lens of unrewarded subgoals
Yannick Molinghen, Tom Lenaerts · 2025
Read Zero-Incentive Dynamics: a look at reward sparsity through the lens of unrewarded subgoals by Yannick Molinghen, Tom Lenaerts (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2507-01470.txt.
Introduction
Yannick Molinghen and Tom Lenaerts examine a hidden weakness in how machines learn from rewards, in a 2025 paper. Reinforcement learning agents chase reward signals. Sparse rewards are usually blamed when learning stalls. The authors point at a structural problem instead. Key subgoals on the path to success often pay nothing at all. They name this pattern zero-incentive dynamics.
Under such dynamics, the steps that matter most give no feedback. The team shows that leading subgoal-based deep methods fail to use these steps well. Performance also swings with the time gap between finishing a subgoal and the eventual reward. A long gap hurts learning badly. The findings expose a basic limit in current approaches. New methods, they argue, must infer hidden task structure without waiting for payment. That shift could unlock tasks where rewards are rare by design.
Catalog record
| CREATOR | Yannick Molinghen, Tom Lenaerts |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 32 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2507-01470
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Zero-Incentive Dynamics: a look at reward sparsity through the lens of unrewarded subgoals Yannick Molinghen, Tom Lenaerts Abstract This work re-examines the commonly held assumption that the frequency of rewards is a reliable measure of task difficulty in reinforcement learning. We identify and formalize a structural challenge that undermines the effectiveness of current policy learning methods: when essential subgoals do not directly yield rewards. We characterize such settings as exhibiting zero-incentive dynamics, where transitions critical to success remain unrewarded. We show that state-of-the-art deep subgoal-based algorithms fail to leverage these dynamics and that learning performance is highly sensitive to the temporal proximity between subgoal completion and eventual reward. These findings reveal a fundamental limitation in current approaches and point to the need for mechanisms that can infer latent task structure without relying on immediate incentives. 1 Introduction In reinforcement learning ( Sutton & Barto 2018 , RL) , agents learn an optimal policy by maximizing the expected cumulative reward through interactions with the environment. In cases where there is little-to-no feedback until the completion of the task, reward is said to be sparse. In such cases, learning becomes significantly more difficult due to the sparsity (even absence) of informative gradients for policy improvement ( Ocana et al. 2023 ) . This challenge has motivated substantial research into mitigation techniques such as reward shaping ( Y. Ng et al. 1999 ) , intrinsic motivation ( Pathak et al. 2017 ; Burda et al. 2018 ) , and hierarchical RL ( Sutton et al. 1999 ) . While these approaches aim to compensate for sparse signals by injecting auxiliary rewards or by introducing structure, they often treat sparsity as a scalar property of the reward function, namely the frequency or density of non-zero rewards. However, this view neglects the structure of the problem and the location of the reward with regard to subgoals in the tasks. In this work, we aim to shed light on why reward sparsity should not be considered to be the sole metric of a problem difficulty and look at reward sparsity through the lens of subtasks whose accomplishment is unrewarded. Two recent works ( Ocana et al. 2023 ; Molinghen et al. 2025 ) respectively introduced Partially Ordered Subtasks and Zero-Incentive Dynamics (ZID) intuitively to refer to essential transitions that are not directly reinforced despite being necessary for the task success. We begin by empirically showing in Section 3 that reward sparsity alone is an insufficient indicator of task difficulty. Specifically, we construct a series of environments with increasing reward density where exploration actually becomes harder, demonstrating that the distribution and alignment of rewards with subtask transitions matters more than their global frequency in that example. Then, we formalize in Section 4.1 the notion of ZID as a structural property of the underlying Markov Decision Process ( Bellman 1957 , MDP) characterized by unrewarded but mandatory transitions that shape the state space in bottlenecks. Using a graph-theoretic perspective, we define ZID in terms of cut-sets over the transition graph of the MDP and distinguish it from general reward sparsity by focusing on the causal and temporal relationship between subgoal completion and eventual reward. Next, in Section 4.2 we provide evidence that current state-of-the-art subgoal-oriented methods ( Sutton et al. 1999 ; Jeon et al. 2022 ; Xu et al. 2023 ) fail to exploit the presence of subgoals when they have ZID. Despite their strong performance in other benchmarks, these methods perform no better than other RL algorithms under ZID. Finally, we investigate in Section 4.3 the impact of reward delay on deep RL by introducing reward shaping with controlled delays between subgoal completion and reward delivery. We show that even when the reward density remains constant, decreasing this delay substantially improves the quality of the policy. This suggests that the proximity of rewards to subtask completions, rather than their mere presence, is critical for effective learning. Our findings highlight a gap in the current RL paradigm: the lack of mechanisms to identify unrewarded subtasks. Addressing this gap may require new architectures or representations that can detect structural dependencies in the environment, even in the absence of immediate reward signals. 2 Related work 2.1 Markov Decision Processes Markov Decision Process ( Bellman 1957 , MDP) provide a model of sequential decision-making under uncertainty and are the main formalism used in Reinforcement Learning ( Sutton & Barto 2018 , RL) . An MDP M are defined by the tuple M=\left<S,A,R,T\right> where S is the set of state (state space), A is the set of actions (action space), R:S\times A\times S\rightarrow\mathbb{R} is a reward function, and T:S\times A\times S\rightarrow\left[0,1\right] is the transition function that determines the probability of that in state s , taking action a results in state s^{\prime} . In particular, we note S_{0}\subset S the set of initial states and S_{G}\subset S the set of goal states. Given an MDP M=\left<S,A,R,T\right> , a directed weighted graph ( Wilson 2009 ) G=(V,E,W) can be constructed to represent M , where V=S is the set of vertices, E=\left\{(s,a,s^{\prime})~|~s,s^{\prime}\in S,~a\in A\right\} is the set of edges, and W=R is the weight function that associates to each edge the reward for taking the corresponding transition. 2.2 Reward sparsity In recent years, the concept of reward sparsity has regularly been discussed in the field of deep RL and has generally been presented as the source of the difficulty of the problem under study ( Andrychowicz et al. 2017 ; Burda et al. 2018 ; Ladosz et al. 2022 ; Trott et al. 2019 ) . However, only a handful of these studies define reward sparsity, let alone a justify why reward sparsity is problematic. Regardless, a wide variety of methods have been introduced to cope with reward sparsity. Meng 2024 splits them into three categories: reward shaping (or manual labelling), intrinsic motivation, and hierarchical RL. Reward shaping is the most straightforward way to tackle reward sparsity and relies on the introduction of additional rewards based on domain knowledge. Randløv & Alstrøm 1998 and Amodei et al. 2016 , among others, have illustrated how naive reward shaping can alter the definition of the task, resulting in a change to the optimal policy \pi^{*} and causing the agent’s policy to no longer fulfil the task designer’s intended objective. This is why, as early as 1999, Y. Ng et al. 1999 introduce Potential-Based Reward Shaping (PBRS) that modifies the reward function R as shown in Equation 1 . PBRS relies on the definition of a potential function \phi and ensures that the optimal policy \pi^{*} remains unchanged. However, even though PBRS guarantees the invariance of \pi^{*} , this technique relies on domain knowledge, is problem-specific and can therefore not readily be generalised to any task. Furthermore, PBRS can be extremely challenging to implement in practice, if not impossible due to a lack of domain-knowledge. R^{\prime}(s,a,s^{\prime})=R(s,a,s^{\prime})+\gamma\phi(s)-\phi(s^{\prime}) (1) Exploration bonuses aim at providing an extra reward to the agent according to the novelty of the state visited and can take various forms. In its most simplistic form, intrinsic curiosity is a statewise counter and the agent receives an extra reward that is annealed over the number of times this state has been visited ( Tang et