Shaping Sparse Rewards in Reinforcement Learning: A Semi-supervised Approach

Wenyun Li, Wenjie Huang, Chen Sun · 2025

Read Shaping Sparse Rewards in Reinforcement Learning: A Semi-supervised Approach by Wenyun Li, Wenjie Huang, Chen Sun (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2501-19128.txt.

Introduction

Wenyun Li, Wenjie Huang, and Chen Sun improve reinforcement learning when rewards are rare, in a research paper from 2025. Agents in many tasks get feedback only at scattered moments, so shaping a useful reward signal is hard. Most past methods learn only from the few rewarded steps and waste the rest.

The approach also learns from the mass of zero-reward steps. A semi-supervised method plus a new data-mixing trick builds a map of how paths through a task relate. Tests on Atari games and robot arm tasks beat supervised rivals, doubling peak scores in the sparsest settings. The mixing trick alone added a clear margin.

Catalog record

CREATORWenyun Li, Wenjie Huang, Chen Sun
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE35 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2501-19128

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Shaping Sparse Rewards in Reinforcement Learning: A Semi-supervised Approach Wenyun Li, Wenjie Huang, Chen Sun Abstract In many real-world scenarios, reward signal for agents are exceedingly sparse, making it challenging to learn an effective reward function for reward shaping. To address this issue, the proposed approach in this paper performs reward shaping not only by utilizing non-zero-reward transitions but also by employing the Semi-Supervised Learning (SSL) technique combined with a novel data augmentation to learn trajectory space representations from the majority of transitions, i.e., zero-reward transitions, thereby improving the efficacy of reward shaping. Experimental results in Atari and robotic manipulation demonstrate that our method outperforms supervised-based approaches in reward inference, leading to higher agent scores. Notably, in more sparse-reward environments, our method achieves up to twice the peak scores compared to supervised baselines. The proposed entropy augmentation enhances performance, showcasing a 15.8% increase in best score over other augmentation methods. I Introduction The sparse reward problem is a core challenge in Reinforcement Learning (RL) when solving real-world tasks [ 1 ] . Many real-world tasks naturally have the feature of delayed or infrequent rewards due to the complexity and nature of the tasks. Such sparse reward signals provide minimal immediate feedback to guide the agent’s exploration, while also inducing high variance in returns. These properties make it challenging to learn an optimal policy effectively [ 2 ] . In general, densifying reward with reward design or reward shaping [ 3 ] methods face the inescapable predicament of reward hacking [ 4 ] . This phenomenon can lead to a shift in the optimal policy, or cause the agent to learn a behavior that deviates from the intended design. In response to this dilemma, supervised-based reward shaping methods demonstrate greater fidelity to the original reward formulation. Moreover, [ 5 ] prove that, under mild assumptions, specific modifications to the reward function can still preserve the same set of optimal policies. However, methods based on non-expert demonstrations under supervised learning suffer from poor sample efficiency [ 6 ] , especially when non-zero-reward transitions are extremely rare in sparse reward cases. In order to bridge the gap in handling sparse rewards, our approach performs reward shaping not only by utilizing non-zero-reward transitions but also by employing Semi-Supervised Learning (SSL) techniques to learn trajectory space representations from the majority of transitions, i.e. , zero-reward transitions, thereby improving reward shaping. Explicitly, we propose an RL algorithm that applies the idea of SSL, called Semi-Supervised Reward Shaping framework (SSRS), which alternates between policy updates using a base RL algorithm and transition-level reward shaping via a learned reward estimator. Consistency regularization [ 7 ] to zero-reward transitions is adopted for the optimization of the reward estimator. The rationale of SSRS is substantiated by identifying state-action pairs analogous to transitions with actual environmental rewards, and by conferring denser rewards accordingly upon those deemed sufficiently confident as “good” state-action pairs. Our empirical investigation reveals that the trajectory space conforms to the mild assumptions requisite for SSL, thereby permitting the application of SSL methodologies to refine the evaluative efficacy of the inferred reward function over zero-reward transitions. The main contributions of the paper are listed as below: • We propose a novel reward shaping framework named SSRS, which operates on any base RL algorithm and employs SSL techniques to improve sample efficiency in sparse-reward environments. • We empirically examine the underlying conditions for SSL to be effective in RL settings, propose an additional data perturbation method to better process agent observations, and suggest potential directions for future applications of SSL in reinforcement learning. • We evaluate the performance of SSRS in reward-sparse Atari and robotic manipulation environments, comparing it with other supervised-based approaches—SORS [ 5 ] , RCP [ 8 ] —and the standard RL baselines. Our framework demonstrates superior performance across the tested scenarios. The remainder of this paper is structured as follows. Section. II introduces related work in reward shaping and SSL. Section. III presents the proposed framework alongside the necessary background on SSL. Section. III describes the experimental setup, provides comprehensive validation results, and offers an in-depth analysis of SSL. II Related Work Sparse reward is a common challenge in RL. In addition to reward shaping, curriculum learning and hierarchical reinforcement learning are also effective approaches to addressing this issue. Curriculum learning structures training data through a progressive sequence, transitioning the agent from elementary to increasingly intricate tasks. Recent research [ 9 ] introduces a mechanism that generates adaptive learning objectives without domain expertise, promoting efficient learning in complex environments. Hierarchical reinforcement learning addresses the issue by decomposing a complex RL problem into subproblems, deploying specialized agents to manage distinct tiers of decision-making. [ 10 ] propose a hierarchical reward modeling framework, which streamlines reward design by leveraging structural hierarchies or surrogate feedback to automatically evaluate trajectories and train reward models. Our work seeks to leverage reward shaping, which relies on densifying the reward function and therefore mitigate challenges in exploration or credit assignment induced by sparse rewards. There are different reward shaping methods for different objectives: Count-based methods [ 11 , 12 , 13 ] incentivize agents based on the rarity of states, while curiosity-driven exploration methods reward the agent’s exploratory behavior. ICM [ 14 ] also pursues rarity, but operationalizes it by defining curiosity as the error in the agent’s ability to predict the consequences of its own actions in a visual feature space. This curiosity signal serves as an intrinsic reward, driving the agent to explore novel states. [ 15 ] propose Hindsight Experience Replay (HER) that enables learning from failures by treating unachieved goals as alternative goals for experience replay. Approaches using supervised learning for reward shaping don’t rely on specific objectives. The study [ 8 ] employs supervised learning techniques, viewing non-expert trajectories collected from sub-optimal policies as optimal supervision to do the reward shaping. [ 5 ] leverage self-supervised methods to extract signals from trajectories while simultaneously updating the policy. We extend supervised learning to a semi-supervised paradigm and introduce SSL techniques into the RL framework, specifically employing consistency regularization and pseudo-labeling [ 7 ] in the trajectory space. In the realm of SSL, pseudo-labeling employs the model itself to generate artificial labels for unlabeled data, while consistency regularization leverages unlabeled data by capitalizing on the invariance assumption—that the model should yield consistent predictions when presented with perturbed variants of identical inputs. [ 16 , 17 ] further improve this methodology on image classification tasks. We will formally introduce these SSL techniques in the later Background section and justify how our reward estimator is optimized through them. III Methodology Fig. 1: The figure illustrates how the SSRS method leverages both non-zero reward transitions and sparse reward transitions for reward shaping through a semi-supervised learning approach incorporating strong-weak data augmentation. The augmentation techniques follow [ 17 ] , including cutout, smooth, gaussian and translation. Entropy augmentation is specified in Section. III-D . In this section, we introduce the proposed method and its pipeline is graphically illustrated in Figure. 1 , which consists of a reward estimator optimized through semi-supervised learning, and the data augmentation technique for transitions that serves as the perturbation required for semi-supervised learning. We will elaborate the key parts

Read the complete text