GCHR : Goal-Conditioned Hindsight Regularization for Sample-Efficient Reinforcement Learning
Xing Lei, Wenyan Yang, Kaiqiang Ke, Shentao Yang, Xuetao Zhang, Joni Pajarinen, Donglin Wang · 2025
Read GCHR : Goal-Conditioned Hindsight Regularization for Sample-Efficient Reinforcement Learning by Xing Lei, Wenyan Yang, Kaiqiang Ke, Shentao Yang, Xuetao Zhang, Joni Pajarinen, Donglin Wang (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2508-06108.txt.
Introduction
Xing Lei and six co-authors present a method for teaching agents to reach goals from very sparse rewards, in this 2025 paper. In goal-conditioned reinforcement learning, a reward may come only when the agent finally hits its target, so most attempts teach it nothing. A popular fix relabels past attempts with the goals they actually reached, letting failures count as lessons.
The authors argue that relabeling alone still wastes much of each attempt. Their approach adds regularizers built from hindsight goals: one shapes the agent's actions directly, and a second, a self-imitation term, reinforces its own best past moves. Combined, the two squeeze more learning out of every episode without fresh data. On a suite of navigation and manipulation tasks, the method reused experience more efficiently and beat the rival goal-conditioned methods they tested, including those already using relabeling and self-imitation.
Catalog record
| CREATOR | Xing Lei, Wenyan Yang, Kaiqiang Ke, Shentao Yang, Xuetao Zhang, Joni Pajarinen, Donglin Wang |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 58 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2508-06108
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
GCHR : Goal-Conditioned Hindsight Regularization for Sample-Efficient Reinforcement Learning Xing Lei, Wenyan Yang, Kaiqiang Ke, Shentao Yang, Xuetao Zhang, Joni Pajarinen, Donglin Wang Abstract Goal-conditioned reinforcement learning (GCRL) with sparse rewards remains a fundamental challenge in reinforcement learning. While hindsight experience replay (HER) has shown promise by relabeling collected trajectories with achieved goals, we argue that trajectory relabeling alone does not fully exploit the available experiences in off-policy GCRL methods, resulting in limited sample efficiency. In this paper, we propose Hindsight Goal-conditioned Regularization (HGR), a technique that generates action regularization priors based on hindsight goals. When combined with hindsight self-imitation regularization (HSR), our approach enables off-policy RL algorithms to maximize experience utilization. Compared to existing GCRL methods that employ HER and self-imitation techniques, our hindsight regularizations achieve substantially more efficient sample reuse and the best performances, which we empirically demonstrate on a suite of navigation and manipulation tasks. 1 Introduction Deep reinforcement learning (RL) has been successful in a variety of tasks, including controlling robots ( Zheng et al. 2024a ; Li et al. 2025 ) , playing computer games ( Roayaei Ardakany and Afroughrh 2024 ; Rao et al. 2025 ) , and understanding natural language ( Uc-Cetina et al. 2023 ; Shinn et al. 2024 ) . Goal-conditioned Reinforcement Learning (GCRL) ( Liu, Zhu, and Zhang 2022 ) is a subdomain that trains agents to reach desired goal, making it an important step toward real-world robot control and general intelligence ( Park et al. 2024 ) . One of the most essential factors affecting sample efficiency in GCRL is the sparse reward, in which case informative learning signals only appear at the end of the trajectory. To address this problem, an active line of research focuses on designing novel learning algorithms that efficiently use the data. One popular approach is Hindsight Experience Replay (HER) Andrychowicz et al. 2017 , which replaces the desired goal in the data with the one the agent actually reached, effectively increasing the frequency of the learning signals ( Zheng et al. 2024b ) . Since HER may introduce some bias during the training, the following works proposed addtional learning or planning techniques to correct the HER bias ( Li, Pinto, and Abbeel 2020 ; He, Zhuang, and Li 2020 ; Schramm et al. 2023 ; Yang et al. 2021 ) . Figure 1: We propose two policy regularizations to achieve sample-efficient goal-conditioned policy learning: H indsight S elf-imitation R egularization ( HSR ) and H indsight G oal R egularization ( HGR ). During training, we aim to regularize the policy \pi to stay close to two action priors through KL regularization: the HSR prior ( \pi_{\text{HSR}} ) and the HGR prior ( \pi_{\text{HGR}} ). The HSR prior \pi_{\text{HSR}} is generated through hindsight experience replay and behavioral cloning, encouraging the policy to reproduce successful past actions. The HGR prior \pi_{\text{HGR}} is constructed by sampling hindsight goals g^{\prime} from the hindsight goal distribution \mu_{g^{\prime}}(\cdot|g) (where g is the desired goal) and aggregates the corresponding actions from a delayed-update target policy \pi^{\prime} . This enables the policy to maximize past experience usage. Based on HER, simple yet efficient self-imitation learning methods have been proposed to learn goal-conditioned policies ( Ghosh et al. 2021 ; Yang et al. 2022 ; Ma et al. 2022 ; Hejna, Gao, and Sadigh 2023 ; Eysenbach et al. 2022 ) . These works leverage the hindsight relabeled experience with weighted behavior cloning to learn goal-conditioned policies. Theoretically, they guarantee to optimize over a lower bound of the objective for goal reaching problem ( Yang et al. 2022 ; Hejna, Gao, and Sadigh 2023 ) . In addition to the purely self-imitation works, some GCRL methods also propose to use self-imitation learning techniques as policy regularizations (such as Imagined Subgoals (RIS) ( Chane-Sane, Schmid, and Laptev 2021 ) and GRSIL ( Li, Wang, and Tan 2023 ) .) However, these methods typically require additional networks or planning, which makes them prone to training instability and high computational ( Li, Pinto, and Abbeel 2020 ; He, Zhuang, and Li 2020 ; Yang et al. 2021 ; Chane-Sane, Schmid, and Laptev 2021 ; Schramm et al. 2023 ) . In addition, we find that HER has limited action coverage of the collected experiences, which limit the ability of maximizing sample-efficiency (see Figure 1 , where the hindsight action set a_{\text{HSR}} covers limited goal-conditioned actions). In this paper, we propose simple yet effective Goal-Conditioned Hindsight Regularization (GCHR), which maximizes the off-policy GCRL’s experience utilization for sample-efficient goal-conditioned policy learning. GCHR incorporates two HGR techniques: Hindsight Self-Imitation Regularization (HSR) and Hindsight Goal Regularization (HGR). These components are designed to complement and enhance policy learning, as illustrated in Figure 1 . First, we integrate the well established hindsight self-imitation regularization (HSR), which is HER with self-imitation regularization to optimize the relabeled goal policy; In addition to HSR, we use HGR to generate wider action prior, which covers the HSR’s action set and is able to continuously optimize its HGR prior throughout training, adapting to the evolving policy distribution. We theoretically show that our method can learn a better policy than both the self-imitation policy and the goal-conditioned RL methods. Our contributions can be summarized as followings: • Our proposed goal-conditioned hindsight regularizations (GCHR), combining HGR and HSR, significantly improve sample efficiency and performance in off-policy GCRL. • Our analysis shows that HGR achieves broader action coverage using only collected experiences, maximizing experience utilization compared to HER with self-imitation techniques. • GCHR is simple to implement: requiring only five lines of code without additional training modules or planning algorithms. 2 Related Work Our work concentrates on sample efficiency in goal-conditioned reinforcement learning (GCRL) with sparse rewards. HER ( Andrychowicz et al. 2017 ) addresses the issue of spare rewards in GCRL by regarding the achieved goal as the desired goal to train the action value. Specifically, the visited trajectories are used to sample the suitable subgoals, where some of the non-sparse rewards are designed to enable effective training. Based on HER, a few studies have been investigated to find more critical goal for desired goal. CHER ( Fang et al. 2019 ) selects goals in a heuristic way to balance the diversity of selected goals and the proximity to original goals. Nevertheless, the curriculum is hand-designed and needs to be adjusted through hyperparameters. MHER ( Yang et al. 2021 ) constructs a dynamics model using historical trajectories and combines current policy and desired goal to generate virtual relabeled goals. Although MHER additionally samples the desired goal and enhances exploration efficiency, they face the issue of inaccurate learning in the dynamics model. In addition to improving the goal sampling methods, recent advances in self-imitation have further improved learning efficiency, as exemplified by methods such as Goal-conditioned Supervised Learning (GCSL) ( Ghosh et al. 2021 ) , Weighted Goal-conditioned Supervised Learning (WGCSL) ( Yang et al. 2022 ) , f -weighted Regression (GoFar) ( Ma et al. 2022 ) and Distance Weighted Supervised Learning (DWSL) ( Hejna, Gao, and Sadigh 2023 ) . However, they always biased toward past experience and can lead to suboptimal ( Eysenbach et al. 2022 ) ; while our HGR’s prior is monotonically improve