Reachability Weighted Offline Goal-conditioned Resampling

Wenyan Yang, Joni Pajarinen · 2025

Read Reachability Weighted Offline Goal-conditioned Resampling by Wenyan Yang, Joni Pajarinen (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2506-02577.txt.

Introduction

Wenyan Yang and Joni Pajarinen improve goal-based robot learning in a 2025 paper. Their setting is offline learning, where a robot learns from a fixed set of past trials. Any state in those trials could serve as a goal, but sampling goals at random creates many that the robot could never reach from a given start.

Their method scores each pairing by reachability, using a classifier trained on goal-conditioned values. Sampling then favours transitions that actually help reach goals. The module plugs into standard offline algorithms without changing them. On six simulated tasks, including a robot arm and a dexterous hand, performance rose a great deal. One hand task improved by nearly fifty percent over the baseline.

Catalog record

CREATORWenyan Yang, Joni Pajarinen
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE41 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2506-02577

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Reachability Weighted Offline Goal-conditioned Resampling Wenyan Yang, Joni Pajarinen Abstract Offline goal-conditioned reinforcement learning (RL) relies on fixed datasets where many potential goals share the same state and action spaces. However, these potential goals are not explicitly represented in the collected trajectories. To learn a generalizable goal-conditioned policy, it is common to sample goals and state–action pairs uniformly using dynamic programming methods such as Q-learning. Uniform sampling, however, requires an intractably large dataset to cover all possible combinations and creates many unreachable state–goal–action pairs that degrade policy performance. Our key insight is that sampling should favor transitions that enable goal achievement. To this end, we propose Reachability Weighted Sampling (RWS). RWS uses a reachability classifier trained via positive–unlabeled (PU) learning on goal-conditioned state–action values. The classifier maps these values to a reachability score, which is then used as a sampling priority. RWS is a plug-and-play module that integrates seamlessly with standard offline RL algorithms. Experiments on six complex simulated robotic manipulation tasks, including those with a robot arm and a dexterous hand, show that RWS significantly improves performance. In one notable case, performance on the HandBlock-Z task improved by nearly 50% relative to the baseline. These results indicate the effectiveness of reachability-weighted sampling. 1 Introduction Goal-conditioned reinforcement learning (GCRL) has emerged as a promising framework for training agents to master diverse skills in complex environments ( Laird et al. 1987 ; Kaelbling 1993 ; Liu et al. 2022 ) . Unlike conventional reinforcement learning methods that rely on carefully shaped rewards, GCRL typically uses sparse rewards that merely indicate whether a goal has been reached, thereby simplifying the feedback mechanism without compromising the agent’s ability to optimize its behavior. Recent efforts have also connected GCRL with unsupervised contrastive learning ( Eysenbach et al. 2022 ; Zheng et al. 2023 ) , linking goal-driven approaches to domains like computer vision and natural language processing. Moreover, the flexibility of goal representations ranging from state coordinates to images or linguistic descriptions broadens the applicability of GCRL. By learning these goals, agents can acquire meta-skills conducive to hierarchical control and planning for more complex tasks ( Ghosh et al. 2023 ; Wang et al. 2023a ; Li et al. 2022 ) . Offline GCRL extends this paradigm by combining goal-conditioned tasks with offline reinforcement learning, which eliminates the need for additional environment interactions and leverages static datasets for policy learning ( Levine et al. 2020 ; Agarwal et al. 2023 ; Liu et al. 2021 ) . This is particularly advantageous in scenarios where active exploration is expensive or unsafe. As datasets containing diverse goals become increasingly available ( Eysenbach et al. 2022 ; Zheng et al. 2023 ) , policies trained in offline GCRL can gain a wide range of reusable primitives for downstream tasks ( Ghosh et al. 2023 ; Wang et al. 2023a ; Li et al. 2022 ) . Nevertheless, effectively learning multi-goal behaviors from purely offline data remains challenging due to the limited coverage of state-goal pairs and the prevalence of suboptimal or noisy samples. It is common to randomly combine state-action pair with random goals to learn a generalizable goal-conditioned policy from a static dataset ( Chebotar et al. 2021 ; Eysenbach et al. 2022 ) , however, random goal sampling tends to decrease the policy performance as it generates suboptimal even non-optimal data pairs Park et al. 2024 ; Eysenbach et al. 2020 . Figure 1: Diagram of Reachability Weighted Sampling (RWS) . The key idea of RWS is to learn a reachability classification function that samples potentially reachable goals (the dark blue circles in the figure) more frequently instead of sampling all goals in the dataset uniformly (the light yellow circles). In this work, we propose a novel sampling strategy, Reachability-Weighted Sampling (RWS), to address challenges in offline GCRL. As illustrated in Figure 3 , RWS employs a reachability classifier that evaluates whether a goal is achievable from a state-action pair by leveraging the goal-conditioned Q-value. We form a positive-unlabeled dataset by generating positive examples via hindsight relabeling and unlabeled examples via uniform goal sampling from the offline dataset. The classifier is optimized with non-negative PU learning to convert Q-values into reachability scores. An exponential transformation and subsequent normalization yield sampling weights over the goals. Consequently, transitions with potentially reachable goals are sampled more frequently, while unreachable ones are down-weighted. This resampling strategy enhances the quality of goal-conditioned experiences, reducing the impact of noisy, suboptimal samples. The main contribution of this work can be summarized as: 1) we present a novel, plug-and-play resampling scheme for offline goal-conditioned reinforcement learning. Our method leverages a Q-value-based reachability classifier, trained using PU learning, to re-weight experience by preferentially sampling transitions with achievable goals. 2) We integrate RWS with standard offline RL algorithms (e.g., MCQ, TD3BC, ReBRAC) and demonstrate that our approach consistently enhances both performance and data efficiency. 2 Related Works Goal-Conditioned Augmentations A classical technique in goal-conditioned reinforcement learning is Hindsight Experience Replay (HER) ( Andrychowicz et al. 2017 ) , which replaces task goals with those corresponding to states actually visited during execution. HER enriches the reward signal, facilitating value function learning and enabling the extraction of useful skills even from trajectories that would otherwise yield zero rewards. Building on HER, goal-conditioned weighted supervised learning methods ( Liu et al. 2021 ; Yang et al. 2022a ; Ma et al. 2022 ; Hejna et al. 2023b ; Lei et al. 2024 ) repeatedly perform weighted imitation learning with relabeled trajectories. However, HER tends to consistently generate successful trajectories, which leads to a bias in the training samples. In contrast, the Actionable Model (AM) ( Chebotar et al. 2021 ) randomly selects goals from the entire dataset to augment goal-conditioned trajectories. This offline goal-conditioned relabeling not only significantly increases the dataset size but also produces many unsuccessful trajectories, as the relabeled goal g^{\prime} is often sampled from states not encountered in the current trajectory. To sample reachable goals, recent works ( Huang et al. 2024 ; Mezghani et al. 2023 ) construct a graph from the offline data and use graph search to identify potentially reachable goals, thereby generating more effective goal-conditioned trajectories or augmenting the reward to train a more generalizable policy. However, building a directed graph becomes impractical when the state space is high-dimensional or the dataset is large. Moreover, random goal sampling may drastically decrease data quality, making it difficult to learn value functions for policy optimization ( Park et al. 2024 ; Eysenbach et al. 2020 ) . Weighted Experience Resampling The effect of dataset quality on offline reinforcement learning (RL) policy learning has been studied in previous works ( Hong et al. 2023 ; Park et al. 2024 ; Zhan et al. 2022 ) . A key observation is that policies may mimic suboptimal actions, which degrades performance. One solution is to modify the sampling strategy so that the policy is constrained to high-quality state-action pairs rather than all pairs in the dataset ( Hong et al. 2023 ) . An intuitive approach is advantage weighting, which uses an advantage function to identify state-action pairs that are likely to yield better performance ( Peng et al. 2019 ; Wang et al. 2020 )

Read the complete text