Adaptive Coarse-to-Fine Subgoal Refinement for Long-Horizon Offline Goal-Conditioned Reinforcement Learning

Kaiqiang Ke 1,2 Shenghong He 1 Chengdong Xu 1,2 Yuheng Luo 1 Xiangyuan Lan 2 Chao Yu 1 1 Sun Yat-sen University 2 Pengcheng Laboratory {kekq,heshh23,xuchd6,luoyh227}@mail2.sysu.edu.cn [email protected] [email protected] · 2026

Read Adaptive Coarse-to-Fine Subgoal Refinement for Long-Horizon Offline Goal-Conditioned Reinforcement Learning by Kaiqiang Ke 1,2 Shenghong He 1 Chengdong Xu 1,2 Yuheng Luo 1 Xiangyuan Lan 2 Chao Yu 1 1 Sun Yat-sen University 2 Pengcheng Laboratory {kekq,heshh23,xuchd6,luoyh227}@mail2.sysu.edu.cn [email protected] [email protected] (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2605-28127.txt.

Introduction

Ke and colleagues tackle long-horizon goal-conditioned reinforcement learning from logged data alone. Their CFHRL framework refines distant goals step by step, proposing intermediate targets until each is judged locally reachable, rather than fixing hierarchy depth in advance. On OGBench it gained substantially on several long tasks, with ablations supporting its refinement and stopping rules.

Catalog record

CREATORKaiqiang Ke 1,2 Shenghong He 1 Chengdong Xu 1,2 Yuheng Luo 1 Xiangyuan Lan 2 Chao Yu 1 1 Sun Yat-sen University 2 Pengcheng Laboratory {kekq,heshh23,xuchd6,luoyh227}@mail2.sysu.edu.cn [email protected] [email protected]
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE60926
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2605.28127

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Adaptive Coarse-to-Fine Subgoal Refinement for Long-Horizon Offline Goal-Conditioned Reinforcement Learning Kaiqiang Ke 1,2 Shenghong He 1 Chengdong Xu 1,2 Yuheng Luo 1 Xiangyuan Lan 2 Chao Yu 1 1 Sun Yat-sen University 2 Pengcheng Laboratory {kekq,heshh23,xuchd6,luoyh227}@mail2.sysu.edu.cn [email protected] [email protected] Abstract Offline goal-conditioned reinforcement learning (GCRL) is challenging in long-horizon tasks, where distant state–goal pairs provide weak supervision and value estimates become vulnerable to accumulated bootstrapping errors. Hierarchical methods mitigate this difficulty by introducing intermediate subgoals, but fixed temporal abstractions or fixed hierarchy depths can be mismatched to state–goal pairs with different reachability horizons. We propose Coarse-to-Fine Hierarchical Goal Reinforcement Learning (CFHRL), a fully offline GCRL framework that adaptively refines distant goals before execution. Starting from the final goal, CFHRL recursively proposes intermediate targets, trained from replay-supported candidates, and stops refinement once the current target is estimated to be locally executable by a learned reachability cost. The key idea is that a subgoal need not be an exact midpoint or globally optimal waypoint; it only needs to provide reliable progress and reduce the remaining reaching difficulty, enabling subsequent refinement over shorter horizons. A stylized analysis further supports the robustness of approximate recursive contraction. Experiments on OGBench show substantial gains on several long-horizon tasks, with ablations validating the proposed refinement and stopping mechanisms. 1 Introduction Offline goal-conditioned reinforcement learning (GCRL) aims to learn policies that reach diverse goals from a fixed dataset of previously collected trajectories ( 27 ; 2 ; 8 ; 23 ) . This setting is appealing for robotics and embodied-control problems, where online exploration may be expensive, unsafe, or impractical ( 16 ; 10 ) . However, offline GCRL becomes particularly challenging when the desired goal is far from the current state. This difficulty mainly arises from long-horizon value estimation: under sparse goal-reaching rewards, distant state–goal pairs provide little direct supervision, and the value function has to propagate goal information through many bootstrapped updates ( 15 ; 25 ; 22 ) . As the temporal distance increases, approximation errors can accumulate along the backup chain and lead to inaccurate evaluations of faraway goals ( 28 ; 21 ; 19 ; 24 ; 1 ) . Consequently, directly training a flat goal-conditioned policy from this value function can be unreliable, as the policy must make long-horizon decisions based on potentially inaccurate value estimates. Hierarchical decomposition provides a natural way to mitigate the long-horizon difficulty in offline GCRL. Rather than requiring a flat policy to directly reach a distant goal, hierarchical methods introduce intermediate subgoals so that long-horizon goal reaching can be reduced to a sequence of shorter-range decisions ( 28 ; 21 ; 19 ; 24 ; 1 ) . Existing methods typically instantiate this idea through fixed temporal abstractions, trajectory-based subgoal extraction, or learned high-level subgoal policies ( 4 ; 24 ; 1 ) . Despite their effectiveness, these methods often rely on a prescribed decomposition scale, such as a fixed subgoal horizon or a fixed number of high-level decisions. Such a fixed scale can be mismatched to state–goal pairs with different reachability horizons: as shown in Figure 1 , overly local subgoals may make only marginal progress toward the final goal, so their small value differences can be dominated by long-horizon estimation errors and lead to misleading subgoal choices; whereas overly distant subgoals may fall outside the reliable execution range of the low-level policy. Figure 1: Motivation of adaptive stopping-time coarse-to-fine refinement. A flat goal-conditioned policy directly depends on noisy long-horizon value estimates. A fixed hierarchy introduces subgoals but uses a prescribed temporal scale or a fixed number of high-level decisions, which can be mismatched to state-goal pairs with different reachability horizons. CFHRL instead refines the current planning target only when it is estimated to be outside the executable region of the low-level policy, and stops once the target becomes locally reachable. This raises a natural question: can an offline hierarchical agent adaptively identify a suitable intermediate target for each state–goal pair? Our key idea is to solve long-horizon goal reaching in a coarse-to-fine manner. Although learned value functions can be noisy, coarse refinement does not require fine-grained discrimination among nearby states. At the initial stage, the candidate targets differ at a larger scale, so the value function only needs to provide a directional signal toward a replay-supported region that reduces the remaining reachability horizon. The selected subgoal therefore need not be optimal; it only needs to make meaningful progress and enable subsequent, finer refinement over shorter and more reliable horizons. As illustrated in the right panel of Figure 1 , this adaptive process converts a distant goal-reaching problem into a sequence of locally manageable subproblems. Motivated by this insight, we propose Coarse-to-Fine Hierarchical Goal Reinforcement Learning (CFHRL), a fully offline GCRL framework that adaptively refines distant goals before execution. Starting from the final goal, CFHRL recursively replaces the current planning target with intermediate targets produced by a planner trained on replay-supported candidates, until the target is estimated to be locally executable by a learned reachability cost. The refinement planner is trained from replay-supported candidates, using a reachability-based objective that favors intermediate targets which reduce the bottleneck cost between the current state and the original goal. After refinement, the selected target is executed through an abstraction policy and a low-level goal-conditioned policy, both trained by progress-weighted imitation over short-horizon offline segments. Overall, CFHRL uses a shared reachability signal to coordinate target selection, adaptive stopping, and local execution, allowing the effective decomposition depth to be determined by the estimated reachability of each state–goal pair in a fully offline manner. Our contributions are threefold: (1) We formulate long-horizon offline GCRL as adaptive stopping-time subgoal refinement, where the effective decomposition depth is determined by learned executability rather than a fixed horizon. (2) We introduce a replay-supported contraction objective that trains an amortized refinement planner without test-time graph search. (3) We provide a stylized motivating analysis and empirical ablations illustrating when approximate contraction and adaptive stopping can improve long-horizon maze-style goal reaching. 2 Preliminaries 2.1 Offline Goal-Conditioned Reinforcement Learning We consider an offline goal-conditioned Markov decision process \mathcal{M}=(\mathcal{S},\mathcal{A},\mathcal{G},P,r,\gamma,\rho_{0}) , where \mathcal{S} and \mathcal{A} denote the state and action spaces, \mathcal{G} denotes the goal space, P is the transition dynamics, \gamma\in[0,1) is the discount factor, and \rho_{0} is the initial-state distribution. In this work, goals share the same representation space as states; we use s and g only to distinguish the current state from the desired target. The objective is to learn a goal-conditioned policy \pi(a\mid s,g) from a fixed offline dataset \mathcal{D}=\{\tau_{i}\}_{i=1}^{N} , where \tau_{i}=(s_{0}^{i},a_{0}^{i},s_{1}^{i},\ldots,s_{T_{i}}^{i}) , without further environment interaction. Following the standard sparse goal-reaching formulation, the reward is defined as r(s_{t},a_{t},g)=\begin{cases}0,&\text{if }s_{t+1}\text{ reaches }g,\\ -1,&\text{otherwise}.\end{cases} Thus, values are non-positive, and values closer to zero indicate shorter expected reaching time. The main difficulty in long-horizon offline GCRL is that supervision for distant goals must be propagated through many transitions using only fixed offline data. 2.2 Goal-Conditioned Implicit Value Learning CFHRL uses a unified goal-conditioned value function as the reachability estimator. Following GCIVL ( 23 ) , we learn a V -only goal-conditioned value function from hindsight-relabeled offline transitions. Let (s,s^{\prime})\sim\mathcal{D} denote a state transition from the offline dataset, and let g\sim p_{\mathrm{mix}}^{\mathcal{D}}(\cdot\mid s) be a relabeled goal sampled from a mixture of

Read the complete text