Latent Representation Alignment for Offline Goal-Conditioned Reinforcement Learning

Hyungkyu Kang, Byeongchan Kim, Min-hwan Oh · 2026

Read Latent Representation Alignment for Offline Goal-Conditioned Reinforcement Learning by Hyungkyu Kang, Byeongchan Kim, Min-hwan Oh (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2605-25740.txt.

Introduction

Kang, Kim and Oh study offline goal-conditioned reinforcement learning, where policies must reach goals from fixed data alone. They trace poor long-horizon results to faulty value generalization, and build the fix into Latent-Aligned Value Learning. It pairs latent value generalization with hierarchical planning in one framework. On OGBench it tops existing offline methods, winning the highest score on 20 of 22 datasets. It leads above all on long-horizon tasks and trajectory stitching.

Catalog record

CREATORHyungkyu Kang, Byeongchan Kim, Min-hwan Oh
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE78754
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2605.25740

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Latent Representation Alignment for Offline Goal-Conditioned Reinforcement Learning Hyungkyu Kang, Byeongchan Kim, Min-hwan Oh Abstract Offline goal-conditioned reinforcement learning (GCRL) provides a practical framework for obtaining goal-reaching policies from fixed datasets. However, learning a reliable goal-conditioned value function in long-horizon tasks remains challenging. In this paper, we identify erroneous generalization in goal-conditioned value functions as a fundamental bottleneck, and demonstrate that appropriate inductive bias in the value function is crucial for addressing the bottleneck. Building on these findings, we propose Latent-Aligned Value Learning (LAVL), an offline GCRL algorithm that integrates latent-representation-based value generalization with hierarchical planning in a unified framework. Extensive experiments on OGBench demonstrate that LAVL consistently outperforms existing offline GCRL methods, achieving the highest performance on 20 out of 22 datasets. Notably, LAVL exhibits strong performance in long-horizon tasks and trajectory stitching datasets, where prior methods suffer significant performance degradation. Our code is available at https://github.com/oh-lab/LAVL.git . 1 Introduction Offline goal-conditioned reinforcement learning (GCRL) aims to learn goal-reaching policies by leveraging pre-collected datasets without additional environment interaction ( 23 ; 33 ) . However, learning a reliable goal-conditioned value in long-horizon environments remains challenging ( 10 ; 3 ) . In particular, under sparse rewards, temporal-difference (TD) learning struggles to propagate rewards, often resulting in noisy and unreliable value estimates that induce brittle control policies ( 34 ) . Recent empirical studies ( 17 ; 1 ; 11 ; 12 ) suggest that the core difficulty in offline GCRL cannot be attributed solely to stochastic noise or local estimation error. Instead, a fundamental bottleneck arises from the inconsistency between the learned goal-conditioned value function and the true temporal distance of the environment. Although several methods have been proposed to address this issue ( 17 ; 1 ; 11 ; 12 ) , they do not fully resolve the problem: (i) their performance degrades sharply with increasing task horizon, (ii) trajectory stitching remains challenging, and (iii) some approaches introduce additional value networks with extra computational and tuning costs. Crucially, there has been limited analysis about why goal-conditioned value learning fails and which components of the learning pipeline are primarily responsible. Figure 1 : Success rates on OGBench maze navigation ( Point , Ant , and Humanoid ) and robot manipulation ( Cube and Scene ) tasks. For the maze environments, we report the average success rate across three maze sizes ( medium / large / giant ) and two dataset types ( navigate / stitch ). For Cube , results are averaged over play datasets ( single / double / triple ), and Scene reports the success rate on the play dataset. Each task setting is evaluated with 8 random seeds. In this work, we identify erroneous generalization in the goal-conditioned value function as a fundamental bottleneck. When value functions generalize by assigning similar values to states that are close in Euclidean distance, they often propagate value estimates across states that are distant in temporal distance, leading to large estimation errors. We demonstrate that the inductive bias in the value function architecture plays a central role in guiding accurate value generalization. While quasimetric architectures ( 24 ; 42 ) introduce useful inductive bias, their performance varies substantially and can even degrade in certain tasks. These findings motivate a new offline GCRL method that achieves reliable value generalization and consistent performance across diverse environments. In response, we propose Latent-Aligned Value Learning (LAVL) , an offline GCRL algorithm built on a new value function architecture, Latent Alignment Network (LAN) . LAN parameterizes the goal-conditioned value as the distance between learned states and goal representations, allowing value estimates to generalize through latent alignment rather than Euclidean proximity in the state space. Across a wide range of tasks, LAN-parameterized value functions improve upon or match existing architectures. This implies that the latent representation in LAN is the key to harnessing value generalization, rather than quasimetric structures. Building on LAN, our proposed algorithm, LAVL, is developed to fully exploit its strengths in long-horizon GCRL. To preserve global Bellman consistency while stabilizing the local value landscape, LAVL augments the TD objective with a continuity regularization term that suppresses sharp local fluctuations in the learned value function. In addition, LAVL naturally incorporates LAN into a hierarchical policy framework, facilitating effective long-horizon control. Together, these components enable LAVL to combine effective generalization, stable value learning, and hierarchical planning under a unified framework. As Figure 1 demonstrates, LAVL consistently outperforms existing offline GCRL methods on OGBench ( 33 ) , from maze navigation to robotic manipulation. Notably, LAVL maintains strong performance in long-horizon tasks and datasets that require extensive trajectory stitching, where existing methods suffer significant performance degradation. Moreover, LAVL substantially outperforms prior hierarchical methods in robotic manipulation tasks, demonstrating that LAN enables robust subgoal representation learning for hierarchical policies. Our contributions are summarized as follows: • We identify erroneous generalization in value functions as a fundamental bottleneck of goal-conditioned value learning, where values propagate by Euclidean proximity instead of temporal distance. To the best of our knowledge, our work is the first to demonstrate that inductive bias in the value function is crucial for reliable value generalization. • We propose LAVL, an offline GCRL algorithm that integrates effective generalization, stable value learning, and hierarchical planning within a unified framework. Through latent alignment between states and goal representations, LAVL enables reliable value generalization and stable long-horizon learning. • Extensive numerical experiments on OGBench demonstrate that LAVL consistently outperforms existing offline GCRL methods, achieving the best performance on 20 out of 22 datasets. Notably, LAVL maintains strong performance on long-horizon tasks and datasets that require extensive trajectory stitching. 2 Related Works GCRL. GCRL aims to learn a goal-conditioned policy that can achieve any goal in the goal space ( 38 ) . We focus on the offline GCRL setting, where the agent learns from pre-collected trajectories without online interaction. Goal-conditioned behavior cloning approaches directly imitate actions in the dataset ( 10 ; 44 ) . This approach is simple, but it cannot stitch trajectories, and the performance largely depends on the quality of the offline dataset. Like standard RL, many approaches are based on learning a goal-conditioned value function. The value-based approaches include TD learning ( 21 ; 34 ; 11 ; 12 ; 17 ; 1 ; 27 ) , state occupancy matching ( 5 ; 25 ) , contrastive learning ( 7 ; 8 ; 28 ; 27 ) , and constrained optimization ( 43 ) . GCRL raises major challenges due to the sparse reward signal and long horizon. To address these issues, hindsight relabeling ( 2 ; 4 ) methods utilize in-trajectory future states as goals, encouraging reward propagation. In addition, hierarchical RL approaches have been studied for planning with a long horizon, by subgoal generation ( 22 ; 40 ; 29 ; 31 ; 30 ; 13 ; 3 ; 34 ; 1 ) or graph-based planning ( 6 ; 15 ; 19 ; 45 ) . Inductive Bias for GCRL. Since the optimal goal-conditioned value function is quasimetric ( 43 ) , some of the prior GCRL works employ specialized quasimetric architectures, such as Metric Residual Network (MRN) ( 24 ) or Interval Quasimetric Embedding (IQE) ( 42 ) , to parameterize the

Read the complete text