Learning World Models for Unconstrained Goal Navigation Yuanlin Duan, Wensen Mao, He Zhu Abstract Learning world models offers a promising avenue for goal-conditioned reinforcement learning with sparse rewards. By allowing agents to plan actions or exploratory goals without direct interaction with the environment, world models enhance exploration efficiency. The quality of a world model hinges on the richness of data stored in the agent’s replay buffer, with expectations of reasonable generalization across the state space surrounding recorded trajectories. However, challenges arise in generalizing learned world models to state transitions backward along recorded trajectories or between states across different trajectories, hindering their ability to accurately model real-world dynamics. To address these challenges, we introduce a novel goal-directed exploration algorithm, MUN (short for "World Models for Unconstrained Goal Navigation"). This algorithm is capable of modeling state transitions between arbitrary subgoal states in the replay buffer, thereby facilitating the learning of policies to navigate between any "key" states. Experimental results demonstrate that MUN strengthens the reliability of world models and significantly improves the policy’s capacity to generalize across new goal settings. 1 Introduction Goal-conditioned reinforcement learning (GCRL) has emerged as a powerful framework for learning diverse skills within an environment and subsequently solving tasks based on user-specified goal commands, without requiring further training( Mendonca et al., (2021) ; Andrychowicz et al., (2017) ). Given that specifying dense task rewards for GCRL requires domain expertise, access to object positions, is time-consuming, and is prone to human errors, rewards in GCRL are typically sparse, signaling success only upon reaching goal states. However, sparse rewards pose a challenge for exploration during training. To address this challenge, several previous methods, e.g., Hafner et al., 2019a () ; Hansen et al., (2023) ; Mendonca et al., (2021) have proposed learning a generative world model of the environment using a reconstruction (decoder) objective, an instantiation of Model-based Reinforcement Learning (MBRL), visualized in Fig. 1 . This approach is appealing because the world model can provide a rich learning signal( Yu et al., (2020) ; Georgiev et al., (2024) ). For example, world models allow agents to plan their actions or exploratory goals without directly interacting with the real environment for more efficient exploration( Hu et al., (2023) ; Sekar et al., (2020) ). Existing MBRL techniques train world models to capture the dynamics of the environment from the agent’s past experiences stored in a replay buffer. The richness of the data stored in the agent’s replay buffer directly impacts the quality of a World Model. It is expected that the world model generalizes reasonably well to the state space surrounding the trajectories recorded in the replay buffer. However, the world model may not generalize well to state transitions backward along recorded trajectories or to states across different trajectories, which impedes the world model’s learning of the real-world dynamics. Figure 1: The general framework of model-based RL. To induce a data-rich replay buffer covering a wide range of dynamic transitions, in this paper, we present a novel goal-directed exploration algorithm for effective world modeling and policy learning, MUN (short for "World M odels for U nconstrained Goal N avigation"). MUN facilitates modeling state transitions between any subgoal states in the replay buffer, whether tracing back along recorded trajectories or transitioning between states on separate trajectories. This enhances the reliability of the learned world model and significantly improves the generalizability of the policy derived from the model to real-world environments, thereby boosting the exploration capabilities of the method. Additionally, we introduce a simple and practical strategy for discovering key subgoal states from the replay buffer. The key subgoals precisely mark the milestones necessary for task completion, such as steps like grasping and releasing blocks in the context of block-stacking scenarios. By world modeling and policy learning for unconstrained navigation between these key states, MUN can generalize to new goal settings, such as block unstacking that was not given to the agent at training time. Our key contributions are as follows. First, we propose a novel goal-directed exploration algorithm MUN for effective world modeling of state transition between arbitrary subgoal states in replay buffers. As the quality of the world model improves, MUN becomes highly effective at learning goal-conditioned policies that excel at exploration in sparse-reward environments. Second, we present a practical strategy for identifying pivotal subgoal states, which serve as milestones in completing sophisticated tasks. By training world models for unconstrained transition between these milestones, our method enables learning policies that can adapt to novel goal scenarios. Finally, we evaluate MUN in challenging robotics environments, such as guiding a multi-legged ant robot through a maze, maneuvering a robot arm amidst cluttered tabletop objects, and rotating items in the grasp of an anthropomorphic robotic hand. Across these environments, MUN exhibits superior efficiency in training generalizable goal-conditioned policies compared to baseline methods and ablations. 2 Problem Setup and Background We consider the problem of goal-conditioned reinforcement learning (GCRL) under a Markov Decision Process (MDP) parameterized by (S,A,P,G,\eta,R,\rho_{0}) . S and A are the state and action spaces, respectively. The probability distribution of the initial states is given by \rho_{0}(s) , and P(s^{\prime}|s,a) is the transition probability. \eta:S\to G is a mapping from the state space to the goal space, which assumes that every state s can be mapped to a corresponding achieved goal g . The reward function R is defined as R(s,a,s^{\prime},g)=1\{\eta(s^{\prime})=g\} . We assume that each episode has a fixed horizon T . For ease of presentation, we further assume S=G and \eta is an identify function in this paper. A goal-conditioned policy is a probability distribution \pi:S\times G\times A\to\mathbb{R}^{+} , which gives rise to trajectory samples of the form \tau=\{s_{0},a_{0},g,s_{1},\ldots,s_{T}\} . The purpose of the policy \pi is to learn how to reach the goals drawn from the goal distribution p_{g} . With a discount factor \gamma\in(0,1) , it maximizes J(\pi)=\mathbb{E}_{g\sim p_{g},\tau\sim\pi(g)}\left[\sum_{t=0}^{T-1}\gamma^{t}\cdot R(s_{t},a_{t},s_{t+1},g)\right] . In the context of model-based reinforcement learning (MBRL), a world model \hat{M} is trained over trajectories sampled from the agent’s interactions with the real environment, which are stored in a replay buffer, to predict the dynamics of the real environment. Fig. 1 illustrates the general MBRL framework. We use the world model structure \hat{M} of Dreamer ( Hafner et al., 2019a () ; Hafner et al., 2019b () ; Hafner et al., (2020) ; Hafner et al., (2023) ) to learn real environment dynamics as a recurrent state-space model (RSSM). We provide a detailed explanation of the network architecture and working principles of the RSSM in Appendix A.1 . Our study focuses on tackling the world model learning problem in goal-conditioned model based reinforcement learning settings. Particularly, we consider GC-Dreamer (goal-conditioned Dreamer) as an important baseline with the following learning components: \text{World Model:\quad}\hat{M}(s_{t}|s_{t-1},a_{t-1})\qquad\text{Actor:\quad}\pi^{G}(a_{t}|s_{t},g)\qquad\text{Critic:\quad}V(s_{t},g) (1) In GC-Dreamer, the goal-conditioned agent \pi^{G}(a|s,g) samples goal commands g\in G from the given environment goal distribution p_{g} to collect trajectories in the real world. These trajectories are used to train the world model \hat{M} , and subsequently, \pi^{G} is trained on imagined rollouts generated by \hat{M} using the model-based actor-critic algorithm in Dreamer Hafner et al., (2020) , with these two steps run in alternation. The critic estimates the sum of future rewards \sum_{t}r^{G}_{t} , and the actor tries to maximize the predicted values from the critic. The goal-reaching reward r^{G} is defined by the self-supervised temporal distance network D_{t} ( Mendonca et al., (2021) ), i.e. r^{G}(s,g)=-D_{t}(s,g) . D_{t} predicts the anticipated number of action steps needed to transition from s to g . Essentially, \pi^{G} is reinforced to minimize the action steps required to transition from the current state s to a sampled goal state g . The temporal distance estimator D_{t} is trained by extracting pairs of states s_{t} and s_{t+k} from an imagined rollout generated by running the policy over the world model and predicting the distance k between them as follows: D_{t}\big(\Psi(s_{t}),\Psi(s_{t+k})\big)\approx k/H (2) Here, \Psi represents the preprocessing for imagined states, such as transforming them into the world model’s latent space (we assume S=G in the paper). H represents the total length of the imagined rollout. Further details on the training procedure of D_{t} can be found in Appendix A.2 . 3 Training World Models for Unconstrained Goal Navigation In this section, we introduce MUN, our main approach to addressing the core challenge in GCRL: efficient exploration in long-horizon, sparse-reward environments. Our approach focuses on enhancing the agent’s understanding of the real-world environment through improved dynamic (world) modeling and latent space representation. As the quality of the world model improves, the goal-conditioned policy developed from it generalizes more effectively to the real environment. By closing the generalization gap between the policy’s behavior in the real environment and the world model, MUN effectively guides the agent’s exploration towards the desired goal region in the real environment. 3.1 Training Generalizable World Models (a) Key subgoal states in a 3-Block Stacking task. (b) Bidirectional Replay Buffer Figure 2: In Fig. 2(a) , we illustrate the key states involved in completing the task of 3-block stacking. In Fig. 2(b) , we demonstrate the significant advantages of the bidirectional replay buffer used in MUN over traditional methods in learning world models. Fig 1 illustrates the general framework of Model-based RL, where world models are trained using agent’s experiences stored in a replay buffer populated with observed environment transitions (s_{t},a_{t},s_{t+1}) linking the environment’s future states s_{t+1} and past states s_{t} along with the corresponding control actions a_{t} . The richness of the environment space and dynamic transitions captured by the replay buffer define the extent of what a world model can learn about the real environment. Through supervised learning, the model can generalize reasonably well within the state space moving forward along the trajectories recorded in the replay buffer. However, it may be inaccurate for the state transitions moving backward along the recorded trajectories or across different trajectories. Consider the task of stacking blocks using a robot manipulator in Fig. 2(a) . When humans learn to stack blocks, they also understand how to reverse the process to unstack the blocks or return to the initial state. In contrast, a world model trained solely on data from policies under training for stacking is unlikely to accurately model the unstacking process. As a result, the model may yield hallucinated trajectories for training policies, causing a significant discrepancy between the policy’s behavior in the model and in the real world, thereby leading to ineffective exploration. To improve model generalizability, in MUN, we proposed to learn world models capable of characterizing state transitions between any states in the replay buffer, whether by tracing back along recorded trajectories or transitioning between states on separate trajectories. Fig. 2(b) visualizes the comparison between the bidirectional replay buffer for learning world models used in MUN and the unidirectional replay buffer in conventional model-based algorithms. The bidirectional replay buffer not only covers a wider observation space but also captures a richer set of dynamic transitions. As discussed in Sec. 2 , due to joint optimization, the richer set of dynamic transitions in MUN allows for a more reliable latent representation of the environmental space and consequently a higher quality reward function (Equation 2 ) for training policies generalizable to the real environment on top of the learned model. Algorithm 1 The main training framework of MUN 1: Input: Policy \pi^{G} , World Model \hat{M} , reward function r^{G} , subgoals transfer number N_{s} , subgoal time limit T_{s} 2: Initialize buffers D,D_{DAD},D_{egc} 3: for i=1 to N_{train} do 4: if Should Plan Subgoals then 5: B_{egc}\leftarrow A batch of episodes from D_{egc} 6: G_{subgoals}\leftarrow DAD( B_{egc} ) with Algorithm 2 7: Initialize empty trajectory \tau 8: for s=1 to N_{s} do 9: t_{s}=0 10: g_{s}= Sample a subgoal randomly from G_{subgoals} 11: while agent has not reached g_{s} and t_{s}