Contextual Latent World Models for Offline Meta Reinforcement Learning Mohammadreza Nakhaei, Aidan Scannell, Kevin Luck, Joni Pajarinen Abstract Offline meta-reinforcement learning seeks to learn policies that generalize across related tasks from fixed datasets. Context-based methods infer a task representation from transition histories, but learning effective task representations without supervision remains a challenge. In parallel, latent world models have demonstrated strong self-supervised representation learning through temporal consistency. We introduce contextual latent world models , which condition latent world models on inferred task representations and train them jointly with the context encoder. This enforces task-conditioned temporal consistency, yielding task representations that capture task-dependent dynamics rather than merely discriminating between tasks. Our method learns more expressive task representations and significantly improves generalization to unseen tasks across MuJoCo, Contextual-DeepMind Control, and Meta-World benchmarks. 1 Introduction One of the shortcomings of reinforcement learning (RL) is generalization , where a policy trained on one task typically performs poorly when applied to a related but unseen task. Meta-reinforcement learning (meta-RL) addresses this issue by training on a distribution of tasks and learning a policy that can adapt quickly to new tasks ( 10 ; 54 ; 3 ) . However, most meta-RL methods require online interaction with many training tasks, which is often expensive or infeasible in the real world. Offline meta-RL (OMRL) instead assumes access to offline datasets collected from related tasks and aims to learn a generalizable policy without additional environment interaction. A common approach in OMRL is context encoding ( 26 ; 14 ; 53 ; 27 ; 33 ; 47 ; 50 ) . In these methods, a context encoder maps a history of transitions (the context) to a latent vector called the task representation . The policy and value functions are conditioned on this task representation, which serves as an implicit task identifier without requiring explicit task labels or knowledge of the underlying task variation factors. As a result, the quality of the task representation plays a central role in generalization to unseen tasks. Many existing methods rely solely on contrastive learning to learn the task representation ( 26 ; 49 ; 51 ) . While contrastive learning encourages task discrimination, it does not explicitly enforce predictive structure over time. As a result, the learned task representation often fails to capture task-specific dynamics and reward functions, which limits generalization. UNICORN ( 27 ) partially addresses this issue by including one-step reconstruction objective as well, but one-step predictions are insufficient to capture long-horizon, task-dependent dynamics. Latent world models provide a strong self-supervised learning signal through multi-step temporal consistency. Rather than learning dynamics directly in the observation or state space, latent world models map observations to a latent state and learn dynamics in this latent space. They have been successfully used for decision-time planning and policy learning via imagination in model-based RL ( 15 ; 41 ; 18 ; 17 ; 40 ) and for representation learning in model-free RL ( 52 ; 11 ; 39 ; 12 ) . Across these settings, a core theme is that self-supervised representation learning based on self-predictive RL ( 42 ) – also known as temporal consistency ( 52 ) – provides a strong signal for learning latent state representations. However, directly applying latent dynamics in the OMRL setting is challenging, as the model must infer task-dependent variations without access to explicit task labels. In this work, we show that enforcing temporal consistency via self-predictive representation learning for context encoding leads to significantly better task representations. Based on these insights, we propose Self-Predictive Contextual Offline Meta-RL ( SPC ) , where a latent world model is conditioned on the inferred task representation. We train the context encoder and world model jointly based on temporal consistency and contrastive learning. This unifies task inference and predictive modeling within offline meta-RL and enables the task representation to capture long-horizon, task-specific dynamics. Contributions. Our main contributions are: C1 Temporal consistency for task inference. We show that enforcing latent temporal consistency during context encoding yields task representations that capture task variation factors more effectively than reconstruction-based objectives. C2 Theoretical analysis. We formally characterize the sources of error in value estimation and motivate our representation learning approach accordingly. C3 Extensive empirical evaluation. Through experiments on MuJoCo, Contextual DeepMind Control, and Meta-World benchmarks, we analyze the task representation quality and show that SPC significantly improves few-shot and zero-shot generalization compared to state-of-the-art OMRL methods. 2 Background Context-based Offline Meta-RL In offline meta-RL, we consider a set of training tasks, each modeled as a Markov Decision Process (MDP) \mathcal{M}_{i}=\langle\mathcal{S},\mathcal{A},R_{i},P_{i},\gamma,\rho_{0}\rangle , where all tasks share the same state space \mathcal{S} , action space \mathcal{A} , discount factor \gamma\in[0,1] , and initial state distribution \rho_{0} , but differ in their reward functions R_{i}:\mathcal{S}\times\mathcal{A}\to\mathbb{R} and transition dynamics P_{i}:\mathcal{S}\times\mathcal{A}\to\Delta\mathcal{S} . Each task \mathcal{M}_{i} is associated with an offline dataset \mathcal{D}_{i} . The goal of offline meta-RL is to learn a meta-policy \pi that generalizes to unseen tasks, i.e. , maximizes the expected return over a distribution of test tasks: \displaystyle J=\mathbb{E}_{\mathcal{M}_{i}\sim p_{\text{test}}(\mathcal{M})}\left[\mathbb{E}\left[\sum_{t=0}^{T}\gamma^{t}R_{i}(\mathbf{s}_{t},\mathbf{a}_{t})\right]\right]. (1) Context-based methods address task generalization by learning a context encoder that infers the underlying task from a small set of transitions. Specifically, a context encoder E_{\theta}:\mathcal{S}\times\mathcal{A}\times\mathbb{R}\times\mathcal{S}\rightarrow\mathcal{Z} maps transitions (\mathbf{s}_{j},\mathbf{a}_{j},r_{j},\mathbf{s}_{j}^{\prime}) to a task representation \mathbf{z} . The inferred task representation is then used to condition the policy \pi(\mathbf{a}_{t}\mid\mathbf{s}_{t},\mathbf{z}) , the value function Q(\mathbf{s}_{t},\mathbf{a}_{t},\mathbf{z}) , or a learned dynamics model, enabling adaptation to different tasks without explicit task labels. Self-predictive RL A state encoder F_{\phi}:\mathcal{S}\rightarrow\mathcal{X} maps states \mathbf{s}\in\mathcal{S} to latent states \mathbf{x}\in\mathcal{X} , also referred to as abstractions. These latent states are subsequently used by downstream components such as the policy, value function, or world model. Different methods train the encoder under different objectives and architectural couplings ( 42 ; 34 ; 39 ) . Self-supervised RL methods exploit (auxiliary) learning signals such as latent temporal consistency and latent reward prediction to train the encoder without relying on state reconstruction. Such an encoder should fulfill the following condition for learning optimal behavior ( 34 ) : \text{RP:}\hskip 3.99994pt\exists\,\hat{R}:\mathcal{X}\times\mathcal{A}\rightarrow\mathbb{R}\hskip 3.99994pt\text{s.t.}\hskip 3.99994pt\mathbb{E}[r_{t}\mid\mathbf{s}_{t},\mathbf{a}_{t}]=\hat{R}(\mathbf{x}_{t},\mathbf{a}_{t}),\\ \text{ZP:}\hskip 3.99994pt\exists\,\hat{P}:\mathcal{X}\times\mathcal{A}\rightarrow\Delta\mathcal{X}\hskip 3.99994pt\text{s.t.}\hskip 3.99994pt\mathbb{E}[\mathbf{x}_{t+1}\mid\mathbf{s}_{t},\mathbf{a}_{t}]=\hat{P}(\mathbf{x}_{t},\mathbf{a}_{t}), (2) where \mathbf{x}_{t}=F_{\phi}(\mathbf{s}_{t}) . The RP condition ensures that the latent state \mathbf{x} contains sufficient information to predict the reward function, while the ZP condition enforces temporal consistency in the latent space \mathcal{X} . However, satisfying the ZP condition alone admits a trivial solution in which the encoder maps all states to a constant vector, resulting in complete representation collapse ( 39 ) . When both RP and ZP conditions are satisfied, the encoder retains the necessary information for predicting returns. 3 Method \mathcal{D}^{(1)} \mathcal{D}^{(2)} \mathcal{D}^{(N)} Task datasets \{\mathbf{s}_{j},\mathbf{a}_{j},r_{j},\mathbf{s}_{j+1}\} Context encoder E_{\theta} \mathbf{z} Latent code \mathbf{c}_{0}\vphantom{\hat{\mathbf{c}}_{{1}}} code idx D_{\phi}(\hat{\mathbf{c}}_{{1}}\mid{\mathbf{c}}_{0},\mathbf{a}_{0},\mathbf{z}) Latent code \hat{\mathbf{c}}_{{1}} code idx D_{\phi}(\hat{\mathbf{c}}_{{2}}\mid\hat{\mathbf{c}}_{{1}},\mathbf{a}_{{1}},\mathbf{z}) Encoder F_{\phi} Encoder F_{\phi} Encoder F_{\phi} Codebook index Codebook index \mathbf{a}_{0} \mathbf{a}_{{1}} r_{0} r_{{1}} \mathbf{s}_{0} \mathbf{s}_{{1}} \mathbf{s}_{{2}} Dynamics ST Gumbel-softmax sampling Dynamics FSQ Cross-entropy loss Cross-entropy loss \mathbf{s}_{t} Encoder F_{\phi} Latent code \mathbf{c}_{t} FSQ \pi_{\eta}(\mathbf{c}_{t},\mathbf{z}) Q_{\psi}(\mathbf{c}_{t},\mathbf{a}_{t},\mathbf{z}) Figure 1: Method overview. Left: A context encoder E_{\theta} maps transitions from each task to a task representation \mathbf{z} , which serves as an implicit task identifier. Middle: An observation encoder maps observations \mathbf{s}_{t} to discrete latent vectors \mathbf{c}_{t} using finite scalar quantization (FSQ). A task-conditioned latent dynamics model D_{\phi} predicts future discrete latent states given the current discrete latent state, action, and task representation. The world model is trained using a classification loss based on temporal consistency. Right: An offline policy is trained using the discrete latent states \mathbf{c}_{t} and task representation \mathbf{z} . We now describe our method: SPC . Figures 1 and 1 provide a high-level overview. Our approach consists of three parts: task inference using a context encoder, learning a task-conditioned world model, and offline policy optimization. Algorithm 1 Self-Predictive Contextual OMRL ( SPC ) 1: Input: offline meta-dataset \{\mathcal{D}_{i}\}_{i=1}^{N} 2: Parameters: world model (\theta,\phi) ; IQL (\psi,\omega,\eta) 3: Initialize \bar{\mathbf{z}}^{i}=\mathbf{0} 4: repeat 5: Sample a meta-batch of tasks \mathcal{I}\subset\{1,\dots,N\} 6: for each task i\in\mathcal{I} do 7: Sample a context set \mathcal{C}_{i}\subset\mathcal{D}_{i} 8: Compute task representation \displaystyle\mathbf{z}^{i}\leftarrow\mathbb{E}_{(\mathbf{s},\mathbf{a},r,\mathbf{s}^{\prime})\sim\mathcal{C}_{i}}\big[E_{\theta}(\mathbf{s},\mathbf{a},r,\mathbf{s}^{\prime})\big] (3) 9: Sample trajectory \{(\mathbf{s}_{t+h},\mathbf{a}_{t+h},r_{t+h})\}_{h=0}^{H}\sim\mathcal{D}_{i} 10: end for 11: World model (inc. context encoder) update: \displaystyle(\theta,\phi)\leftarrow(\theta,\phi)-(\alpha_{\theta},\alpha_{\phi})\nabla_{\theta,\phi}\displaystyle{\sum_{i\in\mathcal{I}}}\mathcal{L}_{\text{TC}}^{i} (4) 12: Context encoder update: \displaystyle\theta\leftarrow\theta-\alpha_{\theta}\nabla_{\theta}\big(\beta\mathcal{L}_{\text{Contrastive}}\big) (5) 13: Value update (IQL): \omega\leftarrow\omega-\alpha_{\omega}\nabla_{\omega}\mathcal{L}_{V} 14: Critic update: \psi\leftarrow\psi-\alpha_{\psi}\nabla_{\psi}\mathcal{L}_{Q} 15: Policy update: \eta\leftarrow\eta-\alpha_{\eta}\nabla_{\eta}\mathcal{L}_{\pi} 16: Update \bar{\mathbf{z}}^{i}=\lambda\mathbf{z}^{i}+(1-\lambda)\bar{\mathbf{z}}^{i} 17: until convergence Overview SPC consists of the following components: Context encoder: \displaystyle\mathbf{z} \displaystyle=\mathbb{E}[E_{\theta}(\mathbf{s}_{t},\mathbf{a}_{t},r_{t},\mathbf{s}_{t+1})] (6a) Obs. encoder: \displaystyle\mathbf{x}_{t} \displaystyle=F_{\phi}(\mathbf{s}_{t}) (6b) Quantization: \displaystyle\mathbf{c}_{t} \displaystyle=f(\mathbf{x}_{t}) (6c) Latent dynamics: \displaystyle\hat{\mathbf{c}}_{t+1} \displaystyle\sim D_{\phi}(\mathbf{c}_{t},\mathbf{a}_{t},\mathbf{z}) (6d) Reward model: \displaystyle\hat{r}_{t} \displaystyle=R_{\phi}(\mathbf{c}_{t},\mathbf{a}_{t},\mathbf{z}) (6e) Q-function: \displaystyle Q \displaystyle=Q_{\psi}(\mathbf{c}_{t},\mathbf{a}_{t},\mathbf{z}) (6f) Value function: \displaystyle V \displaystyle=V_{\omega}(\mathbf{c}_{t},\mathbf{z}) (6g) Policy: \displaystyle\mathbf{a}_{t} \displaystyle\sim\pi_{\eta}(\mathbf{a}_{t}\mid\mathbf{c}_{t},\mathbf{z}) (6h) The latent world model consists of the context encoder E_{\theta} , observation encoder F_{\phi} , quantization module f , latent dynamics D_{\phi} , and reward model R_{\phi} . We denote the combined encoding and quantization as \Phi(\mathbf{s})\coloneqq f(F_{\phi}(\mathbf{s})) . The policy \pi_{\eta} , value function V_{\omega} , and Q-function Q_{\psi} are conditioned on both the discrete latent state \mathbf{c}_{t} and the task representation \mathbf{z} . Unlike prior work, we jointly train the world model and the context encoder, allowing task inference to benefit directly from temporal consistency. Importantly, we avoid observation reconstruction and benefit from the powerful representation learning provided by a self-predictive loss. 3.1 Contextual Discrete Codebook World Model In context-based OMRL, a context encoder E_{\theta} infers the task from a small set of transitions. The observation encoder F_{\phi} and quantization f are then shared across all tasks, while the latent dynamics D_{\phi} and reward model R_{\phi} are conditioned on the inferred task representation \mathbf{z} . This reflects the fact that tasks differ in dynamics and reward functions. An important feature of SPC is the discrete latent embedding realized with codebook embeddings, which 40 showed led to superior performance. Task representation. During training, we sample a meta-batch of tasks. For each task i with dataset \mathcal{D}_{i} , the task representation is given by \mathbf{z}^{i}=\mathbb{E}_{(\mathbf{s}_{t},\mathbf{a}_{t},r_{t},\mathbf{s}_{t+1})\sim\mathcal{D}_{i}}\left[E_{\theta}(\mathbf{s}_{t},\mathbf{a}_{t},r_{t},\mathbf{s}_{t+1})\right]. (7) World model. Observations from all tasks are mapped to continuous latent vectors using Equation 6b and discretized using finite scalar quantization (FSQ). See Appendix A for further details on FSQ. Because the latent space is discrete, the transition dynamics are modeled as categorical distributions over the next latent state. In contrast to self-supervised world models like TD-MPC2 ( 17 ) , this equips our model with the expressiveness to learn stochastic and multimodal transition dynamics, which we show has a big impact on performance. The latent dynamics model outputs logits over possible next codes \mathbf{c}_{t+1} conditioned on the current latent state \mathbf{c}_{t} , action \mathbf{a}_{t} , and task representation \mathbf{z} . To enable backpropagation through time, sampling is performed using the straight-through Gumbel-Softmax estimator ( 21 ) . Rewards are predicted by a task-conditioned reward model. We train the world model – context encoder, observation encoder, latent dynamics, and reward function – using a multi-step temporal consistency objective, given by \mathcal{L}_{\text{TC}}(\theta,\phi)=\sum_{h=0}^{H-1}\gamma^{h}\Big(\mathrm{CE}(D_{\phi}(\hat{\mathbf{c}}_{t+h},\mathbf{a}_{t+h},\mathbf{z}),\mathbf{c}_{t+h+1})\\ +\|R_{\phi}(\hat{\mathbf{c}}_{t+h},\mathbf{a}_{t+h},\mathbf{z})-r_{t+h}\|_{2}^{2}\Big), (8) where \hat{\mathbf{c}}_{t}=f(F_{\phi}(\mathbf{s}_{t})) is the initial latent state, \hat{\mathbf{c}}_{t+h+1} is sampled from the latent dynamics model, and \mathbf{c}_{t+h+1}=\mathrm{sg}(f(F_{\bar{\phi}}(\mathbf{s}_{t+h+1}))) is the target latent state computed using an exponential moving average (EMA) encoder. Here, H is the prediction horizon, \gamma is the discount factor, \bar{\phi} denotes EMA parameters, and \mathrm{CE} is the cross-entropy loss. Contrastive loss. To encourage task discrimination, we also train the context encoder with a contrastive objective. Intuitively, transitions from the same task should produce similar task representations, while representations from different tasks should be dissimilar. We use the InfoNCE loss: \displaystyle\mathcal{L}_{\text{Contrastive}}(\theta)=-\sum_{i}\log\frac{S(\mathbf{z}^{i},\bar{\mathbf{z}}^{i})}{\sum_{j}S(\mathbf{z}^{i},\bar{\mathbf{z}}^{j})}, (9) where \bar{\mathbf{z}}^{i}=\lambda\mathbf{z}^{i}+(1-\lambda)\bar{\mathbf{z}}^{i} is the moving average of task representations controlled by \lambda , and S(\mathbf{z}^{i},\mathbf{z}^{j})=\exp(-\|\mathbf{z}^{i}-\mathbf{z}^{j}\|_{2}^{2}/\alpha) measures similarity. Positive samples are obtained from the moving average of the same task representation, while negatives are drawn from other tasks. The moving average stabilizes training by smoothing updates. This objective provides a lower bound on the mutual information between tasks and task representations, I(\mathbf{z};\mathcal{M}) ( 51 ) . The context encoder is trained using the combined objective \displaystyle\mathcal{L}_{\text{Context}}(\theta)=\mathcal{L}_{\text{TC}}(\theta)+\beta\mathcal{L}_{\text{Contrastive}}(\theta), (10) where \beta controls the strength of the contrastive loss. Discussion. We show that the self-supervised loss in Equation 8 provides a strong learning signal for both the task representation and the latent state space. In particular, it substantially outperforms approaches based on reconstructing observations \mathbf{s} . Intuitively, the loss encourages the task representation to be sufficient for predicting future latent states and rewards within a given MDP – precisely the information required to solve the RL problem in the induced latent MDP. We provide a formal justification of this intuition in Section 3.3 . Additionally, we find that incorporating the contrastive loss in Equation 9 improves task discrimination. This loss encourages dissimilar tasks to occupy distinct regions of the task-representation space, which we hypothesize facilitates learning separate latent MDPs for such tasks. 3.2 Meta Policy Optimization After learning the world model, we train a policy using offline reinforcement learning. We condition the policy and value functions on the discrete latent state and task representation. To address distribution shift in offline RL, we adopt Implicit Q-Learning (IQL, 24 ). In Section D.7 , we show evidence that SPC is compatible with other offline RL methods. IQL avoids out-of-distribution actions by learning a value function using expectile regression \displaystyle\mathcal{L}_{V}(\omega)=\mathbb{E}\big[\mathcal{L}_{2}^{\tau}(Q_{\bar{\psi}}(\mathbf{c}_{t},\mathbf{a}_{t},\mathbf{z})-V_{\omega}(\mathbf{c}_{t},\mathbf{z}))\big], (11) where \mathcal{L}_{2}^{\tau}(x)=(\tau-\mathbb{1}[x<0])x^{2} and \bar{\psi} denotes EMA parameters. The Q-function is trained using standard temporal-difference learning. The policy is optimized using advantage-weighted regression ( 37 ) , given by \displaystyle\mathcal{L}_{\pi}(\eta)=\mathbb{E}\left[\exp(A(\mathbf{c}_{t},\mathbf{a}_{t},\mathbf{z})/\mathcal{B})\log\pi_{\eta}(\mathbf{a}_{t}\mid\mathbf{c}_{t},\mathbf{z})\right], (12) where A(\mathbf{c}_{t},\mathbf{a}_{t},\mathbf{z})=Q_{\psi}(\mathbf{c}_{t},\mathbf{a}_{t},\mathbf{z})-V_{\omega}(\mathbf{c}_{t},\mathbf{z}) is the advantage and \mathcal{B} is a temperature parameter. It is worth noting that the latent world model is used solely for representation learning and not for planning. Specifically, observations are encoded into a latent space via \mathbf{c}=\Phi(\mathbf{s}) , and both the policy and value functions are learned directly over these latent states. Consequently, learning proceeds within the latent MDP induced by the encoder \Phi . 3.3 Theoretical Analysis Our meta policy optimization operates in the learned latent space (\mathbf{c}_{t},\mathbf{z}_{t}) without reconstructing observations. Here we build on the well-known simulation lemma ( 22 ) and bound the value error incurred by (i) latent abstraction via \Phi(\mathbf{s}) , (ii) learned world-model error, and (iii) task inference via the context encoder. All proofs are deferred to Appendix C . Setup. For task i with MDP \mathcal{M}_{i}=\langle\mathcal{S},\mathcal{A},R_{i},P_{i},\gamma,\rho_{0}\rangle and bounded rewards |R_{i}(\mathbf{s},\mathbf{a})|\leq R_{\max} . Let the learned latent state be \mathbf{c}=\Phi(\mathbf{s})\coloneqq f(F_{\phi}(\mathbf{s}))\in\mathcal{C} . Given (\mathbf{c},\mathbf{a},\mathbf{z}) , the learned dynamics model induces a kernel \widehat{P}_{\phi}(\cdot\mid\mathbf{c},\mathbf{a},\mathbf{z}) over \mathcal{C} and the learned reward model outputs \widehat{R}_{\phi}(\mathbf{c},\mathbf{a},\mathbf{z}) . We use the infinite-horizon discounted value in MDP \mathcal{M}_{i} as \displaystyle V_{\mathcal{M}_{i}}^{\pi}(\mathbf{s})=\mathbb{E}\bigg[\sum_{t\geq 0}\gamma^{t}R_{i}(\mathbf{s}_{t},\mathbf{a}_{t})\mid\mathbf{s}_{0}=\mathbf{s},\ \mathbf{a}_{t}\sim\pi(\cdot\mid\mathbf{s}_{t})\bigg]. Sources of error. We quantify three sources of error: (1) Latent abstraction error. Let P_{i}^{\Phi}(\mathbf{c}^{\prime}\mid\mathbf{s},\mathbf{a}) be the pushforward transition induced by \Phi (Appendix C ). We assume there exists a Markov kernel \bar{P}_{i}^{\Phi}(\cdot\mid\mathbf{c},\mathbf{a}) and latent reward function \bar{R}_{i}^{\Phi}(\mathbf{c},\mathbf{a}) such that \displaystyle\epsilon_{\mathrm{abs}}^{P} \displaystyle\coloneq\sup_{\mathbf{s},\mathbf{a}}\big\|P_{i}^{\Phi}(\cdot\mid\mathbf{s},\mathbf{a})-\bar{P}_{i}^{\Phi}(\cdot\mid\Phi(\mathbf{s}),\mathbf{a})\big\|_{1}, (13a) \displaystyle\epsilon_{\mathrm{abs}}^{R} \displaystyle\coloneq\sup_{\mathbf{s},\mathbf{a}}\big|R_{i}(\mathbf{s},\mathbf{a})-\bar{R}_{i}^{\Phi}(\Phi(\mathbf{s}),\mathbf{a})\big|. (13b) (2) World model approximation error (true task). When conditioned on the true task i , define \displaystyle\epsilon_{\mathrm{model}}^{P} \displaystyle\coloneq\sup_{\mathbf{c},\mathbf{a}}\big\|\widehat{P}_{\phi}(\cdot\mid\mathbf{c},\mathbf{a},i)-\bar{P}_{i}^{\Phi}(\cdot\mid\mathbf{c},\mathbf{a})\big\|_{1}, (14a) \displaystyle\epsilon_{\mathrm{model}}^{R} \displaystyle\coloneq\sup_{\mathbf{c},\mathbf{a}}\big|\widehat{R}_{\phi}(\mathbf{c},\mathbf{a},i)-\bar{R}_{i}^{\Phi}(\mathbf{c},\mathbf{a})\big|. (14b) (3) Task inference error (conditioning on \mathbf{z} ). Let \mathbf{z}=g_{\theta}(\mathcal{H}) be the task representation inferred from history \mathcal{H} collected in task i . Define \displaystyle\epsilon_{\mathrm{task}}^{P} \displaystyle\coloneq\sup_{\mathbf{c},\mathbf{a}}\big\|\widehat{P}_{\phi}(\cdot\mid\mathbf{c},\mathbf{a},\mathbf{z})-\widehat{P}_{\phi}(\cdot\mid\mathbf{c},\mathbf{a},i)\big\|_{1}, (15a) \displaystyle\epsilon_{\mathrm{task}}^{R} \displaystyle\coloneq\sup_{\mathbf{c},\mathbf{a}}\big|\widehat{R}_{\phi}(\mathbf{c},\mathbf{a},\mathbf{z})-\widehat{R}_{\phi}(\mathbf{c},\mathbf{a},i)\big|. (15b) Value error bound. Let \pi(\mathbf{a}\mid\mathbf{c},\mathbf{z}) be any latent policy and define the lifted policy \pi^{\Phi}(\mathbf{a}\mid\mathbf{s},\mathbf{z})\coloneqq\pi(\mathbf{a}\mid\Phi(\mathbf{s}),\mathbf{z}) . Let V_{\mathcal{M}_{i}}^{\pi^{\Phi}} be its value in the original MDP and let V_{\widehat{\mathcal{M}}(\mathbf{z})}^{\pi} be its value in the learned latent MDP induced by (\widehat{P}_{\phi}(\cdot\mid\mathbf{c},\mathbf{a},\mathbf{z}),\widehat{R}_{\phi}(\mathbf{c},\mathbf{a},\mathbf{z})) . Theorem 3.1 (Value error from abstraction, model learning, and task inference) . For any task i and any latent policy \pi , \displaystyle\sup_{\mathbf{s}\in\mathcal{S}}\Big|V_{\mathcal{M}_{i}}^{\pi^{\Phi}}(\mathbf{s}) \displaystyle-V_{\widehat{\mathcal{M}}(\mathbf{z})}^{\pi}(\Phi(\mathbf{s}))\Big|\leq\frac{\epsilon^{R}}{1-\gamma}+\frac{\gamma R_{\max}}{(1-\gamma)^{2}}\epsilon^{P}, \displaystyle\text{with }\qquad\epsilon^{R} \displaystyle=\epsilon_{\mathrm{abs}}^{R}+\epsilon_{\mathrm{model}}^{R}+\epsilon_{\mathrm{task}}^{R}, (16a) \displaystyle\epsilon^{P} \displaystyle=\epsilon_{\mathrm{abs}}^{P}+\epsilon_{\mathrm{model}}^{P}+\epsilon_{\mathrm{task}}^{P}. (16b) Interpretation. The bound decomposes value error into: (i) latent abstraction error (how non-Markov the code \mathbf{c} is), (ii) world model error (how accurately D_{\phi} and R_{\phi} match the induced latent MDP), and (iii) task inference error (how well \mathbf{z} substitutes for the true task ID for predicting latent dynamics and rewards). Implications for representation learning. Crucially, the bound in Theorem 3.1 does not require reconstructing observations from the latent state. Unlike reconstruction-based world models, which aim to preserve all information in \mathbf{s}_{t} , our analysis only requires that the learned representations (\mathbf{c},\mathbf{z}) preserve information for predictive control. Concretely, the abstraction error (\epsilon_{\mathrm{abs}}^{P},\epsilon_{\mathrm{abs}}^{R}) is governed by how well the observation encoder \Phi groups states with similar future latent dynamics and rewards. Our temporal consistency loss directly targets this by enforcing that \mathbf{c}_{t+1} is predictable from (\mathbf{c}_{t},\mathbf{a}_{t},\mathbf{z}) , rather than from raw observations. The world model error (\epsilon_{\mathrm{model}}^{P},\epsilon_{\mathrm{model}}^{R}) is reduced by training the latent dynamics and reward models to accurately predict future latent states and rewards under the inferred task. Finally, the task inference error (\epsilon_{\mathrm{task}}^{P},\epsilon_{\mathrm{task}}^{R}) is controlled by jointly training the context encoder with the world model, encouraging \mathbf{z} to capture precisely the task-dependent factors needed for prediction. Together, these objectives yield a compact latent representation that is sufficient for planning and control, without requiring reconstruction of \mathbf{s}_{t} . We provide a more detailed theoretical analysis in Appendix C . Figure 2 : Representation metrics (Feature Rank, Rank, and Dormant ratio) of the context encoder. SPC maintains low dormant neuron ratio and high matrix rank while learning more diverse features (higher feature rank) compared to pure reconstruction (UNICORN-SUP). The shaded area represents 95% confidence interval across 6 random seeds. 4 Experiments We evaluate SPC on multi-task benchmarks from MuJoCo ( 44 ) , Contextual DeepMind Control (DMC) ( 45 ; 38 ) , and Meta-World ( 48 ) . Our experiments address the following questions: RQ1 How does learning the context encoder jointly with the world model’s self-supervised temporal consistency loss affect the task representation? RQ2 How well does SPC generalize to unseen tasks and environments compared to prior methods? RQ3 How do different latent space formulations (classification vs. regression, continuous vs. discrete, bounded vs. unbounded) affect performance? Baselines. We compare SPC against the following context-based OMRL methods: • FOCAL ( 26 ) trains the context encoder using a distance-based objective that pulls representations from the same task together and pushes different tasks apart. • CSRO ( 14 ) extends FOCAL by reducing context distribution shift using CLUB as a mutual information regularizer ( 5 ) . • DORA ( 51 ) uses the InfoNCE loss to train the context encoder. In contrast, we train the context encoder jointly with the world model where the context encoder benefits from the world model’s self-supervised temporal consistency loss. • UNICORN trains a context encoder using task-conditioned dynamics and reward models. UNICORN-SS further augments this with a contrastive objective, whereas UNICORN-SUP relies exclusively on reconstruction. Unlike our approach, UNICORN does not learn a latent state representation and instead performs prediction directly in the observation space. Datasets. We generate offline datasets using Dropout Q-learning (DroQ) ( 19 ) , training a separate agent for each task. Each dataset contains 1000 trajectories collected at different stages of training, ranging from random behavior to near-expert performance over 1M environment steps. Benchmarks. For MuJoCo and Contextual-DMC, we use 20 training tasks, 10 in-distribution test tasks, and 10 out-of-distribution test tasks. Out-of-distribution tasks differ in the ranges of underlying variation factors. In Meta-World, we follow standard Meta-RL settings (ML1, ML10, ML45). ML1 evaluates generalization across goals within a single environment. ML10 and ML45 evaluate generalization to unseen environments without task identifiers, using 10 and 45 training environments respectively, and 5 held-out test environments. 4.1 Investigating Task Representation Learning Figure 3 : Disentanglement metrics (DCI, InfoMEC) for the Cheetah-length-speed (Ls) environment. Latent world models disentangle the variation factors more effectively , while contrastive learning enhances task distinguishability , reflected in informativeness and explicitness. TC denotes training the context encoder solely with the latent temporal consistency objective ( Equation 8 ); FOCAL and InfoNCE represent two contrastive objectives, and UNICORN-SUP indicates training with reconstruction. Averaged over 6 random seeds. We first study how different training objectives affect the quality of the learned task representations. Previous work often relies on t-SNE visualizations to assess task separation, but such visualizations are qualitative and sensitive to hyperparameters.We instead use disentanglement metrics, which measure how well latent dimensions align with underlying task variation factors. We report DCI ( 8 ) metrics (disentanglement, completeness, informativeness) and InfoMEC ( 20 ) metrics (modularity, explicitness, compactness). Together, these metrics capture whether a task representation is structured, informative, and aligned with true task factors. Figure 3 reports results for the Cheetah-LS environment, which varies desired speed and torso length. Training the context encoder using latent temporal consistency ( TC ) leads to better disentanglement than reconstruction-based training (UNICORN-SUP). Adding contrastive objectives improves task distinguishability, reflected in higher informativeness and explicitness. Combining both objectives yields the strongest overall representations. Section D.3 presents disentanglement metrics for more environments, demonstrating consistent behavior. We also analyze representation collapse using feature rank, matrix rank, and dormant neuron ratio ( Figure 2 ). The matrix rank ( 39 ) quantifies correlations among learned features, while the dormant neuron ratio ( 43 ; 7 ) measures the proportion of neurons with near-zero activations, which limits the expressivity of the network. Given the representation matrix – defined as the activations of the last hidden layer of the context encoder – the feature rank ( 29 ; 1 ; 32 ; 35 ) measures feature diversity by identifying the smallest subspace that preserves (1-\varepsilon)\% of the total variance; throughout our experiments, we set \varepsilon=0.01 . Self-prediction learns more diverse and less correlated features than reconstruction (UNICORN SUP), maintains higher matrix rank, and significantly reduces dormant neurons, suggesting that latent temporal consistency leads to more expressive and robust task representations ( 7 ) . 4.2 Generalization to New Tasks and Environments Finally, we evaluate whether improved task representations lead to better generalization. During meta-testing, the agent is not given task identifiers and must infer the task online from interaction data. The task representation is initialized to zero and updated at each step using collected transitions. Figure 5 and Figure 6 report few-shot performance on in-distribution tasks. SPC outperforms prior methods, demonstrating faster adaptation and higher returns. These results suggest that training the context encoder with both the latent temporal consistency and contrastive objectives, while leveraging a discrete latent world model, improves both in-distribution performance and generalization. Section D.1 reports few-shot performance across benchmarks as well as zero-shot results on MuJoCo and Contextual-DMC. We also evaluate generalization to unseen environments in Meta-World ML10 and ML45 ( Figure 7 ). SPC achieves higher success rates on both training and test environments, although generalization to entirely new environments remains challenging for all methods. Increasing the diversity of training environments improves test performance. 4.3 Comparison of Different Latent Spaces Figure 4 : Different world model formulations: IQM and optimality gap of the normalized return for different world modeling methods, evaluated on 9 environments with 6 random seeds per environment. The main advantage of discretizing the latent space is due to classification loss (cross entropy). Bounding or discretizing the latent space alone does not improve performance. We next isolate the effect of different latent space formulations. Prior work on RL uses continuous latent spaces with regression losses ( 18 ) , cosine similarity ( 52 ) , or bounded and discretized representations ( 39 ; 17 ) . Figure 4 shows that simply bounding or discretizing the latent space does not improve performance in our setting. In contrast, formulating temporal consistency as a classification problem consistently leads to better results. This suggests that the cross-entropy loss, rather than discretization itself, is the main driver of performance gains. Similarly, 9 demonstrates the advantages of classification compared to regression in training value functions. We provide a detailed comparison in Table 11 . 4.4 Further Results Figure 5 : Few-shot in-distribution performance on MuJoCo and Contextual DMC benchmarks: IQM and optimality gap of the normalized return (9 environments, each with 6 random seeds). Figure 6 : Few-shot in-distribution performance on Meta-world benchmark: IQM and optimality gap of the success rates (30 environments, each with 6 random seeds). Appendix D in the appendix presents further empirical results including comparisons with DreamerV3 ( Section D.2 ), scaling behavior with model size ( Section D.6 ), ablations on the choice of offline RL algorithm ( Section D.7 ), bounding the task representation \mathbf{z} ( Section D.8 ), and a comparison of computational cost with baseline methods ( Section D.9 ). Section D.5 further analyzes the impact of contrastive learning and latent temporal consistency objectives on generalization. 5 Related Work We review latent world models, self-predictive RL, and context-based offline meta-RL, highlighting how our method bridges gaps between these directions. Latent world models and self-predictive RL. Latent world models learn compact state representations by mapping observations to a latent state space and learning to predict future latent states and rewards. They have been successfully used in model-based RL ( 15 ; 41 ; 18 ; 17 ; 40 ) , and for representation learning in model-free RL ( 52 ; 11 ; 39 ; 12 ) . Across both settings, a unifying principle is self-predictive RL ( 42 ; 52 ) , which provides a strong self-supervised signal for learning latent state spaces. Recent works have also shown that reconstruction is unnecessary and that enforcing multi-step latent consistency leads to improved performance ( 18 ; 40 ; 39 ) . However, these methods are typically task-specific or assume access to a known task identifier, and therefore do not address generalization across tasks or environments. Context-based offline meta-RL. Context-based offline meta-RL (OMRL) methods aim to generalize to unseen tasks by learning a task representation from a context of past transitions. Prior work trains context encoders using contrastive objectives ( 26 ; 49 ) , but such approaches are sensitive to context distribution shift between training and testing. CSRO ( 14 ) and ER-TRL ( 33 ) mitigate this shift by regularizing the dependence between task representations and behavior policies. UNICORN ( 27 ) addresses task inference via predictive objectives, but operates directly in the observation space and does not learn a latent state space for control. We unify these two lines of work. Unlike prior OMRL methods, we learn a latent state space using self-predictive world models without reconstructing observations. Unlike prior self-predictive RL methods, we jointly learn the latent dynamics and the task representation. By training the world model and context encoder together, both the latent state \mathbf{c} and task representation \mathbf{z} capture the task-dependent predictive structure required for control. Figure 7 : Generalization to new environments: SPC demonstrates better generalization to unseen environments. Increasing the number of training environments can improve generalization to testing environments . Averaged over 6 random seeds. 6 Conclusion This paper introduces Self-Predictive Contextual OMRL, which conditions latent world models on learned task representations and jointly trains the world model and context encoder using temporal consistency and contrastive learning. The resulting self-supervised representations capture task-relevant variation while avoiding representation collapse, leading to improved generalization to unseen tasks. We use the learned discrete latent state and task representation for offline policy optimization, outperforming prior methods. We further provide value error bounds in the latent MDP induced by the observation encoder, showing that accurate control is possible without observation reconstruction. Overall, our results demonstrate that predictive latent representations suffice for generalization in offline meta-RL, and that jointly learning the context encoder and world model is both principled and effective. Impact Statement This paper presents work whose goal is to advance the field of Machine Learning. There are many potential societal consequences of our work, none which we feel must be specifically highlighted here. Appendices In this appendix, we provide further background in Appendix A , implementation details in Appendix B , theoretical anaylsis in Appendix C and additional results in Appendix D . Large Language Models We use large language models (LLMs) to assist with paper writing, including proofreading for typos and grammar errors. We also employ LLMs to generate scripts for visualizing results and creating figures. Appendix A Background In this section, we provide further background on finite scalar quantization (FSQ), which we adopted from 40 . A.1 Finite Scalar Quantization Quantization aims to compress continuous representations by mapping them to a discrete set of codes. Finite scalar quantization (FSQ, 30 ) discretizes the latent space by independently quantizing each latent dimension, forming a fixed grid structure. Given a continuous latent vector \mathbf{x}\in\mathbb{R}^{d} , FSQ reshapes it into \mathbf{X}^{\prime}\in\mathbb{R}^{d^{\prime}\times b} , where b is the number of channels and d^{\prime}=d/b . Each channel is quantized independently using a predefined number of quantization levels. Let \mathrm{L}=[L_{1},\ldots,L_{b}] denote the resolution for each channel. Quantization is performed as \displaystyle\mathbf{C}^{\prime}_{:,i}=\mathrm{round}\!\left(\left\lfloor\tfrac{L_{i}}{2}\right\rfloor\tanh(\mathbf{X}^{\prime}_{:,i})\right), (17) which maps each channel to L_{i} discrete values. The quantized representation \mathbf{C}^{\prime} is then reshaped back to \mathbf{c}\in\mathbb{R}^{d} . To allow gradient propagation through the non-differentiable rounding operation, FSQ uses the straight-through estimator (STE, 4 ). Unlike vector quantization methods such as VQ-VAE ( 46 ) , FSQ does not require separate codebook or commitment losses. This results in a simple, efficient, and stable discretization method. Figure 8 provides an example of discretization with FSQ. Figure 8 : Illustration of FSQ with two levels \mathrm{L}=[5,3] The continuous vector \mathbf{x} is first reshaped into |\mathrm{L}| channels, each channel b_{i} is quantized to the nearest integer according to its resolution L_{i} , and the resulting quantized representation is then mapped back to the original dimensionality of \mathbf{x} . –appendices continue on next page– Appendix B Implementation Details We implemented SPC with PyTorch ( 36 ) and used the AdamW optimizer ( 28 ) for training the world model and the Adam optimizer ( 23 ) for the other models. All neural networks are implemented as MLPs where each intermediate linear layer is followed by Layer Normalization ( i.e. pre-norm) ( 2 ) and a Mish activation function ( 31 ) . Below we summarize the architecture of SPC for the Cheetah-LS environment. ⬇ Context Encoder : MLP ( ( net ): Sequential ( (0): NormedLinear ( in_features =41, out_features =256, bias = True , act = Mish ) (1): NormedLinear ( in_features =256, out_features =256, bias = True , act = Mish ) (2): NormedLinear ( in_features =256, out_features =256, bias = True , act = Mish ) (3): Linear ( in_features =256, out_features =10, bias = True ) ) ) World Model : ContextualWorldModel ( ( Fsq ): FSQ ( levels =[5, 3]) ( Encoder ): MLP ( ( net ): Sequential ( (0): NormedLinear ( in_features =17, out_features =512, bias = True , act = Mish ) (1): Linear ( in_features =512, out_features =1024, bias = True ) ) ) ( Encoder_tar ): MLP ( ( net ): Sequential ( (0): NormedLinear ( in_features =17, out_features =512, bias = True , act = Mish ) (1): Linear ( in_features =512, out_features =1024, bias = True ) ) ) ( Trans ): MLP ( ( net ): Sequential ( (0): NormedLinear ( in_features =1040, out_features =512, bias = True , act = Mish ) (1): NormedLinear ( in_features =512, out_features =512, bias = True , act = Mish ) (2): Linear ( in_features =512, out_features =7680, bias = True ) ) ) ( Reward ): Vectorized [ MLP ( ( net ): Sequential ( (0): NormedLinear ( in_features =1040, out_features =512, bias = True , act = Mish ) (1): NormedLinear ( in_features =512, out_features =512, bias = True , act = Mish ) (2): Linear ( in_features =512, out_features =1, bias = True ) ) )] ) Policy : MLP ( ( net ): Sequential ( (0): NormedLinear ( in_features =1034, out_features =256, bias = True , act = Mish ) (1): NormedLinear ( in_features =256, out_features =256, bias = True , act = Mish ) (2): Linear ( in_features =256, out_features =12, bias = True ) ) ) Q - Functions : Vectorized [ MLP ( ( net ): Sequential ( (0): NormedLinear ( in_features =1040, out_features =256, bias = True , act = Mish ) (1): NormedLinear ( in_features =256, out_features =256, bias = True , act = Mish ) (2): Linear ( in_features =256, out_features =1, bias = True ) ) ), MLP ( ( net ): Sequential ( (0): NormedLinear ( in_features =1040, out_features =256, bias = True , act = Mish ) (1): NormedLinear ( in_features =256, out_features =256, bias = True , act = Mish ) (2): Linear ( in_features =256, out_features =1, bias = True ) ) )] Value Function : MLP ( ( net ): Sequential ( (0): NormedLinear ( in_features =1034, out_features =256, bias = True , act = Mish ) (1): NormedLinear ( in_features =256, out_features =256, bias = True , act = Mish ) (2): Linear ( in_features =256, out_features =1, bias = True ) ) ) Learnable parameters : 7.55 M Hardware We used AMD Instinct MI250X GPUs to run our experiments. All experiments have been run on a single GPU with 2 CPU workers and 32GB of RAM. Hyperparameters Table 1 illustrates the hyperparameters for our experiments. We use the same hyperparameters for all of the experiments. For a fair comparison, we use the same network architecture for all the baselines. Table 1 : Hyperparameters of our method SPC . Hyperparameter Value Description Data Collection Train steps 10^{6} Random steps 5\times 10^{4} Num. random steps at start Num. eval episodes 50 Num. trajectories in evaluation Eval. every steps 5\times 10^{4} Policy MLP dims [512,512] Value Function MLP dims [512,512] Dropout ratio 0.1 Learning rate 10^{-4} Value function, entropy coeff 3\times 10^{-4} Policy Target Entropy -\|\mathcal{A}\|_{1} Batch size 1024 Discount factor \gamma 0.99 Momentum coef 0.005 Soft update target networks Contextual Latent World Model Observation Encoder MLP dims [512] Context Encoder MLP dims [256,256] Latent Dynamics and Reward MLP dims [512,512] Task representation dim 5 Latent dim 1024 FSQ Levels [5, 3] Consistency coeff 1.0 Reward Coeff 1.0 Discount Factor \gamma 0.99 Training Horizon H 5 Learning Rate 10^{-4} Contrastive Obj Weight \beta 1.0 Momentum coef 0.005 Soft update target encoder Offline Meta-RL Meta batch size 16 Batch size 256 Num. train task 20 mujoco & dmc 40 metaworld Num. eval task 10 Num. OOD task 10 mujoco & dmc 0 metaworld Context size 64 Buffer size 2\times 10^{5} for each task Discount factor \gamma 0.99 (Q-)Value Function MLP dims [256,256] Num. Q Functions 2 Policy MLP dims [256,256] Learning rate 3\times 10^{-4} Policy and (Q-)Value functions Expectile regression \tau 0.8 Inverse temperature \mathcal{B} 3.0 In policy optimization Momentum coef. 0.005 soft update target networks Num. test trajectories 3 k few-shot Environments We evaluated SPC on 3 MuJoCo ( 44 ) environments, 6 Contextual DeepMind Control ( 45 ; 38 ) environments, and 50 Meta-World ML1 ( 3 ) environments. Table 2 provides details of the environments we used, including the dimensionality of the observation and action spaces, and the distribution of variation factors. Below we summarize the environments: • Ant-direction: an ant (quadruped) robot moving in different desired directions in different tasks. • Hopper-mass: a hopper (one-legged robot) must move as fast as it can, while the mass is different for different tasks. • Walker-friction: a walker (bi-legged) robot must move as fast as it can, while the friction coefficient is different for different tasks. • Cheetah-speed: a cheetah robot moving forwards/backwards with different desired speeds in different tasks. • Finger-speed: a planar finger robot rotating a body on an unactuated hinge with different desired angular speeds (both directions) for different tasks. • Walker-speed: a walker robot moving forwards/backwards with different desired speeds in different tasks. • Cheetah-length-speed (LS): a cheetah robot moving forwards, where torso length (change in morphology) and/or desired speed are different for each task. • Finger-length-speed (LS): a planar finger robot rotating a body on an unactuated hinge, while the length of the link and/or desired angular speed differ in each task. • Walker-length-speed (LS): a walker robot moving forwards, where torso length (change in morphology) and/or desired speed are different for each task. • Meta-World ML1 : consists of 50 robotic manipulation environments featuring a Sawyer arm with various everyday objects. Each environment consists of 50 different tasks where the position of objects and goals is different for each task. • Meta-World ML10: evaluates generalization to new environments. Similar to the ML1 setting, it consists of robotic manipulation environments, where 10 environments are used for training and 5 environments are reserved for testing generalization capabilities. The testing environments share structural similarities with the training environments. During testing, no prior information about the environment (such as environment ID) is provided, and agents must identify and adapt to the environment solely based on interaction data. • Meta-World ML45: evaluates generalization to new environments, similar to the ML10 setting, but with a larger and more diverse set of 45 training environments. Table 2 : Environment used for evaluation of different methods. Environment Obs dim Action dim ID Variation Factors OOD Variation Factors ant-direction 29 8 \theta\sim[-\pi,\pi] \theta\sim[-1.5\pi,-\pi]\cup[\pi,1.5\pi] hopper-mass 11 3 \log f_{m}\sim[-1.5,1.5] \log f_{m}\sim[-2,-1.5]\cup[1.5,2] walker-friction 17 6 \log f_{f}\sim[-1.5,1.5] \log f_{f}\sim[-2,-1.5]\cup[1.5,2] cheetah-speed 17 6 v\sim[-10,-6]\cup[-2,2]\cup[6,10] v\sim[-6,-2]\cup[2,6] finger-speed 17 6 v\sim[-15,-9]\cup[-3,3]\cup[9,15] v\sim[-9,-3]\cup[3,9] walker-speed 24 6 v\sim[-5,-3]\cup[-1,1]\cup[3,5] v\sim[-3,-1]\cup[1,3] cheetah-length-speed 17 6 v\sim[3,8] v\in\{1,2,9,10\} L\sim[0.4,0.6] L\in\{0.3,0.35,0.65,0.7\} finger-length-speed 9 2 v\sim[5,10] v\in\{3,4,11,12\} L\sim[0.15,0.25] L\in\{0.1,0.12,0.27,0.3\} walker-length-speed 24 6 v\sim[2,4.5] v\in\{1,1.5,5,5.5\} L\sim[0.2,0.4] L\in\{0.1,0.15,0.45,0.5\} Meta-World 39 4 Open-source code For full details of the implementation, model architectures, and training, please check the code, which is available in the submitted supplementary material and will be made public upon acceptance to guarantee reproducibility. –appendices continue on next page– Appendix C Theoretical Analysis Our actor-critic operates in the learned latent space (\mathbf{c}_{t},\mathbf{z}_{t}) without reconstructing observations. We give conditions under which planning/control in (\mathbf{c},\mathbf{z}) is sufficient and derive value-error bounds. Our bounds build on the simulation lemma ( e.g. , 22 ), which relates value differences between two MDPs to discrepancies in reward and transition kernels. We extend it in two steps: (i) from the original task MDP to the latent MDP induced by the observation encoder \Phi , and (ii) from conditioning on the true task to conditioning on the learned context representation \mathbf{z} . C.1 Preliminaries and notation For each task i , let \mathcal{M}_{i}=\langle\mathcal{S},\mathcal{A},R_{i},P_{i},\gamma,\rho_{0}\rangle with bounded rewards |R_{i}(\mathbf{s},\mathbf{a})|\leq R_{\max} for all \mathbf{s} and \mathbf{a} . We use the infinite-horizon discounted value \displaystyle V_{i}^{\pi}(\mathbf{s})=\mathbb{E}\bigg[\sum_{t\geq 0}\gamma^{t}R_{i}(\mathbf{s}_{t},\mathbf{a}_{t})\mid\mathbf{s}_{0}=\mathbf{s},\ \mathbf{a}_{t}\sim\pi(\cdot\mid\mathbf{s}_{t})\bigg]. (18) Latent state and task representation. Our latent state is the discrete code \displaystyle\mathbf{c}=\Phi(\mathbf{s})\coloneqq f(F_{\phi}(\mathbf{s}))\in\mathcal{C}, (19) and the context encoder maps a context set \mathcal{H}_{t} of transitions to a task representation \displaystyle\mathbf{z}_{t}=g_{\theta}(\mathcal{H}_{t})\coloneqq\mathbb{E}_{(\mathbf{s},\mathbf{a},r,\mathbf{s}^{\prime})\sim\mathcal{H}_{t}}\big[E_{\theta}(\mathbf{s},\mathbf{a},r,\mathbf{s}^{\prime})\big]. (20) At test time \mathbf{z}_{t} may change as more context is accumulated. In the bounds below, we treat \mathbf{z} as fixed after a context set is collected (matching few-shot evaluation with a fixed context window). World model (task-conditioned). Given (\mathbf{c},\mathbf{a},\mathbf{z}) , the latent dynamics D_{\phi} defines a distribution \widehat{P}_{\phi}(\cdot\mid\mathbf{c},\mathbf{a},\mathbf{z}) over \mathcal{C} , and the reward model defines \widehat{R}_{\phi}(\mathbf{c},\mathbf{a},\mathbf{z}) . Norms. For distributions p,q over a finite set, \|p-q\|_{1}=\sum_{x}|p(x)-q(x)| . For bounded functions V:\mathcal{S}\to\mathbb{R} , \|V\|_{\infty}=\sup_{\mathbf{s}}|V(\mathbf{s})| . Simulation lemma. We use a standard simulation-lemma style inequality in total variation (TV). Lemma C.1 (Simulation lemma (TV form)) . Let M=\langle\mathcal{X},\mathcal{A},R,P,\gamma\rangle and \widehat{M}=\langle\mathcal{X},\mathcal{A},\widehat{R},\widehat{P},\gamma\rangle share the same state space \mathcal{X} and action space \mathcal{A} , with |R(\mathbf{x},\mathbf{a})|\leq R_{\max} and |\widehat{R}(\mathbf{x},\mathbf{a})|\leq R_{\max} for all (\mathbf{x},\mathbf{a}) . Assume for all (\mathbf{x},\mathbf{a}) : \displaystyle|R(\mathbf{x},\mathbf{a})-\widehat{R}(\mathbf{x},\mathbf{a})|\leq\epsilon_{R},\qquad\|P(\cdot\mid\mathbf{x},\mathbf{a})-\widehat{P}(\cdot\mid\mathbf{x},\mathbf{a})\|_{1}\leq\epsilon_{P}. (21) Then for any stationary policy \pi(\mathbf{a}\mid\mathbf{x}) , \displaystyle\|V_{M}^{\pi}-V_{\widehat{M}}^{\pi}\|_{\infty}\leq\frac{\epsilon_{R}}{1-\gamma}+\frac{\gamma R_{\max}}{(1-\gamma)^{2}}\,\epsilon_{P}. (22) Proof. See 22 . ∎ C.2 Step 1: From the original MDP to the latent MDP induced by \Phi Fix a task i and suppress i in notation when clear. Pushforward (latent) transition. Given P_{i} on \mathcal{S} and \Phi:\mathcal{S}\to\mathcal{C} , we define the pushforward transition for any (\mathbf{s},\mathbf{a}) as \displaystyle P_{i}^{\Phi}(\mathbf{c}^{\prime}\mid\mathbf{s},\mathbf{a})\coloneqq\sum_{\mathbf{s}^{\prime}\in\mathcal{S}}P_{i}(\mathbf{s}^{\prime}\mid\mathbf{s},\mathbf{a})\,\mathbbm{1}\{\Phi(\mathbf{s}^{\prime})=\mathbf{c}^{\prime}\}. (23) Latent Markov approximation. Because \Phi may be many-to-one, the induced process on \mathbf{c} need not be Markov. As such, we define a Markov approximation using conditional expectation within preimages of \Phi , given by \displaystyle\bar{P}_{i}^{\Phi}(\mathbf{c}^{\prime}\mid\mathbf{c},\mathbf{a}) \displaystyle\coloneqq\mathbb{E}\left[P_{i}^{\Phi}(\mathbf{c}^{\prime}\mid\mathbf{s},\mathbf{a})\mid\Phi(\mathbf{s})=\mathbf{c}\right], (24) \displaystyle\bar{R}_{i}^{\Phi}(\mathbf{c},\mathbf{a}) \displaystyle\coloneqq\mathbb{E}\left[R_{i}(\mathbf{s},\mathbf{a})\mid\Phi(\mathbf{s})=\mathbf{c}\right]. (25) Assumption C.2 . (Approximately Markov). There exist \epsilon_{\mathrm{abs}}^{P},\epsilon_{\mathrm{abs}}^{R}\geq 0 such that for all (\mathbf{s},\mathbf{a}) with \mathbf{c}=\Phi(\mathbf{s}) we have \displaystyle\|P_{i}^{\Phi}(\cdot\mid\mathbf{s},\mathbf{a})-\bar{P}_{i}^{\Phi}(\cdot\mid\mathbf{c},\mathbf{a})\|_{1}\leq\epsilon_{\mathrm{abs}}^{P},\qquad|R_{i}(\mathbf{s},\mathbf{a})-\bar{R}_{i}^{\Phi}(\mathbf{c},\mathbf{a})|\leq\epsilon_{\mathrm{abs}}^{R}. (26) Intuitively, states that map to the same \mathbf{c} should induce similar one-step latent transitions and rewards. Essentially, \epsilon_{\mathrm{abs}}^{P} represents errors in the transition dynamics due to the encoder’s \Phi latent space breaking the Markov property. Similarly, \epsilon_{\mathrm{abs}}^{R} , quantifies how much reward-relevant information is lost by \Phi . Latent policy lifting. Any latent policy \pi(\cdot\mid\mathbf{c},\cdot) induces a state-space policy \pi^{\Phi}(\mathbf{a}\mid\mathbf{s},\cdot)=\pi(\mathbf{a}\mid\Phi(\mathbf{s}),\cdot) . Lemma C.3 (Original MDP \rightarrow latent Markov MDP) . Fix task i and latent policy \pi(\mathbf{a}\mid\mathbf{c},i) . Under C.2 , the value of \pi^{\Phi} in \mathcal{M}_{i} is close to the value of \pi in the latent Markov MDP \bar{\mathcal{M}}_{i}^{\Phi}=\langle\mathcal{C},\mathcal{A},\bar{R}_{i}^{\Phi},\bar{P}_{i}^{\Phi},\gamma\rangle : \displaystyle\sup_{\mathbf{s}\in\mathcal{S}}\Big|V_{\mathcal{M}_{i}}^{\pi^{\Phi}}(\mathbf{s})-V_{\bar{\mathcal{M}}_{i}^{\Phi}}^{\pi}(\Phi(\mathbf{s}))\Big|\leq\frac{\epsilon_{\mathrm{abs}}^{R}}{1-\gamma}+\frac{\gamma R_{\max}}{(1-\gamma)^{2}}\,\epsilon_{\mathrm{abs}}^{P}. (27) Proof. Consider two (latent) MDPs under the same policy \pi : one uses the (state-dependent) pushforward kernel P_{i}^{\Phi}(\cdot\mid\mathbf{s},\mathbf{a}) and reward R_{i}(\mathbf{s},\mathbf{a}) , and the other uses \bar{P}_{i}^{\Phi}(\cdot\mid\mathbf{c},\mathbf{a}) and \bar{R}_{i}^{\Phi}(\mathbf{c},\mathbf{a}) . C.2 , bounds the one-step reward and transition discrepancies uniformly for any \mathbf{s} with code \mathbf{c} . Applying Section C.1 with \epsilon_{R}=\epsilon_{\mathrm{abs}}^{R} and \epsilon_{P}=\epsilon_{\mathrm{abs}}^{P} yields ( 27 ). ∎ C.3 Step 2: From the latent MDP to the learned world model (true task ID) Assumption C.4 . (Latent model error with true task). There exist \epsilon_{\mathrm{model}}^{P},\epsilon_{\mathrm{model}}^{R}\geq 0 such that for all (\mathbf{c},\mathbf{a}) we have \displaystyle\|\widehat{P}_{\phi}(\cdot\mid\mathbf{c},\mathbf{a},i)-\bar{P}_{i}^{\Phi}(\cdot\mid\mathbf{c},\mathbf{a})\|_{1}\leq\epsilon_{\mathrm{model}}^{P},\qquad|\widehat{R}_{\phi}(\mathbf{c},\mathbf{a},i)-\bar{R}_{i}^{\Phi}(\mathbf{c},\mathbf{a})|\leq\epsilon_{\mathrm{model}}^{R}. (28) Lemma C.5 (Latent Markov MDP \rightarrow learned world model (true task)) . Fix task i and latent policy \pi(\mathbf{a}\mid\mathbf{c},i) . Under C.4 , \displaystyle\|V_{\bar{\mathcal{M}}_{i}^{\Phi}}^{\pi}-V_{\widehat{\mathcal{M}}_{i}}^{\pi}\|_{\infty}\leq\frac{\epsilon_{\mathrm{model}}^{R}}{1-\gamma}+\frac{\gamma R_{\max}}{(1-\gamma)^{2}}\epsilon_{\mathrm{model}}^{P}, (29) where \widehat{\mathcal{M}}_{i}=\langle\mathcal{C},\mathcal{A},\widehat{R}_{\phi}(\cdot,\cdot,i),\widehat{P}_{\phi}(\cdot\mid\cdot,\cdot,i),\gamma\rangle . Proof. Apply Section C.1 on \mathcal{X}=\mathcal{C} with \epsilon_{R}=\epsilon_{\mathrm{model}}^{R} and \epsilon_{P}=\epsilon_{\mathrm{model}}^{P} . ∎ Theorem C.6 (Original task MDP vs learned latent world model (true task)) . Fix task i and latent policy \pi(\mathbf{a}\mid\mathbf{c},i) . Under C.2 and C.4 , \displaystyle\sup_{\mathbf{s}\in\mathcal{S}}\Big|V_{\mathcal{M}_{i}}^{\pi^{\Phi}}(\mathbf{s})-V_{\widehat{\mathcal{M}}_{i}}^{\pi}(\Phi(\mathbf{s}))\Big|\leq\frac{\epsilon_{\mathrm{abs}}^{R}+\epsilon_{\mathrm{model}}^{R}}{1-\gamma}+\frac{\gamma R_{\max}}{(1-\gamma)^{2}}(\epsilon_{\mathrm{abs}}^{P}+\epsilon_{\mathrm{model}}^{P}). (30) Proof. By the triangle inequality, \displaystyle\Big|V_{\mathcal{M}_{i}}^{\pi^{\Phi}}(\mathbf{s})-V_{\widehat{\mathcal{M}}_{i}}^{\pi}(\Phi(\mathbf{s}))\Big|\leq\Big|V_{\mathcal{M}_{i}}^{\pi^{\Phi}}(\mathbf{s})-V_{\bar{\mathcal{M}}_{i}^{\Phi}}^{\pi}(\Phi(\mathbf{s}))\Big|+\Big|V_{\bar{\mathcal{M}}_{i}^{\Phi}}^{\pi}(\Phi(\mathbf{s}))-V_{\widehat{\mathcal{M}}_{i}}^{\pi}(\Phi(\mathbf{s}))\Big|. (31) Bound the first term using Section C.2 and the second term using Section C.3 . ∎ C.4 Step 3: Adding the context encoder (unknown task at test time) So far, the latent model is conditioned on the true task ID i . At meta-test time i is unknown and the agent instead conditions on \mathbf{z} inferred via our context encoder. Assumption C.7 . (Task-inference sufficiency in latent space). There exist \epsilon_{\mathrm{task}}^{P},\epsilon_{\mathrm{task}}^{R}\geq 0 such that for any task i and any (\mathbf{c},\mathbf{a}) , if \mathbf{z}=g_{\theta}(\mathcal{H}) is computed from history collected in task i , then \displaystyle\|\widehat{P}_{\phi}(\cdot\mid\mathbf{c},\mathbf{a},\mathbf{z})-\widehat{P}_{\phi}(\cdot\mid\mathbf{c},\mathbf{a},i)\|_{1}\leq\epsilon_{\mathrm{task}}^{P},\qquad|\widehat{R}_{\phi}(\mathbf{c},\mathbf{a},\mathbf{z})-\widehat{R}_{\phi}(\mathbf{c},\mathbf{a},i)|\leq\epsilon_{\mathrm{task}}^{R}. (32) This is an approximate sufficiency requirement. That is, we assume that conditioning on \mathbf{z} is nearly as informative as conditioning on i for predicting latent dynamics and rewards. Lemma C.8 (Extra value error from task inference in the learned latent MDP) . Fix task i and latent policy \pi(\mathbf{a}\mid\mathbf{c},\mathbf{z}) . Under C.7 , \displaystyle\sup_{\mathbf{c}\in\mathcal{C}}\Big|V_{\widehat{\mathcal{M}}(\mathbf{z})}^{\pi}(\mathbf{c})-V_{\widehat{\mathcal{M}}_{i}}^{\pi_{i}}(\mathbf{c})\Big|\leq\frac{\epsilon_{\mathrm{task}}^{R}}{1-\gamma}+\frac{\gamma R_{\max}}{(1-\gamma)^{2}}\,\epsilon_{\mathrm{task}}^{P}, (33) where \widehat{\mathcal{M}}(\mathbf{z}) uses (\widehat{P}_{\phi}(\cdot\mid\mathbf{c},\mathbf{a},\mathbf{z}),\widehat{R}_{\phi}(\mathbf{c},\mathbf{a},\mathbf{z})) and \pi_{i}(\mathbf{a}\mid\mathbf{c})\coloneqq\pi(\mathbf{a}\mid\mathbf{c},\mathbf{z}) is the realized policy after inferring \mathbf{z} . Proof. Apply Section C.1 on \mathcal{X}=\mathcal{C} between MDP {}_{1}=\widehat{\mathcal{M}}_{i} and MDP {}_{2}=\widehat{\mathcal{M}}(\mathbf{z}) , using C.7 to bound the one-step reward and transition discrepancies. ∎ Final combined bound (unknown task, learned context). Combining Theorem C.6 and Section C.4 gives, for any latent policy \pi , \displaystyle\sup_{\mathbf{s}\in\mathcal{S}}\Big|V_{\mathcal{M}_{i}}^{\pi^{\Phi}}(\mathbf{s})-V_{\widehat{\mathcal{M}}(\mathbf{z})}^{\pi}(\Phi(\mathbf{s}))\Big|\leq\frac{\epsilon_{\mathrm{abs}}^{R}+\epsilon_{\mathrm{model}}^{R}+\epsilon_{\mathrm{task}}^{R}}{1-\gamma}+\frac{\gamma R_{\max}}{(1-\gamma)^{2}}(\epsilon_{\mathrm{abs}}^{P}+\epsilon_{\mathrm{model}}^{P}+\epsilon_{\mathrm{task}}^{P}). (34) Discussion. The bounds depend on: 1. how Markov the latent state space \mathbf{c}=\Phi(\mathbf{s}) is for predicting future latent states and rewards ( C.2 ), 2. the learned latent model error ( C.4 ), 3. the quality/sufficiency of task inference via \mathbf{z} ( C.7 ). Importantly, none of these require reconstructing \mathbf{s} from \mathbf{c} . Instead, they require that (\mathbf{c},\mathbf{z}) preserves predictive information for control. –appendices continue on next page– Appendix D Further Results Section D.1 provides additional comparisons with baselines, evaluating few-shot and zero-shot adaptation in both in-distribution and out-of-distribution task settings. Section D.2 compares our contextual latent world model with DreamerV3 ( 16 ) , a state-of-the-art model-based RL method based on recurrent state-space models (RSSMs), to assess generalization to unseen tasks from history alone. Section D.3 reports detailed disentanglement metrics across methods, highlighting the effects of temporal consistency and contrastive learning objectives on task representation learning. Ablation studies are presented in Section D.5 (contrastive learning and task representation \mathbf{z} ), Section D.6 (model size and parameter count), Section D.7 (choice of offline RL algorithm), and Section D.8 (bounding strategies for \mathbf{z} ). Finally, Section D.9 analyzes the computational cost of SPC relative to prior work, demonstrating that the proposed contextual world model is computationally efficient while achieving improved performance. D.1 Generalization to New Tasks Table 3 summarizes the results across all environments in the Meta-World ML1 benchmark. We collected datasets for these environments at varying difficulty levels using the same RL algorithm (DroQ) with identical hyperparameters. In certain environments, such as Assembly, the datasets lack successful trajectories. Consequently, various OMRL methods fail to learn the corresponding tasks. In the majority of environments, SPC outperforms the baselines, achieving higher success rates during few-shot adaptation. Table 5 summarize zero-shot adaptation performance for in-distribution tasks and Table 6 summarize zero-shot adaptation performance for out-of-distribution tasks. For environments where some variation factors in the out-of-distribution tasks interpolate those seen during training ( e.g. , [Cheetah, Finger, Walker]-speed), the performance on out-of-distribution tasks is relatively close to in-distribution performance. In contrast, when out-of-distribution generalization requires extrapolating beyond the training variation factors ( e.g. , [Cheetah, Finger, Walker]-LS, Ant-Dir), a larger performance gap between in-distribution and out-of-distribution tasks is observed. SPC performs more consistently compared to baselines when generalizing to out-of-distribution tasks, where it significantly outperforms the baselines (two-sided t-test, p<0.05 ) in 5 out of 9 and 7 out of 9 environments for in-distribution and out-of-distribution zero-shot adaptation, respectively, indicating improved generalization to OOD tasks. These results indicate that latent temporal consistency can improve performance on in-distribution tasks by converting the observation space to a latent space and can increase generalization to out-of-distribution tasks by encouraging the context encoder to capture latent dynamics. –appendices continue on next page– Table 3 : Few-shot in-distribution performance on all environments in Meta-world benchmarks. Average success rate over 6 random seeds, \pm represents 95\% confidence intervals. Since we use DroQ with the same hyperparameters to collect the datasets for all environments, there might be a lack of successful trajectories in the datasets for some environments. Bold indicates the highest value with statistical significance according to the t-test with p-value <0.05 . Environment SPC (ours) CSRO DORA FOCAL UNICORN-SS UNICORN-SUP Assembly 0.0\pm 0.0 0.0\pm 0.0 0.0\pm 0.0 0.0\pm 0.0 0.0\pm 0.0 0.0\pm 0.0 Basketball \textbf{8.3}\pm 6.0 0.0\pm 0.0 0.0\pm 0.0 0.0\pm 0.0 0.0\pm 0.0 0.0\pm 0.0 Bin-picking 1.7\pm 3.3 3.3\pm 4.1 0.0\pm 0.0 0.0\pm 0.0 0.0\pm 0.0 0.0\pm 0.0 Box-close \textbf{26.7}\pm 9.7 11.7\pm 6.0 1.7\pm 3.3 6.7\pm 6.5 3.3\pm 4.1 11.7\pm 9.4 Button-press 100.0\pm 0.0 95.0\pm 6.7 90.0\pm 5.1 98.3\pm 3.3 91.7\pm 6.0 90.0\pm 7.2 Button-press-topdown 83.3\pm 33.1 68.3\pm 9.4 75.0\pm 18.8 78.3\pm 6.0 83.3\pm 6.5 68.3\pm 7.9 Button-press-topdown-wall \textbf{90}.0\pm 5.1 55.0\pm 14.1 36.7\pm 8.3 35.0\pm 15.0 35.0\pm 11.0 31.7\pm 11.8 Button-press-wall 96.7\pm 4.1 83.3\pm 4.1 86.7\pm 4.1 91.7\pm 7.9 91.7\pm 6.0 81.7\pm 11.8 Coffee-button 100.0\pm 0.0 98.3\pm 3.3 96.7\pm 4.1 95.0\pm 6.7 100.0\pm 0.0 100.0\pm 0.0 Coffee-pull 3.3\pm 4.1 1.7\pm 3.3 0.0\pm 0.0 1.7\pm 3.3 0.0\pm 0.0 3.3\pm 4.1 Coffee-push 25.0\pm 8.4 18.3\pm 13.8 13.3\pm 8.3 13.3\pm 12.0 18.3\pm 7.9 18.3\pm 6.0 Dial-turn 88.3\pm 9.4 93.3\pm 6.5 90.0\pm 10.1 90.0\pm 8.8 86.7\pm 8.3 65.0\pm 16.6 Disassemble 25.0\pm 8.4 16.7\pm 8.3 8.3\pm 6.0 5.0\pm 6.7 18.3\pm 10.6 6.7\pm 6.5 Door-close 100.0\pm 0.0 100.0\pm 0.0 100.0\pm 0.0 100.0\pm 0.0 100.0\pm 0.0 96.7\pm 4.1 Door-lock 95.0\pm 4.4 86.7\pm 4.1 93.3\pm 6.5 81.7\pm 9.4 90.0\pm 5.1 85.0\pm 6.7 Door-open 98.3\pm 3.3 93.3\pm 9.7 93.3\pm 6.5 90.0\pm 7.2 93.3\pm 4.1 61.7\pm 24.5 Door-unlock 96.7\pm 4.1 90.0\pm 7.2 91.7\pm 6.0 90.0\pm 5.1 96.7\pm 4.1 86.7\pm 8.3 Drawer-close 100.0\pm 0.0 100.0\pm 0.0 98.3\pm 3.3 98.3\pm 3.3 98.3\pm 3.3 98.3\pm 3.3 Drawer-open 50.0\pm 12.4 33.3\pm 10.9 31.7\pm 11.8 30.0\pm 14.3 21.7\pm 14.7 45.0\pm 13.1 Faucet-close 96.7\pm 4.1 95.0\pm 6.7 83.3\pm 10.9 83.3\pm 18.0 90.0\pm 7.2 78.3\pm 11.8 Faucet-open 90.0\pm 5.1 90.0\pm 10.1 68.3\pm 24.0 88.3\pm 3.3 80.0\pm 7.2 65.0\pm 20.7 Hammer 33.3\pm 14.0 20.0\pm 11.3 33.3\pm 10.9 36.7\pm 15.7 40.0\pm 16.0 40.0\pm 13.4 Hand-insert 30.0\pm 11.3 16.7\pm 10.9 26.7\pm 8.3 25.0\pm 8.4 15.0\pm 4.4 25.0\pm 8.4 Handle-press 98.3\pm 3.3 91.7\pm 6.0 93.3\pm 4.1 93.3\pm 6.5 91.7\pm 7.9 93.3\pm 4.1 Handle-press-side 95.0\pm 4.4 88.3\pm 7.9 85.0\pm 8.4 93.3\pm 9.7 93.3\pm 9.7 88.3\pm 7.9 Handle-pull \textbf{68.3}\pm 12.8 45.0\pm 11.0 26.7\pm 9.7 40.0\pm 10.1 36.7\pm 4.1 50.0\pm 14.3 Handle-pull-side 75.0\pm 15.8 78.3\pm 13.8 36.7\pm 6.5 68.3\pm 16.3 66.7\pm 21.3 58.3\pm 9.4 Lever-pull 25.0\pm 11.0 21.7\pm 6.0 23.3\pm 9.7 25.0\pm 4.4 25.0\pm 8.4 28.3\pm 6.0 Peg-insert-side \textbf{41.7}\pm 10.6 23.3\pm 10.9 25.0\pm 13.1 18.3\pm 6.0 26.7\pm 9.7 23.3\pm 8.3 Peg-unplug-side 71.7\pm 12.8 68.3\pm 10.6 51.7\pm 22.3 60.0\pm 11.3 58.3\pm 15.5 50.0\pm 13.4 Pick-out-of-hole 25.0\pm 6.7 26.7\pm 14.0 18.3\pm 9.4 30.0\pm 7.2 28.3\pm 17.1 0.0\pm 0.0 Pick-place 1.7\pm 2.2 3.3\pm 2.8 0.0\pm 0.0 0.0\pm 0.0 1.7\pm 2.2 0.0\pm 0.0 Pick-place-wall 0.0\pm 0.0 5.0\pm 6.7 1.7\pm 3.3 1.7\pm 3.3 3.3\pm 4.1 0.0\pm 0.0 Plate-slide 56.7\pm 12.0 56.7\pm 12.0 48.3\pm 11.8 60.0\pm 8.8 53.3\pm 14.9 63.3\pm 9.7 Plate-slide-back \textbf{38.3}\pm 9.4 23.3\pm 18.0 20.0\pm 13.4 15.0\pm 15.0 15.0\pm 9.8 8.3\pm 10.6 Plate-slide-back-side 74.0\pm 17.5 66.7\pm 6.5 74.0\pm 13.3 68.0\pm 25.1 63.3\pm 13.1 76.7\pm 8.3 Plate-slide-side 75.0\pm 23.3 61.7\pm 24.0 61.7\pm 13.8 53.3\pm 23.6 66.7\pm 12.0 63.3\pm 14.0 Push \textbf{38.3}\pm 11.8 13.3\pm 4.1 20.0\pm 7.2 16.7\pm 6.5 8.3\pm 12.8 13.3\pm 9.7 Push-back 25.0\pm 5.7 20.0\pm 4.8 25.0\pm 5.7 16.7\pm 4.4 21.7\pm 5.3 16.7\pm 5.6 Push-wall \textbf{60.0}\pm 11.3 21.7\pm 7.9 41.7\pm 10.6 21.7\pm 10.6 21.7\pm 7.9 36.7\pm 15.7 Reach 6.7\pm 3.3 5.0\pm 4.4 3.3\pm 4.1 1.7\pm 3.3 6.7\pm 9.7 1.7\pm 3.3 Reach-wall 6.7\pm 4.1 3.3\pm 4.1 5.0\pm 4.4 0.0\pm 0.0 4.0\pm 4.8 5.0\pm 6.7 Shelf-place 15.0\pm 11.0 5.0\pm 6.7 6.7\pm 6.5 5.0\pm 6.7 5.0\pm 6.7 1.7\pm 3.3 Soccer 38.3\pm 6.0 18.3\pm 7.9 21.7\pm 9.4 20.0\pm 10.1 10.0\pm 0.0 30.0\pm 13.4 Stick-pull 6.7\pm 8.3 0.0\pm 0.0 0.0\pm 0.0 0.0\pm 0.0 0.0\pm 0.0 1.7\pm 3.3 Stick-push 0.0\pm 0.0 5.0\pm 4.4 0.0\pm 0.0 1.7\pm 3.3 1.7\pm 3.3 5.0\pm 6.7 Sweep 88.3\pm 8.3 75.0\pm 8.4 91.7\pm 6.0 76.7\pm 9.7 56.7\pm 8.3 88.3\pm 9.4 Sweep-into 88.3\pm 9.4 68.3\pm 14.7 65.0\pm 15.0 58.3\pm 15.5 60.0\pm 18.2 85.0\pm 13.1 Window-close 100.0\pm 0.0 95.0\pm 4.4 95.0\pm 6.7 98.3\pm 3.3 100.0\pm 0.0 98.3\pm 3.3 Window-open \textbf{100.0}\pm 0.0 88.3\pm 6.0 78.3\pm 11.8 81.7\pm 6.0 73.3\pm 9.7 81.7\pm 10.6 Table 4 : Few-shot in-distribution performance on MuJoCo and Contextual-DMC benchmarks. SPC with the latent world model outperforms baselines . Average returns over 6 random seeds, \pm represents 95\% confidence intervals. Bold indicates the highest value with statistical significance according to the t-test with p-value <0.05 . Environment SPC (ours) CSRO DORA FOCAL UNICORN-SS UNICORN-SUP Ant-dir \textbf{863.1}\pm 36.2 798.0\pm 39.3 596.5\pm 54.6 804.0\pm 35.0 812.9\pm 24.5 429.0\pm 30.6 Cheetah-LS \textbf{944.8}\pm 4.9 831.2\pm 60.0 895.3\pm 24.1 852.2\pm 26.4 795.9\pm 39.4 832.0\pm 52.9 Cheetah-speed \textbf{751.2}\pm 27.9 576.3\pm 78.2 547.0\pm 45.6 515.7\pm 62.7 554.3\pm 71.6 586.4\pm 40.7 Finger-LS \textbf{968.0}\pm 5.5 869.2\pm 46.6 822.3\pm 47.5 880.8\pm 39.2 885.6\pm 14.1 753.2\pm 56.3 Finger-speed \textbf{967.4}\pm 2.0 631.6\pm 56.1 441.2\pm 33.6 609.5\pm 25.0 515.9\pm 25.5 526.9\pm 36.8 Hopper-mass 590.6\pm 3.5 476.4\pm 68.7 563.3\pm 26.8 572.7\pm 13.0 540.9\pm 36.5 442.5\pm 119.2 Walker-friction 563.6\pm 33.5 521.8\pm 34.4 487.7\pm 27.8 532.3\pm 46.7 485.5\pm 57.8 539.1\pm 13.7 Walker-LS 934.6\pm 20.1 899.2\pm 41.8 862.5\pm 55.5 875.0\pm 51.5 880.7\pm 54.7 914.2\pm 23.0 walker-speed \textbf{835.6}\pm 37.3 771.2\pm 20.0 390.9\pm 84.0 768.9\pm 30.1 730.7\pm 48.6 518.6\pm 55.3 Table 5 : Zero-shot in-distribution performance on MuJoCo and Contextual-DMC benchmarks. SPC with the latent world model outperforms baselines . Average returns over 6 random seeds, \pm represents 95\% confidence intervals. Bold indicates the highest value with statistical significance according to the t-test with p-value <0.05 . Environment SPC (ours) CSRO DORA FOCAL UNICORN-SS UNICORN-SUP Ant-dir 726.7\pm 38.1 699.3\pm 26.6 526.9\pm 28.0 678.4\pm 40.3 668.1\pm 46.4 366.6\pm 41.6 Cheetah-LS \textbf{935.0}\pm 11.9 828.3\pm 34.9 901.7\pm 25.3 825.3\pm 36.6 794.5\pm 44.5 841.9\pm 50.9 Cheetah-speed \textbf{706.4}\pm 33.1 556.5\pm 31.3 497.1\pm 44.0 447.3\pm 73.0 490.2\pm 82.1 447.9\pm 49.1 Finger-LS \textbf{972.0}\pm 5.0 897.8\pm 41.8 869.2\pm 46.4 863.0\pm 58.3 885.3\pm 49.6 824.0\pm 30.6 Finger-speed \textbf{943.3}\pm 8.4 773.7\pm 48.5 492.4\pm 51.7 746.7\pm 49.3 671.9\pm 54.6 614.4\pm 45.0 Hopper-mass 566.0\pm 13.5 450.8\pm 79.2 555.0\pm 20.6 535.0\pm 33.3 533.7\pm 38.3 491.6\pm 145.6 Walker-friction \textbf{578.2}\pm 13.6 503.7\pm 39.5 513.9\pm 29.1 522.0\pm 32.8 522.1\pm 21.6 476.7\pm 32.4 Walker-LS 937.2\pm 9.9 882.5\pm 100.3 885.5\pm 40.8 898.9\pm 30.3 900.0\pm 49.2 889.9\pm 36.9 Walker-speed 829.7\pm 53.5 767.1\pm 31.7 446.3\pm 43.8 653.1\pm 99.4 598.6\pm 54.4 513.1\pm 48.7 Table 6 : Zero-shot out-of-distribution performance on MuJoCo and Contextual-DMC benchmarks. SPC with the latent world model outperforms baselines . Average returns over 6 random seeds, \pm represents 95\% confidence intervals. Bold indicates the highest value with statistical significance according to the t-test with p-value <0.05 . Environment SPC (ours) CSRO DORA FOCAL UNICORN-SS UNICORN-SUP Ant-dir 410.7\pm 36.9 399.2\pm 63.9 156.8\pm 44.7 368.8\pm 64.2 405.9\pm 38.5 -211.4\pm 195.5 Cheetah-LS \textbf{865.5}\pm 20.4 813.9\pm 28.1 785.8\pm 39.6 826.6\pm 20.6 806.1\pm 28.6 795.8\pm 44.3 Cheetah-speed \textbf{756.0}\pm 92.5 603.5\pm 96.5 573.0\pm 59.0 607.8\pm 160.4 598.8\pm 103.4 554.8\pm 75.9 Finger-LS \textbf{886.7}\pm 11.8 762.7\pm 57.9 717.8\pm 53.3 786.8\pm 32.6 816.1\pm 47.7 691.5\pm 54.8 Finger-speed \textbf{948.1}\pm 9.3 822.8\pm 32.3 532.9\pm 85.5 771.3\pm 39.0 709.6\pm 43.6 675.3\pm 66.0 Hopper-mass \textbf{583.4}\pm 4.1 543.6\pm 32.1 534.2\pm 23.3 547.3\pm 13.3 550.7\pm 11.3 463.9\pm 133.6 Walker-friction 474.1\pm 30.3 475.1\pm 22.5 462.4\pm 26.5 473.7\pm 36.5 484.6\pm 25.0 435.2\pm 53.8 Walker-LS \textbf{788.0}\pm 27.8 611.5\pm 31.6 650.9\pm 42.5 658.0\pm 41.0 657.6\pm 41.7 649.9\pm 50.7 Walker-speed \textbf{831.5}\pm 44.5 767.2\pm 24.0 425.3\pm 56.1 659.6\pm 120.1 623.7\pm 98.0 535.5\pm 44.2 D.2 Comparison to DreamerV3 DreamerV3 ( 16 ) is a model-based RL method that employs a Recurrent State-Space Model (RSSM; 6 ) for latent dynamics while jointly predicting rewards, observations, and terminations. Its latent space includes both continuous and discrete variables, resembling the discrete latent space used in SPC . The policy and value function are optimized within the world model. However, DreamerV3 uses one-hot encoding for its discrete latent variables, whereas SPC employs a codebook-based representation. To compare DreamerV3 with SPC in the OMRL setting, we use a public PyTorch implementation 1 1 1 https://github.com/NM512/dreamerv3-torch with default hyperparameters. Table 7 reports the zero-shot performance on in-distribution tasks; for DreamerV3, we reset the RSSM hidden state at the initial timestep during meta-testing. DreamerV3 struggles to generalize to new tasks in OMRL settings, particularly in environments where optimal policies differ significantly across tasks. For example, in [Cheetah, Finger, Walker]-speed environments, the agent must move both forward and backward at different speeds, and in Ant-dir environments, the agent must move in different directions. On the other hand, DreamerV3 shows better generalization in environments where optimal task-specific policies are more similar, such as Hopper-mass and Walker-friction. We also hypothesize that the policy may exploit inaccuracies in the world model, since the world model is trained solely on static datasets. The policy is optimized to maximize expected return under the world model’s predictions, without any penalty for acting in uncertain or poorly modeled regions. In online RL settings, the policy’s actions would be executed in the real environment, and the world model would be updated accordingly; however, this corrective mechanism is absent in the offline OMRL scenario. Table 7 : DreamerV3 fails to generalize in OMRL settings . Zero-shot generalization to in-distribution tasks. Average returns/success rates over 3 random seeds, \pm represents 95\% confidence intervals. Environment SPC (ours) DreamerV3 Ant-dir 649.9\pm 50.7 -3.6\pm 3.0 Cheetah-LS 936.5\pm 10.8 584.8\pm 53.2 Cheetah-speed 664.4\pm 51.7 178.7\pm 20.8 Finger-LS 966.4\pm 5.5 438.6\pm 85.3 Finger-speed 946.9\pm 9.6 187.0\pm 72.7 Hopper-mass 579.9\pm 9.5 555.1\pm 3.6 Walker-friction 580.6\pm 4.7 523.6\pm 41.1 Walker-LS 939.7\pm 8.3 643.6\pm 46.0 Walker-speed 705.8\pm 70.9 149.9\pm 10.3 Button-press 96.7\pm 4.1 2.2\pm 3.2 Coffee-button 100.0\pm 0.0 72.8\pm 21.1 Dial-turn 91.7\pm 6.0 0.6\pm 1.1 Door-open 100.0\pm 0.0 0.0\pm 0.0 Door-unlock 100.0\pm 0.0 0.0\pm 0.0 Handle-press 95.0\pm 4.4 4.4\pm 4.0 –appendices continue on next page– D.3 Disentanglement Metrics Table 9 and Table 10 illustrate disentanglement metrics for Finger-LS and Walker-LS environments respectively. We observe similar trends as Table 8 , where training the context encoder based on world modeling (WM) results in higher disentanglement than training the context encoder with reconstruction (UNICORN-SUP), and including contrastive learning can improve task distinguishability (informativeness, explicitness). However, while the reconstruction objective (UNICORN-SUP) results in higher disentanglement than contrastive objectives (FOCAL, InfoNCE) in Cheetah-LS ( Table 8 ), in [Finger, Walker]-LS reconstruction objective results in a lower disentanglement score. Table 8 : Disentanglement metrics (DCI, InfoMEC) for the Cheetah-length-speed (Ls) environment. Latent world models disentangle the variation factors more effectively , while contrastive learning enhances task distinguishability , reflected in informativeness and explicitness. WM denotes training the context encoder solely with the world model objective ( Equation 8 ); FOCAL and InfoNCE represent two contrastive objectives, and UNICORN-SUP indicates training with reconstruction (decoder). Average metrics over 6 random seeds, \pm represents 95\% confidence intervals. Disentanglement Completeness Informativeness Modularity Explicitness Compactness FOCAL 0.33\pm 0.11 0.25\pm 0.08 0.82\pm 0.04 0.77\pm 0.04 0.75\pm 0.05 0.20\pm 0.05 CSRO 0.31\pm 0.05 0.32\pm 0.05 0.83\pm 0.04 0.75\pm 0.03 0.78\pm 0.03 0.21\pm 0.04 DORA (InfoNCE) 0.24\pm 0.04 0.29\pm 0.05 0.75\pm 0.04 0.70\pm 0.04 0.78\pm 0.03 0.13\pm 0.08 UNICORN-SS 0.32\pm 0.06 0.29\pm 0.09 0.83\pm 0.04 0.76\pm 0.01 0.76\pm 0.04 0.23\pm 0.05 UNICORN-SUP 0.36\pm 0.09 0.23\pm 0.06 0.54\pm 0.06 0.76\pm 0.05 0.71\pm 0.02 0.24\pm 0.07 WM 0.45\pm 0.06 0.49\pm 0.07 0.81\pm 0.03 0.70\pm 0.02 0.82\pm 0.02 0.23\pm 0.07 WM+FOCAL 0.42\pm 0.05 0.47\pm 0.06 0.87\pm 0.02 0.71\pm 0.03 0.86\pm 0.02 0.24\pm 0.06 WM+InfoNCE (ours) 0.50\pm 0.05 0.49\pm 0.05 0.89\pm 0.02 0.74\pm 0.03 0.87\pm 0.01 0.26\pm 0.05 Table 9 : Disentanglement metrics (DCI, InfoMEC) for the Finger Length/Speed environment. Average metrics over 6 random seeds, \pm represents 95\% confidence intervals. Disentanglement Completeness Informativeness Modularity Explicitness Compactness FOCAL 0.36\pm 0.06 0.30\pm 0.07 0.70\pm 0.01 0.83\pm 0.02 0.80\pm 0.01 0.24\pm 0.01 CSRO 0.36\pm 0.03 0.38\pm 0.08 0.71\pm 0.02 0.82\pm 0.02 0.79\pm 0.01 0.24\pm 0.01 DORA (InfoNCE) 0.41\pm 0.09 0.43\pm 0.09 0.64\pm 0.03 0.77\pm 0.03 0.79\pm 0.02 0.23\pm 0.01 UNICORN-SS 0.33\pm 0.05 0.31\pm 0.08 0.69\pm 0.02 0.79\pm 0.02 0.78\pm 0.01 0.15\pm 0.01 UNICORN-SUP 0.25\pm 0.05 0.34\pm 0.06 0.57\pm 0.01 0.85\pm 0.02 0.73\pm 0.01 0.30\pm 0.04 WM 0.41\pm 0.03 0.43\pm 0.02 0.70\pm 0.01 0.82\pm 0.02 0.85\pm 0.01 0.23\pm 0.01 WM+FOCAL 0.41\pm 0.06 0.43\pm 0.04 0.80\pm 0.04 0.83\pm 0.02 0.86\pm 0.02 0.23\pm 0.01 WM+InfoNCE (ours) 0.46\pm 0.06 0.43\pm 0.04 0.82\pm 0.03 0.82\pm 0.02 0.87\pm 0.02 0.23\pm 0.01 Table 10 : Disentanglement metrics (DCI, InfoMEC) for the Walker Length/Speed environment. Average metrics over 6 random seeds, \pm represents 95\% confidence intervals. Disentanglement Completeness Informativeness Modularity Explicitness Compactness FOCAL 0.33\pm 0.07 0.31\pm 0.06 0.83\pm 0.04 0.69\pm 0.03 0.82\pm 0.02 0.23\pm 0.01 CSRO 0.42\pm 0.08 0.41\pm 0.09 0.79\pm 0.03 0.72\pm 0.03 0.82\pm 0.02 0.23\pm 0.01 DORA (InfoNCE) 0.22\pm 0.07 0.17\pm 0.14 0.67\pm 0.05 0.68\pm 0.05 0.76\pm 0.03 0.23\pm 0.01 UNICORN-SS 0.38\pm 0.07 0.31\pm 0.05 0.84\pm 0.03 0.73\pm 0.03 0.84\pm 0.03 0.24\pm 0.01 UNICORN-SUP 0.20\pm 0.03 0.18\pm 0.07 0.38\pm 0.04 0.80\pm 0.04 0.64\pm 0.02 0.26\pm 0.02 WM 0.39\pm 0.05 0.27\pm 0.06 0.75\pm 0.04 0.79\pm 0.05 0.78\pm 0.03 0.24\pm 0.02 WM+FOCAL 0.43\pm 0.04 0.29\pm 0.02 0.88\pm 0.05 0.77\pm 0.05 0.86\pm 0.03 0.23\pm 0.01 WM+InfoNCE (ours) 0.44\pm 0.09 0.30\pm 0.09 0.84\pm 0.04 0.79\pm 0.04 0.88\pm 0.03 0.24\pm 0.01 –appendices continue on next page– D.4 Ablation: Latent Space Formulation Table 11 provides a detailed comparison of how different latent space formulation ( e.g. , (i) continuous vs discrete, (ii) regression vs classification, (iii) bounded vs unbounded) impact the performance on different benchmarks. Aggregate statistics were presented in Figure 4 . Table 11 : Ablation on latent space, different choice on latent space, and temporal consistency loss function. Average returns over 6 random seeds, \pm represents 95\% confidence intervals. Environment Cont MSE Cont Cosine SimNorm MSE SimNorm Cosine Dis MSE Dis Cosine Dis CE (ours) Ant-dir 841.6\pm 25.9 865.6\pm 41.4 851.5\pm 30.9 854.5\pm 34.0 784.0\pm 54.0 800.7\pm 65.9 863.1\pm 36.2 Cheetah-LS 938.0\pm 9.2 895.8\pm 49.8 898.3\pm 25.1 905.6\pm 25.0 920.0\pm 15.2 897.0\pm 33.2 944.8\pm 4.9 Cheetah-speed 645.3\pm 127.1 735.6\pm 27.6 572.2\pm 49.6 453.9\pm 96.0 647.2\pm 63.3 648.5\pm 49.7 764.1\pm 39.2 Finger-LS 972.2\pm 6.1 971.6\pm 3.6 971.6\pm 4.8 973.7\pm 2.3 974.7\pm 2.2 976.8\pm 2.5 968.0\pm 5.5 Finger-speed 949.4\pm 2.9 932.3\pm 18.9 915.0\pm 63.1 947.3\pm 25.3 929.8\pm 39.7 968.0\pm 1.9 967.4\pm 2.0 Hopper-mass 578.8\pm 17.8 557.0\pm 31.8 563.6\pm 11.7 513.2\pm 71.7 406.0\pm 116.0 429.5\pm 61.0 590.6\pm 3.5 Walker-friction 559.9\pm 20.6 573.7\pm 13.4 502.1\pm 35.5 521.8\pm 49.4 549.4\pm 28.0 451.5\pm 40.0 563.6\pm 33.5 Walker-LS 927.8\pm 23.2 930.1\pm 56.5 942.0\pm 9.1 931.9\pm 31.3 944.9\pm 4.6 938.6\pm 11.6 934.6\pm 20.1 Walker-speed 813.0\pm 30.9 847.6\pm 23.4 843.1\pm 20.6 824.1\pm 31.9 814.6\pm 28.7 831.3\pm 22.2 835.7\pm 37.3 Button-press 100.0\pm 0.0 98.3\pm 3.3 100.0\pm 0.0 96.7\pm 6.5 98.3\pm 3.3 98.3\pm 3.3 100.0\pm 0.0 Coffee-button 100.0\pm 0.0 100.0\pm 0.0 100.0\pm 0.0 100.0\pm 0.0 100.0\pm 0.0 100.0\pm 0.0 100.0\pm 0.0 Door-close 100.0\pm 0.0 100.0\pm 0.0 100.0\pm 0.0 100.0\pm 0.0 100.0\pm 0.0 98.3\pm 3.3 100.0\pm 0.0 Door-open 100.0\pm 0.0 98.3\pm 3.3 100.0\pm 0.0 96.7\pm 4.1 98.3\pm 3.3 98.3\pm 3.3 100.0\pm 0.0 Door-unlock 95.0\pm 6.7 96.7\pm 4.1 98.3\pm 3.3 96.7\pm 4.1 98.3\pm 3.3 100.0\pm 0.0 100.0\pm 0.0 Drawer-close 98.3\pm 3.3 100.0\pm 0.0 100.0\pm 0.0 98.3\pm 3.3 100.0\pm 0.0 100.0\pm 0.0 100.0\pm 0.0 Drawer-open 45.0\pm 8.4 40.0\pm 8.8 63.3\pm 10.9 56.7\pm 9.7 48.3\pm 9.4 48.3\pm 12.8 51.7\pm 6.0 Faucet-close 91.7\pm 6.0 90.0\pm 5.1 98.3\pm 3.3 100.0\pm 0.0 100.0\pm 0.0 93.3\pm 4.1 100.0\pm 0.0 Faucet-open 71.7\pm 17.8 95.0\pm 4.4 86.7\pm 12.0 95.0\pm 4.4 90.0\pm 5.1 90.0\pm 10.1 91.7\pm 6.0 Handle-press 91.7\pm 6.0 93.3\pm 4.1 95.0\pm 4.4 96.7\pm 6.5 96.7\pm 4.1 98.3\pm 3.3 98.3\pm 3.3 Handle-pull 61.7\pm 15.5 70.0\pm 14.3 75.0\pm 12.1 73.3\pm 17.3 70.0\pm 20.9 86.7\pm 8.3 70.0\pm 17.5 Push-wall 53.3\pm 17.3 48.3\pm 12.8 46.7\pm 18.0 45.0\pm 12.1 48.3\pm 19.2 56.7\pm 14.0 63.3\pm 6.5 Sweep 68.3\pm 11.8 78.3\pm 9.4 73.3\pm 9.7 71.7\pm 13.8 81.7\pm 9.4 93.3\pm 4.1 85.0\pm 8.4 Sweep-into 73.3\pm 12.0 73.3\pm 12.0 71.7\pm 15.5 73.3\pm 19.4 71.7\pm 19.9 76.7\pm 9.7 80.0\pm 11.3 Window-close 100.0\pm 0.0 100.0\pm 0.0 100.0\pm 0.0 100.0\pm 0.0 98.3\pm 3.3 100.0\pm 0.0 100.0\pm 0.0 Window-open 100.0\pm 0.0 88.3\pm 9.4 96.7\pm 6.5 96.7\pm 4.1 98.3\pm 3.3 98.3\pm 3.3 98.3\pm 3.3 –appendices continue on next page– D.5 Ablation: Contrastive Learning In this section, we perform an ablation study on the contrastive and world modeling objectives used to train the context encoder. We compare two contrastive learning objectives commonly employed in OMRL: InfoNCE and FOCAL ( 26 ) (also referred to as distance metric learning). The FOCAL objective is defined as: \mathrm{L}_{\text{FOCAL}}(\phi)=\mathbb{1}\{i=j\}\|z^{i}-z^{j}\|^{2}_{2}+\mathbb{1}\{i\neq j\}\frac{\beta}{\|z^{i}-z^{j}\|^{2}_{2}+\epsilon_{0}}. (35) For a fair comparison, we convert the observation space to a discrete latent space for all the methods with the same world model (DCWM). Table 12 reports few-shot in-distribution testing and Table 13 reports few-shot out-of-distribution testing. Here, WM refers to training the context encoder with the world modeling objective, and + indicates the combination of objectives. For environments in Meta-World benchmarks (last 6 rows in Table 12 ), we observe no significant difference in the performance of different objectives. Training the context encoder solely with the world modeling objective (WM) is insufficient, as it fails to distinguish between different tasks. This limitation is particularly pronounced in environments where variation factors affect only the reward function, rather than the transition dynamics ( e.g. , Ant-dir, where the desired forward direction varies, or environments requiring the inference of desired speed). Only using contrastive learning results in good performance across most tasks, while InfoNCE outperforms FOCAL significantly in certain environments, especially on out-of-distribution testing. Adding the world modeling objective to the contrastive objective has an insignificant impact on in-distribution performance; however, it can improve generalization to out-of-distribution tasks for certain environments. We employ the same relative weighting of the contrastive objective with respect to the world modeling objective across all environments. Overall, combining InfoNCE with the world modeling objective produces more robust results across environments compared to combining FOCAL with the world modeling objective. Table 12 : Ablation on contrastive learning and world modeling, few-shot in-distribution performance. Average returns/success rates over 6 random seeds, \pm represents 95\% confidence intervals. Bold indicates the highest value with statistical significance according to the t-test with p-value <0.05 . Environment FOCAL InfoNCE WM WM+FOCAL WM+InfoNCE (ours) Ant-dir 841.6\pm 31.1 857.7\pm 42.3 487.5\pm 91.8 859.0\pm 20.5 863.1\pm 36.2 Cheetah-LS 940.0\pm 16.2 937.7\pm 17.7 941.5\pm 16.6 933.2\pm 15.0 944.8\pm 4.9 Cheetah-speed 721.1\pm 54.2 727.3\pm 23.8 395.0\pm 36.2 711.2\pm 94.0 764.1\pm 39.2 Finger-LS 971.0\pm 10.5 968.4\pm 10.8 974.6\pm 5.5 973.7\pm 2.8 968.0\pm 5.5 Finger-speed 789.7\pm 189.6 958.1\pm 5.7 706.1\pm 150.3 770.0\pm 185.2 \textbf{967}.4\pm 2.0 Walker-LS 929.8\pm 24.4 947.7\pm 15.1 904.5\pm 24.9 928.9\pm 21.6 934.6\pm 20.1 Walker-speed 622.6\pm 65.9 842.2\pm 35.6 522.1\pm 95.4 552.0\pm 82.4 835.7\pm 37.3 Button-press 100.0\pm 0.0 100.0\pm 0.0 98.3\pm 3.3 100.0\pm 0.0 100.0\pm 0.0 Coffee-button 100.0\pm 0.0 100.0\pm 0.0 100.0\pm 0.0 100.0\pm 0.0 100.0\pm 0.0 Dial-turn 93.3\pm 4.1 93.3\pm 6.5 98.3\pm 3.3 96.7\pm 4.1 98.3\pm 3.3 Door-open 100.0\pm 0.0 100.0\pm 0.0 100.0\pm 0.0 96.7\pm 6.5 100.0\pm 0.0 Door-unlock 98.3\pm 3.3 95.0\pm 6.7 98.3\pm 3.3 98.3\pm 3.3 100.0\pm 0.0 Handle-press 96.7\pm 4.1 96.7\pm 4.1 95.0\pm 6.7 95.0\pm 6.7 98.3\pm 3.3 Table 13 : Ablation on contrastive learning and world modeling, few-shot out-of-distribution performance. Average returns over 6 random seeds, \pm represents 95\% confidence intervals. Bold indicates the highest value with statistical significance according to the t-test with p-value <0.05 . Combining contrastive learning with latent temporal consistency enhances generalization to out-of-distribution tasks . Environment FOCAL InfoNCE WM WM+FOCAL WM+InfoNCE (ours) Ant-dir 529.0\pm 41.0 363.2\pm 173.9 203.6\pm 102.9 540.8\pm 111.2 401.8\pm 92.4 Cheetah-LS 864.4\pm 21.7 864.7\pm 29.9 867.2\pm 7.5 876.1\pm 24.7 860.6\pm 10.2 Cheetah-speed 729.0\pm 42.4 754.9\pm 70.6 486.8\pm 111.1 908.7\pm 74.2 \textbf{967.7}\pm 10.5 Finger-LS 836.3\pm 45.5 809.6\pm 53.1 838.3\pm 64.8 860.0\pm 42.7 850.9\pm 41.5 Finger-speed 793.8\pm 189.0 868.5\pm 10.3 766.6\pm 176.2 755.7\pm 185.0 \textbf{978.5}\pm 6.0 Walker-LS 740.2\pm 52.9 793.2\pm 62.2 757.0\pm 65.2 738.9\pm 51.8 792.3\pm 41.3 Walker-speed 619.4\pm 111.4 782.7\pm 32.2 507.7\pm 103.4 568.7\pm 128.5 \textbf{833.2}\pm 64.5 –appendices continue on next page– D.6 Ablation: Number of Parameters In this section, we investigate how increasing the number of trainable parameters affects performance across different methods. SPC maps the observation space to a latent space using a latent world model, which increases the total number of parameters. Figure 9 illustrates the few-shot in-distribution performance of each method for different model sizes. For the baselines, we vary the number of hidden layers in \{2,3\} and the number of hidden units in \{256,512,1024\} , resulting in six model sizes (default is two hidden layers with 256 neurons). To ensure architectural consistency with SPC , we apply Layer Normalization and the Mish activation function across all baseline networks. For SPC , we set the latent dimension and the number of neurons in the dynamic head of the latent world model to \{64,128,256,512\} while setting the number of neurons in the offline RL (IQL) networks to \{64,128,256,512\} (6 combinations, default is 512 latent dimensions and neurons in the dynamic head, 256 neurons for the offline RL networks). SPC exhibits better scaling with model size: performance generally improves as the number of parameters increases. However, for smaller model sizes, SPC underperforms the baselines in some environments. Figure 9 : SPC scales more effectively. Few-shot generalization to in-distribution tasks for different model sizes. The shaded area represents the standard deviation over 6 random seeds. –appendices continue on next page– D.7 Ablation: Offline RL As described in Section 3.2 , policy optimization with offline data requires regularization to avoid OOD action selection when computing the target for the value function. Offline RL methods address this issue in different ways, and in principle, any offline RL method can be used for policy optimization in SPC . By default, we use Implicit Q-Learning (IQL) for all methods, which predicts an upper expectile of the TD targets in SARSA style without querying OOD actions. We also evaluate SPC with Conservative Q-Learning (CQL, 25 ) and TD3+BC ( 13 ) for policy optimization, summarized in Table 14 . CQL regularizes the value function by reducing the q-value for OOD actions, resulting in a pessimistic value function. TD3+BC, on the other hand, regularizes the policy to stay close to the behavior policy by adding a behavior cloning objective to the policy optimization. We used one set of hyperparameters (default values) for all methods without further fine-tuning. We find that IQL in general is more robust, performing well in diverse environments. CQL generally performs on par with IQL, even outperforming significantly in two environments. However, the computation cost of CQL is generally higher than IQL. We sometimes observe a performance drop when training for a larger number of steps. TD3+BC generally has a lower performance than CQL and IQL in our settings. We hypothesize that fine-tuning the regularization weight for each environment can increase the performance. Table 14 : Ablation: different offline RL methods for policy optimizations. Few-shot generalization to in-distribution tasks. Average returns/success rates over 6 random seeds, \pm represents 95\% confidence intervals. Bold indicates the highest value with statistical significance according to the t-test with p-value <0.05 . Environment CQL IQL (Def) TD3+BC Ant-dir 780.7\pm 54.4 \textbf{863.1}\pm 36.2 731.8\pm 29.8 Cheetah-LS 949.8\pm 12.1 943.0\pm 11.5 846.2\pm 60.6 Cheetah-speed \textbf{813.3}\pm 3.5 751.2\pm 27.9 620.8\pm 133.0 Finger-LS \textbf{987.5}\pm 3.8 957.1\pm 23.9 330.7\pm 26.7 Finger-speed 970.2\pm 4.3 962.0\pm 9.2 208.7\pm 31.8 Hopper-mass 584.2\pm 1.1 587.5\pm 4.9 159.4\pm 26.1 Walker-friction 585.6\pm 32.6 563.6\pm 33.5 535.6\pm 26.6 Walker-LS 948.0\pm 7.2 937.1\pm 16.6 710.2\pm 84.9 Walker-speed 814.0\pm 42.4 827.6\pm 34.6 655.7\pm 108.2 Button-press 95.0\pm 9.8 100.0\pm 0.0 80.0\pm 5.1 Coffee-button 100.0\pm 0.0 100.0\pm 0.0 91.7\pm 7.9 Dial-turn 90.0\pm 19.6 91.7\pm 10.6 95.0\pm 4.4 Door-open 100.0\pm 0.0 100.0\pm 0.0 23.3\pm 4.1 Door-unlock 100.0\pm 0.0 100.0\pm 0.0 51.7\pm 10.6 Handle-press 60.0\pm 19.6 \textbf{93.3}\pm 4.1 76.7\pm 6.5 –appendices continue on next page– D.8 Ablation: Bounding the Task Representation In this section, we evaluate how bounding the task representation z affects the generalization. By default, SPC utilize \mathrm{Tanh} as the activation function for the context encoder, consistent with prior OMRL methods. Table 15 summarize the results. We compare unbounded representations (Identity), \ell_{2} -normalization, FSQ, and \mathrm{Tanh} for bounding the latent space. Since we utilize FSQ in our latent world model, we investigate whether discretizing the task representation with FSQ influences generalization. The results indicate that bounding the task representation significantly enhances generalization in certain environments ( e.g. , Ant-dir, Cheetah-speed). Discretizing the task representation with FSQ yields no advantages over \mathrm{Tanh} , and \ell_{2} -normalization is less robust across different environments. Table 15 : Bounding the task representation enables better generalization in certain environments , though discretizing with FSQ yields no advantages. Average returns and success rates across 6 random seeds; \pm represents 95% confidence intervals. Environment Identity \ell_{2} -Norm FSQ Tanh Ant-dir 452.7\pm 121.3 838.0\pm 72.9 866.0\pm 46.2 863.1\pm 36.2 Cheetah-LS 933.8\pm 10.5 932.3\pm 16.7 938.3\pm 5.8 944.8\pm 4.9 Cheetah-speed 670.3\pm 75.1 778.0\pm 27.8 730.7\pm 30.2 764.1\pm 39.2 Finger-LS 962.2\pm 3.8 956.6\pm 11.4 969.6\pm 5.4 968.0\pm 5.5 Finger-speed 715.4\pm 133.6 855.1\pm 83.8 950.4\pm 30.4 967.4\pm 2.0 Walker-LS 922.7\pm 20.0 924.7\pm 15.7 932.4\pm 42.9 934.6\pm 20.1 Button-press 98.3\pm 3.3 100.0\pm 0.0 98.3\pm 3.3 100.0\pm 0.0 Coffee-button 100.0\pm 0.0 100.0\pm 0.0 100.0\pm 0.0 100.0\pm 0.0 Dial-turn 96.7\pm 4.1 95.0\pm 4.4 78.0\pm 33.6 96.7\pm 4.1 Door-open 95.0\pm 6.7 98.3\pm 3.3 100.0\pm 0.0 100.0\pm 0.0 Door-unlock 98.3\pm 3.3 96.7\pm 4.1 98.3\pm 3.3 100.0\pm 0.0 –appendices continue on next page– D.9 Computation Cost Figure 10 compares the computation cost for different methods. All experiments are conducted using the same hardware, as described in Appendix B , to ensure a fair comparison. Although SPC has a longer training time per step, it generally converges faster than the baselines, compensating for the higher per-step computational cost. During testing, SPC is slightly slower because it first maps the observation to the latent space using the observation encoder, after which the policy produces actions. UNICORN-SUP trains the context encoder solely using the prediction loss and has the lowest computational cost per training step. However, incorporating contrastive learning can improve task representation learning and, consequently, generalization to new tasks. DORA, which uses the InfoNCE loss for contrastive learning, has a lower per-step computational cost than FOCAL, which uses distance metric learning, suggesting that InfoNCE is more computationally efficient than the FOCAL objectives in Equation 35 . CSRO and UNICORN-SS aim to reduce context distribution shift by minimizing a CLUB upper bound of mutual information and by adding a prediction loss, respectively. These approaches require additional networks, increasing their computational cost per training step. During test time, all baselines have the same computational cost since the policy and context encoder architectures are identical across the baseline methods. Figure 10 : Comparing the computation cost. Left: Few-shot performance on in-distribution tasks with training time. Middle: Number of training steps (full backpropagation and updating the networks) per second. Right: Number of time steps per second during testing. SPC is computationally more expensive during training, yet it converges faster. Mapping the observation space to the latent space adds insignificant computation overhead during testing. Results are averaged over 3 random seeds while considering the standard deviation.