Horizon Imagination: Efficient On-Policy Rollout in Diffusion World Models Lior Cohen, Ofir Nabati, Kaixin Wang, Navdeep Kumar, Shie Mannor Abstract We study diffusion-based world models for reinforcement learning, which offer high generative fidelity but face critical efficiency challenges in control. Current methods either require heavyweight models at inference or rely on highly sequential imagination, both of which impose prohibitive computational costs. We propose Horizon Imagination (HI), an on-policy imagination process for discrete stochastic policies that denoises multiple future observations in parallel. HI incorporates a stabilization mechanism and a novel sampling schedule that decouples the denoising budget from the effective horizon over which denoising is applied while also supporting sub-frame budgets. Experiments on Atari 100K and Craftium show that our approach maintains control performance with a sub-frame budget of half the denoising steps and achieves superior generation quality under varied schedules. Code is available at https://github.com/leor-c/horizon-imagination . 1 Introduction World models ( Ha & Schmidhuber 2018 ) have emerged as a powerful paradigm for achieving sample-efficient reinforcement learning (RL). By learning a generative model of environment dynamics, world models enable agents to produce large quantities of simulated experience that can be used to train controllers, thereby reducing reliance on costly real environment interactions. Recently, diffusion world models have become particularly attractive, owing to their unmatched generative fidelity across modalities such as images and video ( Esser et al. 2024 ; Blattmann et al. 2023 ) . Despite these advantages, current approaches face key limitations when applied to control. Practical deployment often requires controllers that are lightweight and highly computationally efficient, enabling real-time inference with low power consumption. In contrast, the prevailing trend in world models, particularly diffusion-based models, is toward ever-larger architectures ( Agarwal et al. 2025 ; Parker-Holder et al. 2024 ; Ball et al. 2025 ; Assran et al. 2025 ) . This poses a significant barrier to their direct use at test time, as generation becomes increasingly time- and compute-intensive. Furthermore, methods that rely on repeated interaction with the world model during controller inference are therefore impractical in many real-world settings. Alternatively, approaches that train an independent controller via world model imagination ( Alonso et al. 2024 ) encounter prohibitive computational overhead with diffusion-based models. Since each observation must be generated through a costly multi-step denoising process, and imagination requires sequentially interleaving policy decisions with world model predictions, the resulting imagination process is inherently sequential and computationally intensive. In this work, we propose a more efficient approach for training discrete stochastic policies with diffusion world models. We introduce Horizon Imagination, an on-policy imagination process that denoises multiple future observations simultaneously, reducing the sequential burden of diffusion generation. We devise (i) a mechanism to mitigate policy-induced instability during horizon imagination, and (ii) a novel sampling schedule that disentangles the denoising budget from the schedule’s decay horizon, enabling independent control over both parameters, supporting sub-frame budgets, and yielding superior generation quality at higher budgets. Our approach is training-agnostic and applies to any pre-trained world model with observation-level time conditioning. We validate our approach on a subset of Atari 100K and Craftium environments, showing that the agent maintains control performance with only half the denoising budget. We further analyze world model generation quality under different configurations of the proposed Horizon schedule, spanning fully autoregressive to highly parallel regimes and a range of denoising budgets. Parallel generation consistently proves advantageous, achieving strong performance even under sub-frame budgets. 2 Related Work Large Diffusion World Models Diffusion-based methods have emerged as the state of the art in generative modeling, delivering unmatched fidelity across modalities such as images and video ( Ho et al. 2020 ; Rombach et al. 2022 ; Ho et al. 2022 ) . Building on this success, recent large-scale world models adopt diffusion backbones for their superior generative capabilities ( Ball et al. 2025 ; Agarwal et al. 2025 ; Decart et al. 2024 ; Assran et al. 2025 ) . While motivated by applications such as agent training, these works are limited to conditional video generation and do not address control. Concurrent Multi-step Generation The idea of generating multiple steps in parallel with diffusion world models has been explored in several recent works ( Chen et al. 2024 ; Rigter et al. 2024 ; Ding et al. 2024 ; Jackson et al. 2024 ) . In particular, Diffusion Forcing ( Chen et al. 2024 ) introduces a multi-step simultaneous generation scheme together with a control framework in which the world model serves as both policy and planner, akin to model predictive control ( García et al. 1989 ) . Although effective for learning control, this approach is computationally intensive and often impractical for deployment. In contrast, our work addresses this limitation by enabling efficient training in imagination, allowing a lightweight policy to be extracted from the world model for deployment. Rigter et al. 2024 and Jackson et al. 2024 proposed policy-guided imagination methods where during the denoising process the actions are updated in the direction of the score of the policy action distribution \nabla_{a}\pi(a|s) , as in Langevin dynamics ( Song & Ermon 2019 ) and classifier guidance ( Dhariwal & Nichol 2021 ) . However, these approaches are limited to continuous action spaces. Ding et al. 2024 proposed an off-policy multi-step generation approach for the offline RL setting. Importantly, none of the above approaches examined how world model generation quality is affected by sampling configurations, including sequential versus parallel generation and the influence of the generation budget. Diffusion World Model Agents for Control Alonso et al. 2024 and Yang et al. 2024 proposed diffusion-based world model methods that follow a traditional step-by-step sequential imagination. Both reported significant computational overhead. In particular, Alonso et al. 2024 provided a detailed runtime analysis, showing that sequential imagination dominates the overall cost. 3 Preliminaries Reinforcement Learning Setup Consider a Partially Observable Markov Decision Process (POMDP) environment. Here, we consider a practical state-agnostic formulation, as the agent has no knowledge about the POMDP hidden state space. At each step t=1,2,\ldots , starting from some initial observation {\mathbf{o}}_{1} , the agent picks an action {\textnormal{a}}_{t} and the environment evolves according to {\mathbf{o}}_{t+1},r_{t},d_{t}\sim p({\mathbf{o}}_{t+1},r_{t},d_{t}|{\mathbf{o}}_{\leq t},{\mathbf{a}}_{\leq t}) , where r_{t}\in\mathbb{R},d_{t}\in\{0,1\} are the reward and termination signals, respectively. This process repeats until a non-zero termination signal is observed. The agent’s goal is to maximize its expected return \mathbb{E}[\sum_{t=0}^{\infty}\gamma^{t}r_{t+1}] . World Model Agents In reinforcement learning (RL), “world models” ( Ha & Schmidhuber 2018 ) denote a class of model-based methods where the controller is trained entirely on simulated rollouts generated by a learned dynamics model, whereas real environment interactions are used for training the model. World-model agents typically consist of four components: a representation model for encoding and decoding observations, a dynamics model, a controller, and a replay buffer. World model agents are trained either in an online or an offline setting. Online training learns the components jointly from scratch, fitting the representation and dynamics to real interactions while optimizing the controller via model-generated rollouts, whereas offline training uses pre-collected data to train the components sequentially: representation, dynamics, and controller. Our method is applicable in both settings. Diffusion Framework In this work, we employ the 1-Rectified Flow (RF) framework ( Liu et al. 2023 ) for its simplicity and robustness, though our method remains compatible with other diffusion variants. Formally, given samples from an unknown target distribution p_{\text{data}} , RF learns a time-dependent vector field v_{\theta} parameterized by \theta that transports samples x^{0}\sim p_{\text{prior}} from a chosen prior distribution p_{\text{prior}} toward data samples x^{1}\sim p^{1}\approx p_{\text{data}} . The evolution is governed by \mathop{}\!\mathrm{d}x^{\tau}=v_{\theta}(x^{\tau},\tau)\mathop{}\!\mathrm{d}\tau,\quad\tau\in[0,1], where superscripts denote denoising time. The vector field v_{\theta} is trained to estimate \mathbb{E}[x^{1}-x^{0}\mid x^{\tau}] , with x^{\tau}=\tau x^{1}+(1-\tau)x^{0} , by minimizing the regression objective L(\theta)=\mathbb{E}_{x^{0},x^{1},\tau}\left\|v_{\theta}(x^{\tau},\tau)-(x^{1}-x^{0})\right\|^{2}. 4 Method Multi-step trajectory generation naturally alternates between producing an observation {\mathbf{o}}_{t} and sampling an action {\textnormal{a}}_{t}\sim\pi(\cdot\mid{\mathbf{o}}_{\leq t},{\mathbf{a}}_{0\\ \pi_{\rho(i)}&S_{i}(\pi)=0\end{cases},\qquad S_{i}(\pi)=\sum_{j=i}^{N}\pi_{\rho(j)}. Intuitively, \alpha_{i}(\pi) is the conditional probability of choosing the i -th action in the order \rho , given that all earlier actions were skipped. Proposition 1 . The sampling scheme {\textnormal{a}}(\cdot,\cdot) satisfies the following properties: 1. For any distribution {\bm{p}}\in\Delta^{N-1} , the random variable {\textnormal{a}}({\bm{p}},\omega) with \omega\sim\mathcal{U}([0,1))^{N-1} is distributed according to {\bm{p}} , i.e., {\textnormal{a}}({\bm{p}},\cdot)\sim{\bm{p}} . 2. For any {\bm{p}},{\bm{q}}\in\Delta^{N-1} (e.g., {\bm{p}}=\pi^{\tau_{1}} and {\bm{q}}=\pi^{\tau_{2}} with \tau_{1}<\tau_{2} ), consider the event A\;=\;\{\omega\ |\ {\textnormal{a}}({\bm{p}},\omega)\neq{\textnormal{a}}({\bm{q}},\omega)\}. Let \delta({\bm{p}},{\bm{q}})=\tfrac{1}{2}\,\|{\bm{p}}-{\bm{q}}\|_{1} denote the total variation distance between {\bm{p}} and {\bm{q}} . Then \delta({\bm{p}},{\bm{q}})\;\leq\;\Pr(A)\;\leq\;\|\alpha({\bm{p}})-\alpha({\bm{q}})\|_{1}. We defer the proof to Appendix E . As an immediate corollary, if p=q then \Pr(A)=0 , meaning that when the distribution does not vary between denoising steps, the actions necessarily remain unchanged. In contrast, naively sampling a fresh action at each step is likely to induce frequent action changes under sufficiently high-entropy policies. Figure 2: Comparison of the Pyramidal schedule ( Chen et al. 2024 ) and the proposed Horizon schedule (transposed). Horizon fixes the decay horizon ( \nu=3 ), yielding consistent schedules across budgets, whereas in the Pyramidal schedule the decay horizon drifts with budget, as the two are entangled, leading to degraded generation quality at higher budgets. The Horizon Schedule We propose a novel time schedule inspired by the pyramidal schedule of Chen et al. 2024 , which denoises near future observations before far future ones. Unlike prior approaches, our schedule disentangles the rate at which denoising progresses across time, i.e., the decay horizon , from the denoising budget , i.e., how many denoising steps are performed in total. This design provides precise and independent control over both parameters. Formally, let \nu\in[1,h] specify the decay horizon, i.e., the number of steps over which the denoising schedule decays from 1 to 0 , and let B\in\mathbb{N} denote the total budget of denoising steps. We define the sampling schedule as a matrix {\bm{K}}\in[0,1]^{(B+1)\times h} , where each of the first B rows specifies the denoising times \tau assigned to all h observations at a given step. The last row ensures that \tau_{t}=1 for all 1\leq t\leq h . To determine the values of the entries of {\bm{K}} , consider the lines \kappa(t,b)=-\frac{1}{\nu}t+\frac{b}{B}(1+\frac{h-1}{\nu}), with slope -\frac{1}{\nu} , sequence time index 0\leq t0 then 6: Sample first action {\textnormal{a}}_{{\textnormal{k}}}\sim\pi(\cdot\,|\,{\mathbf{z}}_{\leq{\textnormal{k}}},{\mathbf{a}}_{<{\textnormal{k}}}) 7: end if 8: for each denoising iteration b from 1 to B do 9: Set \tau\leftarrow{\bm{K}}_{b,:} and \mathrm{d}\tau\leftarrow{\bm{K}}_{b+1,:}-{\bm{K}}_{b,:} 10: Denote {\mathbf{z}}_{\leq{\textnormal{k}}+t}^{\tau}=({\mathbf{z}}^{1}_{\leq{\textnormal{k}}},{\mathbf{z}}_{{\textnormal{k}}+1}^{\tau_{1}},\ldots,{\mathbf{z}}_{{\textnormal{k}}+t}^{\tau_{t}}) 11: for 1\leq t\leq h-1 do 12: Compute action distributions \pi_{{\textnormal{k}}+t}^{\tau}\leftarrow\pi(\cdot\,|\,{\mathbf{z}}_{\leq{\textnormal{k}}+t}^{\tau},{\mathbf{a}}_{<{\textnormal{k}}+t}^{\tau},\tau_{\leq t}) 13: Generate actions {\textnormal{a}}_{{\textnormal{k}}+t}^{\tau_{t}}={\textnormal{a}}\left(\pi_{{\textnormal{k}}+t}^{\tau},\omega_{t}\right) \triangleright (Eq. 2 ) 14: end for 15: Perform a denoising step: {\mathbf{z}}^{\tau_{t}+\mathrm{d}\tau_{t}}_{{\textnormal{k}}+t}\leftarrow{\mathbf{z}}^{\tau_{t}}_{{\textnormal{k}}+t}+v_{\theta}({\mathbf{z}}_{\leq{\textnormal{k}}+t}^{\tau},{\mathbf{a}}_{<{\textnormal{k}}+t}^{\tau},\tau_{\leq t})\,\mathrm{d}\tau_{t} 16: for all 1\leq t\leq h with \mathrm{d}\tau_{t}>0 \triangleright (single forward pass) 17: end for 18: Compute rewards and terminations \bar{r}_{\leq{\textnormal{k}}+h},d_{\leq{\textnormal{k}}+h} with r_{\phi},d_{\phi} 19: return imagined trajectory {\mathbf{z}}_{\leq{\textnormal{k}}+h}^{1},{\mathbf{a}}_{<{\textnormal{k}}+h},\bar{r}_{\leq{\textnormal{k}}+h},d_{\leq{\textnormal{k}}+h} 5 Experiments 5.1 Empirical Analysis of Action Consistency We conduct two controlled experiments to evaluate the proposed stable action sampling mechanism. In the first experiment (Figure 3(a) ), we examine the empirical distribution of average action changes between random source and target distributions {\bm{p}},{\bm{q}} using our method. We sample 10^{4} pairs of categorical distributions from a uniform Dirichlet prior with N actions. For each pair, we draw 10^{6} (\omega,\rho) samples, compute the induced actions, and record the average number of changes. This is repeated for various N values. Empirically, our method performs very close to the tight theoretical lower bound, the total variation \delta(p,q) , across all N values, demonstrating near-optimal performance. In the second experiment (Figure 3(b) ), we compare naive sampling with the proposed stable mechanism in a simulated denoising process. We interpolate between a source and target categorical distribution with N=10 actions across 8 steps, then fix the target distribution for 8 additional steps. Both source and target are sampled from a Dirichlet distribution under three entropy settings: low (0.2,\ldots,0.2) , uniform, and high (5,\ldots,5) . For each entropy setting, we simulate the process for 1{,}000 source-target pairs and estimate the average number of action changes across all 16 steps over 10^{6} simulations per pair. Our results show that while the proposed method yields at most one action change on average across all 16 steps, naive sampling incurs changes in most steps, altering more than half overall. For high-entropy distributions, the number of action changes decreases under our method but increases with naive sampling, consistent with our theoretical results. (a) Distributions of the average number of action changes, in \delta({\bm{p}},{\bm{q}}) units, for various N values. Minimum, mean, and maximum are indicated. (b) Comparing the number of action changes between naive sampling and the proposed method (mean \pm std). Figure 3: Empirical study of the average number of action changes under various settings. 5.2 Control Performance To study whether Horizon Imagination enables a more efficient training, we benchmark our world model agent in an online RL setting. The agent’s training process involves a repeated cycle of four stages: data collection through environment interactions, representation learning, world model learning, and actor-critic learning in imagination. For comparison, we consider three baselines that differ only in their imagination configuration: a standard autoregressive baseline with 32 denoising steps in total (one step per observation), denoted (\nu=1,B=32) , and two variants with decay horizon \nu=4 and budgets of B=16 and B=32 . 5.2.1 Setup Agent Implementation For latent representations, we employ the continuous autoencoder image tokenizer of Agarwal et al. 2025 , and apply a final \tanh activation to the encoder output to constrain values to [-1,1] . The denoiser network v_{\theta} is implemented as an action-conditioned causal Diffusion Transformer (DiT) ( Peebles & Xie 2023 ) architecture. This model operates on sequences of latent observations, where action and denoising time conditioning are carried via the adaptive layer normalization (AdaLN) layers of the DiT architecture. Rewards and termination signals are modeled using a lightweight recurrent neural network (RNN) with a compact convolutional neural network (CNN) feature extractor and two output heads, operating on encoded trajectories. The same RNN architecture is used for both the actor and critic networks. For simplicity, we omit action conditioning in the actor, critic, and reward-termination models, as in all benchmarks considered the observations alone provide sufficient information. The agent uses fixed hyperparameters across all experiments and comprises a tokenizer ( 22.5 M), world model ( 67 M), and actor-critic ( 7.5 M), totaling 97 M parameters. We defer the full implementation details to Appendix A . Benchmarks We evaluate on four Atari100K ( Kaiser et al. 2020 ) games and four Craftium ( Malagón et al. 2025 ) games, both benchmarks involving visual inputs and discrete actions. All environments were run for 100K interaction steps, except Craftium/SmallRoom-v0 , which was capped at 30K due to its lower difficulty. The well-established Atari benchmark serves as a solid control evaluation suite and is visually simpler. In practice, we found that even a single denoising step can yield satisfactory generation quality in Atari. By contrast, Craftium presents richer and more complex observations, providing a more challenging test for the world model. Training Time and Hardware All experiments were conducted on NVIDIA A100 GPUs, except for Craftium/SmallRoom-v0 , which was run on RTX 5090, RTX 4090, and A40 GPUs. On Atari, training with a denoising budget of B=32 required approximately 27 hours per run, while reducing the budget to B=16 shortened training to about 19 hours. Because the tokenizer and world model training stages are unaffected by this parameter, the overall speed-up is smaller than a full 2\times . On Craftium, runs take longer due to slower environment interactions. 5.2.2 Results As shown in Figure 4 , both baselines with decay horizon \nu=4 maintain the performance of the autoregressive baseline across all environments, achieving comparable results at reduced cost. Notably, our approach sustains full performance with a sub-frame budget of B=16 , requiring only half the denoising steps of the autoregressive baseline. Moreover, our results suggest that a single denoising step per observation generally suffices, with baselines at B=32 performing comparably across environments. In contrast, in Craftium/ChopTree-v0 , the most visually complex environment, the (\nu=4,B=32) baseline achieves superior performance, suggesting improved effectiveness under the B=32 budget. Figure 4: Actor-Critic Performance. Average episodic return curves of key baselines during training. Each baseline is evaluated over 5 seeds. Curves show the mean and standard deviation, smoothed by a moving average (window size 15 ). A dashed horizontal line denotes Atari human-level performance. 5.2.3 Ablation Study: Stable Action Sampling To assess the influence of the proposed stable action sampling mechanism on control performance, we perform an ablation study by reverting to naive action sampling. In this baseline, a fresh action is independently drawn from the policy distribution before each denoising step. The results in Figure 5 show that replacing the stable action sampling mechanism with naive sampling leads to a substantial drop in control performance (returns). Notably, while the naive baseline is consistently weaker, we observe a particularly prominent collapse on Atari Boxing and Gopher. These findings further highlight the importance of preventing unnecessary action fluctuations during the denoising process. Figure 5: Actor–critic performance comparison between the proposed stable action sampling method and the naive baseline. Each baseline is evaluated over 5 seeds. Curves show the mean and standard deviation, smoothed by a moving average (window size 15 ). A dashed horizontal line denotes Atari human-level performance. 5.3 Parallel vs. Sequential Generation Quality To study the impact of parallel multi-step denoising on generation quality, we evaluate the trained world models from Section 5.2 under varying decay horizon ( \nu ) and denoising budget ( B ) values. Our central question is how generation quality differs between parallel and sequential configurations, and which settings prove most effective. In our evaluation, 512 episode segments of 33 frames are sampled from the training data. For each combination of \nu and B , 512 corresponding 32-frame continuations are generated using the first frame as context. To isolate the impact of \nu and B from other factors such as the action sampling scheme, the original recorded actions are provided to the model at every denoising step. Generation quality is assessed using Fréchet Video Distance (FVD) ( Unterthiner et al. 2019 ) and mean squared error (MSE) against the ground-truth segments. FVD better reflects perceptual quality, while MSE is more indicative of deviations from the ground truth sequence. Our results (Figure 6 ) show a consistent FVD trend across environments, indicating a clear advantage for parallel configurations ( 4\leq\nu\leq 16 ) under low to medium budgets. Remarkably, in several cases, parallel variants with sub-frame budgets achieve quality comparable to baselines that rely on the maximal budget, 16 times larger. At extreme budgets ( B\geq 128 ), performance tends to degrade slightly as \nu increases, with the autoregressive baseline consistently ranking at the top. Interestingly, the MSE results (bottom row of Figure 6 ) suggest that increasing the denoising budget may cause generated sequences to drift further from the ground truth, even as perceptual quality improves. Finally, in Appendix D , we present the corresponding Atari results and further repeat this experiment under the Pyramidal schedule of Chen et al. 2024 , which suffers a pronounced collapse in performance as budgets grow. Figure 6: World model generation quality versus denoising steps budget. Each point shows the average FVD/MSE over 512 sampled 33-frame segments, where the first frame was given as context and the last 32 were generated conditioned on the recorded actions. A dashed vertical line indicates the transition out of sub-frame budgets. 6 Limitations While our control results consistently highlight the advantage of \nu=4 , we did not extend our study to additional configurations. Evaluating each baseline requires running full experiments across all environments with five seeds per environment, which is computationally prohibitive. Consequently, we had to restrict our study to eight environments to keep the evaluations tractable. Future work could explore additional configurations. 7 Conclusions In this work, we addressed the computational challenge of on-policy imagination in diffusion world models with discrete actions, motivated by the need to train lightweight policies for deployment. We introduced Horizon Imagination (HI), a parallel multi-step imagination procedure that incorporates two novel mechanisms: stable discrete action sampling and the Horizon schedule. Our analysis shows that stable sampling ensures consistent action selection, thereby stabilizing imagination, while the widely applicable Horizon schedule delivers sub-frame performance and preserves robust decay horizons across budgets. Together, these advances allow HI to sustain strong performance even under sub-frame budgets and at half the computational cost, making diffusion world models markedly more practical for training deployable policies. Appendix A Experimental Setup A.1 Model Architectures A.1.1 Representation Model (Tokenizer) We use the continuous image autoencoder of Agarwal et al. 2025 as the representation model of our agent. We modify the output of the encoder network by adding a final \tanh activation to enforce values in [-1,1] . Empirically, we observed no loss of performance due to this modification. In addition, we adjust the configuration to fit our settings and computational budget. Further details on the architecture and hyperparameters can be found in Tables 1 and 3 , and in our open-source code. We use a standard mean squared error (MSE) regression objective for the reconstruction loss instead of the original L_{1} objective, as we empirically found it to perform better in our settings. The full optimization objective is the sum of the reconstruction loss and a perceptual loss \mathcal{L}_{\text{perceptual}} ( Agarwal et al. 2025 ) (Eq. 3, p. 15). Formally, the loss is given by \mathcal{L}_{\text{tokenizer}}=\|\hat{o}-o\|^{2}+\mathcal{L}_{\text{perceptual}}, (4) where \mathcal{L}_{\text{perceptual}}=\frac{1}{L}\sum_{l=1}^{L}\alpha_{l}\|\texttt{VGG}_{l}(\hat{o})-\texttt{VGG}_{l}(o)\|_{1}, is the perceptual loss, \hat{o}=\varphi_{\text{dec}}(\varphi_{\text{enc}}(o)) is the reconstructed observation produced by the autoencoder, \texttt{VGG}_{l} are the features from the l -th layer of a pretrained VGG-19 network with L layers, and \alpha_{l} is a weight associated with the l -th loss term. We use the same hyperparameter values as Agarwal et al. 2025 . Table 1: The encoder and decoder architectures of the representation model. “Conv2d(a,b,c)” represents a 2d-convolutional layer with kernel size a\times a , stride of b and padding c . “GN” represents a GroupNorm operator with 32 groups, \epsilon=1e-6 and learnable per-channel affine parameters. “Upsample” represents a repeat-interleave operation that doubles the spatial dimensions, followed by a Conv2d(3,1,1) layer. Module Output Shape Encoder Input 3\times 64\times 64 Conv2d(3, 1, 1) 64\times 64\times 64 ResnetBlock 128\times 64\times 64 ResnetBlock 128\times 64\times 64 Conv2d(3, 2, 0) 128\times 32\times 32 ResnetBlock 256\times 32\times 32 ResnetBlock 256\times 32\times 32 Conv2d(3, 2, 0) 256\times 16\times 16 ResnetBlock 256\times 16\times 16 AttnBlock 256\times 16\times 16 ResnetBlock 256\times 16\times 16 AttnBlock 256\times 16\times 16 Conv2d(3, 2, 0) 256\times 8\times 8 ResnetBlock 256\times 8\times 8 AttnBlock 256\times 8\times 8 ResnetBlock 256\times 8\times 8 GN 256\times 8\times 8 Conv2d(3, 1, 1) 16\times 8\times 8 Conv2d(1, 1, 0) 16\times 8\times 8 \tanh 16\times 8\times 8 Module Output Shape Decoder Input 16\times 8\times 8 Conv2d(1, 1, 0) 16\times 8\times 8 Conv2d(3, 1, 1) 256\times 8\times 8 ResnetBlock 256\times 8\times 8 AttnBlock 256\times 8\times 8 ResnetBlock 256\times 8\times 8 ResnetBlock 256\times 8\times 8 ResnetBlock 256\times 8\times 8 ResnetBlock 256\times 8\times 8 Upsample 256\times 16\times 16 ResnetBlock 256\times 16\times 16 AttnBlock 256\times 16\times 16 ResnetBlock 256\times 16\times 16 AttnBlock 256\times 16\times 16 ResnetBlock 256\times 16\times 16 AttnBlock 256\times 16\times 16 Upsample 256\times 32\times 32 ResnetBlock 128\times 32\times 32 ResnetBlock 128\times 32\times 32 ResnetBlock 128\times 32\times 32 Upsample 128\times 64\times 64 GN 128\times 64\times 64 Conv2d(3, 1, 1) 3\times 64\times 64 A.1.2 World Model Our world model consists of two components: the denoiser v_{\theta} and a separate reward–termination predictor network. We view the dynamics model as a large-scale, general module that captures the complexity of the environment, whereas reward–termination predictors should remain lightweight, task-specific, and operate on the dynamics outputs. This design disentangles their dependencies: dynamics models can be scaled and pre-trained in advance, while downstream tasks require learning only small, efficient reward–termination predictors upon receiving a new task. For the DiT-based denoiser v_{\theta} ( Peebles & Xie 2023 ) , we build on the Cosmos implementation ( Agarwal et al. 2025 ) with several adaptations to our setting. As in Agarwal et al. 2025 , inputs are patchified with a 2\times 2 spatial patch layer, and a corresponding reverse patchify layer is applied at the output. We also adopt 3D RoPE positional embeddings ( Su et al. 2024 ; Agarwal et al. 2025 ) , but in contrast to Agarwal et al. 2025 , we omit any additional positional encodings. Since no textual conditioning is required, we remove the cross-attention operator, yielding a vanilla DiT block structure. Attention is modified with a frame-level causal mask: tokens can attend to all other tokens in the same frame as well as to tokens from preceding frames. To condition on actions, we use a learned embedding table mapping discrete actions to vectors, which are summed with the denoising-time embeddings to form the conditioning input to the AdaLN layers. Note that we only used the architecture implementation of Agarwal et al. 2025 . Specifically, our work is based on the rectified flow diffusion framework ( Liu et al. 2023 ) , which is different from the EDM ( Karras et al. 2022 ) framework used by Agarwal et al. 2025 . The reward–termination network processes latent observation sequences {\mathbf{z}}_{1},\ldots,{\mathbf{z}}_{t} , analogous to v_{\theta} . Each latent is first encoded by a compact CNN consisting of a single ResNet block with 256 channels, followed by a 1{\times}1 convolution that reduces the dimensionality to 64 channels. The resulting features are flattened and passed through a linear layer with a Sigmoid Linear Unit (SiLU) activation Hendrycks & Gimpel 2017 ; Ramachandran et al. 2018 , yielding a vector representation for each 3D latent observation. Finally, the sequence of vectors is processed by an LSTM ( Hochreiter & Schmidhuber 1997 ) followed by two linear output heads that predict rewards and terminations, respectively. The reward–termination model is trained on the same clean trajectory segments used for optimizing v_{\theta} . To accommodate reward signals that vary widely in scale and sparsity, we model rewards in symlog space. Formally, the reward objective is \|\,r_{\phi}({\mathbf{z}}_{\leq t+1},{\mathbf{a}}_{\leq t})-\symlog(r_{t})\,\|^{2}, where r_{\phi}({\mathbf{z}}_{\leq t+1},{\mathbf{a}}_{\leq t}) denotes the raw prediction and \symlog(x)=\mathrm{sign}(x)\,\log(|x|+1) is the symlog transform ( Hafner et al. 2025 ) . At inference time, the reward estimate is recovered via \hat{r}_{t}=\symexp\!\left(r_{\phi}({\mathbf{z}}_{\leq t+1},{\mathbf{a}}_{\leq t})\right),\qquad\symexp(x)=\mathrm{sign}(x)\,\big(\exp(|x|)-1\big), where \symexp is the inverse of the symlog transform. The termination predictor is trained using a standard cross-entropy loss. A.1.3 Actor-Critic The critic \hat{V}^{\pi} is trained exclusively on fully denoised outputs via the following regression objective in symlog space: L_{\hat{V}^{\pi}}=\frac{1}{h}\sum_{t=1}^{h}\|\symlog(G_{t})-\hat{V}^{\pi}({\mathbf{z}}_{\leq t},{\mathbf{a}}_{