Optimistic World Models: Efficient Exploration in Model-Based Deep Reinforcement Learning Akshay Mete, Shahid Aamir Sheikh, Tzu-Hsiang Lin, Dileep Kalathil, P. R. Kumar Abstract Efficient exploration remains a central challenge in reinforcement learning (RL), particularly in sparse-reward environments. We introduce Optimistic World Models (OWMs), a principled and scalable framework for optimistic exploration that brings classical reward-biased maximum likelihood estimation (RBMLE) from adaptive control into deep RL. In contrast to upper confidence bound (UCB)–style exploration methods, OWMs incorporate optimism directly into model learning by augmentation with an optimistic dynamics loss that biases imagined transitions toward higher-reward outcomes. This fully gradient-based loss requires neither uncertainty estimates nor constrained optimization. Our approach is plug-and-play with existing world model frameworks, preserving scalability while requiring only minimal modifications to standard training procedures. We instantiate OWMs within two state-of-the-art world model architectures, leading to Optimistic DreamerV3 and Optimistic STORM, which demonstrate significant improvements in sample efficiency and cumulative return compared to their baseline counterparts. 1 Introduction World models such as DreamerV3 ( 23 ) , STORM ( 53 ) , and DIAMOND ( 2 ) have achieved state-of-the-art sample efficiency in a wide range of reinforcement learning tasks. However, they are known to underperform in sparse-reward environments as exploration strategies such as policy (actor) entropy regularization are often insufficient. We address this shortcoming by proposing an Optimistic World Model framework that introduces an optimistic dynamics loss function on top of the standard world model training objective. This loss function gently steers the model’s transition probabilities towards higher rewards, training the world model to generate optimistic imaginations . (a) Private Eye (b) Enduro (c) Montezuma’s Revenge (d) Cartpole Swing. Sparse Figure 1 : Optimistic World Models on challenging environments. Our Optimistic World Models (OWMs) incorporate optimistic exploration while retaining similar scalability and computational efficiency of standard world models, as the underlying neural network architectures are identical. The introduction of the optimistic dynamics loss, inspired by the Reward Biased Maximum Likelihood Estimation (RBMLE) principle ( 31 ) , avoids the computational pitfalls commonly experienced by UCB-based approaches to optimism ( 33 ; 5 ; 1 ) , namely non-convex constraints and uncertainty estimates. Empirically, the proposed OWM variants – Optimistic DreamerV3 (O-DreamerV3) and Optimistic STORM (O-STORM) show improvement over DreamerV3 and STORM across many popular RL benchmarks, and achieve exceptional improvement in sparse-reward environments such as Private Eye, Acrobot Swingup Sparse, Cartpole Swingup Sparse, as well as dense-reward environments such as Enduro and UpNDown, as highlighted in Figures 1 and 3 . Through this work, we also highlight that the current world model framework of separating model learning and policy learning is analogous to the certainty equivalence principle from adaptive control theory. The certainty equivalence principle is known to suffer from the closed-loop identification problem, which can lead to convergence to sub-optimal policies. This closed-loop identification problem is the underlying problem making exploration necessary in RL. RBMLE, which was notably one of the first approaches that was shown to overcome the closed-loop identification problem, forms the theoretical foundation of Optimistic World Models. Contributions Our key technical contributions are summarized as follows: • Gradient-Based RBMLE for Deep MBRL : We present a fully gradient-based computational approach to implement the RBMLE principle in MBRL. This enables the application of the RBMLE principle to large-scale DRL, which has been primarily limited to small-scale RL scenarios until now. • Optimistic World Models : We introduce Optimistic World Models, a plug-and-play framework with an additional loss function that steers the model estimate towards models with higher rewards. The OWMs incorporate principled optimistic exploration with minimal computational overload and no architectural changes. We develop two specific instances of OWMs – Optimistic DreamerV3 and Optimistic STORM. • Strong Empirical Gains : OWMs exhibit exceptional improvement on challenging sparse-reward environments in the Atari100K and DMC benchmarks, highlighting their efficient exploration. Optimistic DreamerV3 achieves a mean human-normalized score (HNS) of 152.68% – a 55% improvement compared to the 97.45% mean HNS of DreamerV3. 2 Related Work Model-Based Reinforcement Learning (MBRL): MBRL algorithms have been shown to achieve sub-linear regret guarantees in a variety of RL setups such as MDPs ( 5 ; 10 ; 41 ; 29 ) , LQG systems ( 1 ; 47 ; 28 ) , and non-linear systems ( 30 ; 50 ) . Despite strong theoretical guarantees, most of these algorithms are computationally intractable or expensive. MBPO ( 27 ) and PETS ( 15 ) demonstrated how model-based algorithms can outperform model-free algorithms in deep reinforcement learning by utilizing model-generated trajectories. World Models in Reinforcement Learning. World models ( 20 ) have gained popularity in various areas such as reinforcement learning, generative modeling, and robotics ( 23 ; 24 ; 11 ; 3 ) . In model-based reinforcement learning, world models have achieved improved sample complexity due to their ability to learn efficient representations of the environment. These world model-based RL algorithms can be broadly categorized into two categories: (a) algorithms that use policy gradient in imagination such as the Dreamer family ( 21 ; 22 ; 23 ) , STORM ( 53 ) , IRIS ( 42 ) , and DIAMOND ( 2 ) ; and (b) algorithms that use Monte-Carlo Tree Search (MCTS) methods such as MuZero ( 45 ) , EfficientZero ( 52 ) , and EfficientZero-V2 ( 51 ) . In many scenarios, the MCTS-based algorithms outperform the imagination-based algorithms at a higher computational cost, one of the underlying reasons being poor exploration by imagination-based algorithms. Exploration in Model Based RL. World models such as Dreamer, STORM, and Diamond use exploration strategies such as policy entropy regularization, which are often found insufficient in sparse-reward environments. The optimism in the face of uncertainty (OFU) principle, embodied by the UCB algorithms, is a popular exploration method in small-scale RL setups ( 33 ; 4 ; 6 ; 5 ; 1 ) . These algorithms are computationally prohibitive in large-scale deep RL due to non-convex constraints and the requirement of precise uncertainty estimates. Several algorithms, such as H-UCRL ( 16 ) and SOMBRL ( 48 ) , implement UCB-like optimistic exploration algorithms by utilizing Gaussian Processes to quantify model uncertainty. Other popular heuristic-based exploration approaches include probabilistic ensemble-based approaches ( 15 ; 32 ; 46 ) , and the intrinsic motivation approaches ( 12 ; 43 ) . 3 Preliminaries A Markov Decision Process (MDP) is defined by the tuple ({\mathcal{S}},{\mathcal{A}},p^{\star},r,\mu) , where {\mathcal{S}} is the state space, {\mathcal{A}} is the action space, p^{\star}:{\mathcal{S}}\times{\mathcal{A}}\to\Delta({\mathcal{S}}) denotes the unknown transition probability function, r:{\mathcal{S}}\times{\mathcal{A}}\to\mathbb{R} is the reward function, and \mu is the initial state distribution. Given a model p\in P and a policy \pi\in\Pi , the expected cumulative reward is defined as \displaystyle J(p,\pi)=\mathbb{E}_{p,\pi}\left[\sum_{t=0}^{T-1}r(s_{t},a_{t})\mid s_{0}\sim\mu\right]. (1) The goal of a reinforcement learning algorithm is to find the optimal policy \pi^{\star} such that \pi^{\star}\in\arg\max J(p^{\star},\pi) when the true model p^{\star} is unknown. In particular, the MBRL algorithm focuses on learning the model p^{\star} or an approximation of it using the data, and uses this learned model to learn the optimal policy. Recently, MBRL algorithms using the framework of the world model, such as Dreamer ( 23 ; 22 ; 21 ) , STORM ( 53 ) , DIAMOND ( 2 ) , and IRIS ( 42 ) , have shown remarkable successes in many challenging tasks. The model and policy learning in this class of algorithms follows the following general framework. 1. Environment Interaction: The agent deploys its current policy in the real environment, and stores observations ( o_{t} ), actions ( a_{t} ), and rewards ( r_{t} ) in a replay buffer. 2. Training the world model: The world model is trained to learn an accurate representation of the environment by fitting on data stored in the replay buffer. Typically, the world model includes an encoder that maps observations ( o_{t} ) to a latent space representation ( s_{t} ), a dynamics model that predicts the next state given the current state and action ( p(s_{t+1}|s_{t},a_{t}) ), a reward model ( r(s_{t},a_{t}) ), and a decoder to reconstruct the observation from the latent space representation. 3. Learning in imagination: The current world model and policy ( \pi(a_{t}|s_{t}) ) are used to generate imagined trajectories. These imagined trajectories are used to improve the policy using methods such as actor-critic algorithms. These steps are performed iteratively to improve both the world model of the environment and the agent’s policy. In this framework, the world model is trained on the real data, while the actor is only trained on the imaginary trajectories. This separation between world model and policy learning is analogous to the certainty equivalence principle described below. 3.1 Bias of MLE For fully observable MDPs ( o_{t}=s_{t} ), the certainty equivalence approach to MBRL/adaptive control is as follows: The learning agent computes the maximum likelihood estimate, \displaystyle p_{t}\in\arg\max_{p\in P}\sum_{\tau=0}^{t-1}\log p(s_{\tau+1}\mid s_{\tau},a_{\tau}). Then it selects a policy \pi_{t}\in\Pi that is optimal for the estimated model p_{t} : \pi_{t}\in\arg\max_{\pi\in\Pi}J(p_{t},\pi). A natural question is whether the certainty equivalence approach always converges to the true dynamics p^{\star} and the optimal policy \pi^{\star} ? This fundamental question was first studied in ( 37 ) , and it was shown that p_{t} converges to p^{\star} if the following condition holds: \text{ For any }p^{1}\neq p^{2}\in P,p^{1}(\cdot\mid s,a)\neq p^{2}(\cdot\mid s,a)\text{ for all }s,a. This requirement, known as the “identifiability condition”, is highly restrictive, and is not satisfied even in simple settings such as multi-armed bandits. 9 further showed that in the absence of this identifiability condition, the parameter identification only occurs for the closed-loop system, i.e., p_{\infty}(s^{\prime}\mid s,\pi_{\infty}(s))=p^{\star}(s^{\prime}\mid s,\pi_{\infty}(s))~\forall~s,s^{\prime} (2) where p_{\infty} and \pi_{\infty} are the limiting estimate and policy, respectively. This is known as the “closed-loop identification”. However, this does not guarantee that p_{\infty}(s^{\prime}\mid s,a)=p^{\star}(s^{\prime}\mid s,a)~\forall~s,a,s^{\prime} , nor does it guarantee that \pi_{\infty} is optimal for p^{\star} . This central issue is known as the “closed-loop identification problem” in adaptive control. As a consequence, p_{t} and \pi_{t} can converge to a model p_{\infty} and its optimal policy \pi_{\infty} that are self-consistent but suboptimal for the true model ( 9 ) : J(p_{\infty},\pi_{\infty})=J(p^{\star},\pi_{\infty})\implies J(p^{\star},\pi_{\infty})\leq J(p^{\star},\pi^{\star}). This bias of the maximum likelihood estimate toward models with lower optimal rewards underscores the need for exploration, making the exploration-exploitation trade-off a fundamental challenge in reinforcement learning. 3.2 Reward Biased Maximum Likelihood Estimation One of the first solutions to the above closed-loop identifiability issue, the reward-biased maximum likelihood estimation (RBMLE) method, was proposed by 31 . To overcome the inherent bias of MLE, RBMLE introduces an explicit, counter-acting bias towards models with higher rewards as follows: p_{t}\in\arg\max_{p\in P}~\left\{\alpha(t)\max_{\pi\in\Pi}J(p,\pi)+\frac{1}{t}\sum_{\tau=0}^{t-1}\log p(s_{\tau+1}\mid s_{\tau},a_{\tau})\right\}. (3) \pi_{t}\in\arg\max_{\pi\in\Pi}J(p_{t},\pi), The term \alpha(t) has to be carefully designed. It should be small enough to preserve the good properties of the maximum likelihood estimate, i.e., closed-loop identification. At the same time, it has to be large enough to overcome the bias of MLE towards models with lower average rewards. Both of these properties are achieved by choosing \alpha(t) such that \lim_{t\to\infty}t\alpha(t)\to\infty and \lim_{t\to\infty}\alpha(t)\to 0 . 31 showed that the model estimate p_{t} converges to p^{\star} and the policy \pi_{t} to an optimal policy, in a Cesaro sense. The RBMLE principle has been adapted and studied in a wide range of adaptive control/MBRL scenarios, such as stochastic bandits, LQG control, and RLHF ( 8 ; 31 ; 13 ; 17 ; 36 ; 25 ; 41 ; 38 ; 26 ; 14 ) . A detailed account of various works based on the RBMLE principle can be found in ( 39 ; 40 ) . In the next section, we provide a computationally efficient method to implement the RBMLE principle, which has been a challenge so far. We then propose the Optimistic World Models based on it. Figure 2 : Optimistic World Model framework: In standard world models, the dynamics model is trained only to fit the real replay buffer. In OWMs, the optimistic dynamics loss uses imagined trajectories to push model transitions toward high-reward outcomes (highlighted by the red arrow), leading to optimistic imaginations . 4 Optimistic World Models In this section, we introduce Optimistic world models , a principled optimistic exploration framework for world models. We begin by introducing a gradient based computational approach to adapt the RBMLE approach in deep reinforcement learning. 4.1 RBMLE in Deep Reinforcement Learning Consider a model-based reinforcement learning agent with two neural networks: A model network, parameterized by \phi , that learns p_{\phi}(s^{\prime}\mid s,a) , the transition probabilities of the next state s^{\prime} given current state s and action a , and a policy network parameterized by \theta that defines the policy \pi_{\theta}(a|s) . Let D_{t}=\{(s_{u},a_{u},s_{u+1})\}_{u=1}^{t-1} be the replay buffer, which stores tuples collected from real environment interaction. The RBMLE objective is given by: M_{\phi,\theta}=\alpha(t)J(p_{\phi},\pi_{\theta})+\frac{1}{t}\sum_{(s,a,s^{\prime})\in D_{t}}{\log p_{\phi}(s^{\prime}\mid s,a)}. (4) Let \tau_{\phi,\theta}=\{s_{0},a_{0},\cdots,s_{H+1}\} denote an imagined trajectory generated by simulating the policy \pi_{\theta} on the model p_{\phi} , then the gradient of J(p_{\phi},\pi_{\theta}) can be computed using the log-derivative trick similar to the vanilla policy gradient ( 49 ) , and is given by: \nabla_{\phi}J(p_{\phi},\pi_{\theta})=\mathbb{E}_{\tau_{\phi,\theta}}\left[\sum_{h=0}^{H}R(\tau_{\phi,\theta})\nabla_{\phi}\log p_{\phi}(s_{h+1}\mid s_{h},a_{h})\right]. (5) In practice, this can be empirically computed by generating N trajectories \{\tau^{i}_{\phi,\theta}\}_{i=1}^{N} , and then computing the sample mean. \displaystyle\nabla_{\phi}J(p_{\phi},\pi_{\theta})=\frac{1}{N}\sum_{i=1}^{N}\left[\sum_{h=0}^{H}R(\tau^{i}_{\phi,\theta})\nabla_{\phi}\log p_{\phi}(s_{h+1}\mid s_{h},a_{h})\right]. (6) This yields us the gradient estimate of M_{\phi,\theta} , \displaystyle\nabla_{\phi}M_{\phi,\theta}= \displaystyle\frac{\alpha(t)}{N}\left[\sum_{i=1}^{N}\sum_{h=0}^{H}R(\tau^{i}_{\phi,\theta})\nabla_{\phi}\log p_{\phi}(s_{h+1}\mid s_{h},a_{h})\right]+\frac{1}{t}\sum_{(s,a,s^{\prime})\in D_{t}}{\nabla_{\phi}\log p_{\phi}(s^{\prime}\mid s,a)}. (7) Similarly, an estimate of \nabla_{\theta}J(p_{\phi},\pi_{\theta}) can be computed using the imagined trajectories. \nabla_{\theta}J(p_{\phi},\pi_{\theta})=\frac{1}{N}\sum_{i=1}^{N}\left[\sum_{h=0}^{H}R(\tau^{i}_{\phi,\theta})\nabla_{\theta}\log\pi_{\theta}(a_{h}\mid s_{h})\right]. (8) Therefore, ( 7 ) and ( 8 ) can be used to update the model and policy parameters as follows: \displaystyle\phi_{t+1}=\phi_{t}+\beta\nabla_{\phi}M_{\phi,\theta}|_{\phi_{t},\theta_{t}},~\theta_{t+1}=\theta_{t}+\beta\nabla_{\theta}J(p_{\phi},\pi_{\theta})|_{\phi_{t},\theta_{t}}. The key insights from the RBMLE approach are: • Optimistic Dynamics Loss: RBMLE can be implemented by augmenting the certainty equivalence framework with an additional loss term. This loss function steers the log-probabilities of the model transitions, \log p_{\phi}(s^{\prime}|s,a) towards higher reward outcomes, analogous to how policy gradient methods push \log\pi_{\theta}(a\mid s) towards actions with higher rewards. • The optimism function \alpha(t) : The parameter \alpha(t) controls the degree of optimism by balancing the optimistic dynamics loss and the log-likelihood loss. Based on these insights, we develop Optimistic World Models, generalizing the RBMLE principle to deep reinforcement learning for partially observable MDPs. 4.2 Incorporating Optimism in World Models We introduce the optimistic dynamics loss, {\mathcal{L}}_{t}^{opt} in the imagination-based world model framework as follows: Let \tau_{t}=\{(s_{\ell},a_{\ell},r_{\ell})\}_{\ell=1}^{L} denote the imagined trajectory generated by executing the current policy \pi_{\theta_{t}} on the learned world model dynamics p_{\phi_{t}} . \displaystyle{\mathcal{L}}_{t}^{opt}=- \displaystyle\alpha(t)\left[\sum_{\ell=1}^{L-1}A_{\ell}\log p_{\phi_{t}}(s_{\ell+1}\mid s_{\ell},a_{\ell})\right]-\eta\sum_{\ell=0}^{L-1}H(p_{\phi_{t}}(s_{\ell+1}\mid s_{\ell},a_{\ell}))~~,~~ (9) where A_{\ell} is the advantage function computed using r_{\ell} , the value V(s_{\ell}) given by the critic network, and H is the entropy function of the distribution p_{\phi_{t}}(\cdot\mid s_{\ell},a_{\ell}) . This optimistic dynamics loss improves upon ( 7 ) by utilizing the advantage function A_{\ell} and the additional model entropy term H . These improvements mirror the advances in policy gradient algorithms to address issues such as high variance and entropy collapse. As a consequence, the optimistic dynamics loss is structurally the same as that of the actor loss and is simply obtained by substituting \log\pi_{\theta}(a_{\ell}|s_{\ell}) with \log p_{\phi}(s_{\ell+1}|s_{\ell},a_{\ell}) . The OWM framework can be adapted to any world model based on the framework described in Section 3 , such as Dreamer ( 23 ) , STORM ( 53 ) , TWM ( 44 ) , IRIS ( 42 ) , and DIAMOND ( 2 ) . In this paper, we propose two instances of OWMs: Optimistic DreamerV3 and Optimistic STORM. 4.2.1 Optimistic DreamerV3 We introduce Optimistic DreamerV3, the optimistic variant of DreamerV3. DreamerV3 is one of the leading MBRL algorithms known for its high sample efficiency without requiring task-specific hyperparameter tuning. Beyond the addition of the optimistic dynamics loss, Optimistic DreamerV3 is identical to the original DreamerV3; all other components and learning objectives remain unchanged. The imagined state s_{\ell} is comprised of a stochastic component z_{\ell} and a deterministic component h_{\ell} . Therefore, we replace p_{\phi}(s_{\ell+1}\mid s_{\ell},a_{\ell}) with p_{\phi}(z_{\ell+1}\mid h_{\ell+1}) . The optimistic dynamics loss term in Optimistic DreamerV3 is: \displaystyle{\mathcal{L}}_{t}^{opt}= \displaystyle-\alpha(t)\left[\sum_{\ell=0}^{L-1}A_{\ell}\log p_{\phi}(z_{\ell+1}\mid h_{\ell+1})\right]-\eta\sum_{\ell=0}^{L-1}H\left(p_{\phi}(z_{\ell+1}\mid h_{\ell+1})\right). (10) The advantage term A_{\ell} , is computed as \left(\frac{G_{\ell}^{\lambda}-V(s_{\ell})}{\max\{1,S\}}\right) . Here, G_{\ell}^{\lambda} is the \lambda -return, V is the value function estimated by the critic, \eta is an entropy coefficient, H is the entropy function, and S is an exponential moving average of the range between the 5th and 95th percentiles of returns, defined as, S=EMA(Per(G_{\ell}^{\lambda},95)-Per(G_{\ell}^{\lambda},5),0.99). 4.2.2 Optimistic STORM We propose Optimistic STORM (O-STORM), the optimistic variant of the STORM world model. 53 shows that the transformer-based STORM world model outperforms the DreamerV3 algorithm in many Atari games in the Atari100K benchmark. The optimistic dynamics loss function is computed as: \displaystyle{\mathcal{L}}_{t}^{opt}=- \displaystyle\alpha(t)\left[\sum_{\ell=0}^{L-1}A_{\ell}\log p_{\phi}(s_{\ell+1}\mid s_{\ell},a_{\ell})\right]-\eta\sum_{\ell=0}^{L-1}H(p_{\phi}(s_{\ell+1}\mid s_{\ell},a_{\ell})). The definition of the advantage function, A_{\ell} , is identical to the one from Optimistic DreamerV3. All other components and learning objectives of Optimistic STORM are identical to those of STORM and can be found in ( 53 ) . (a) Acrobot Swingup Sparse (b) Cartpole Swingup Sparse (c) Freeway (d) Private Eye (e) UpNDown (f) Freeway (O-STORM) (g) Private Eye (O-STORM) (h) UpNDown (O-STORM) Figure 3 : Optimistic World Models on sparse reward environments. 5 Experiments We evaluate the OWM variants: Optimistic DreamerV3 and Optimistic STORM on the widely adopted Atari100K and DeepMind Control (DMC) benchmarks. Our main results are presented in this section; comprehensive learning curves, interquartile mean (IQM) and median scores, ablations, and experimental details are provided in Appendix A . All learning curves are reported as mean reward \pm the standard error of the mean. The results for O-DreamerV3 on the Atari100K and the DMC benchmarks are averaged over 10 seeds. All other results, including those for Optimistic STORM and the ablation studies, are averaged over 5 seeds due to computational resource limitations. Performance on Sparse Reward Environments Figure 3 highlights the performance of OWMs on challenging, sparse-reward environments, including Private Eye, Freeway, Acrobot Swingup Sparse, and Cartpole Swingup Sparse. OWMs outperform baselines on these sparse environments, where standard world models suffer due to a lack of efficient exploration. They also outperform baselines on dense-reward environments such as UpNDown. Additionally, Figure 1 shows the long-term performance (40 million samples) of O-DreamerV3 and DreamerV3 on hard exploration Atari games, Private Eye and Montezuma’s Revenge. The O-DreamerV3 achieves nearly 2X mean return compared to DreamerV3 on Private Eye at 40M samples. While it achieves the same score as DreamerV3 on Montezuma’s Revenge with fewer samples. Performance on Atari100K Benchmark The Atari100K benchmark is designed to test the performance of RL algorithms in a limited sample regime. Optimistic DreamerV3 achieves a mean human-normalized score (HNS) of 152.68\% on the Atari100K benchmark, compared to 97.45\% mean HNS of DreamerV3. O-STORM outperforms STORM in sparse-reward games such as Private Eye and Freeway, notably achieving a positive score on Freeway, unlike STORM, DreamerV3, and O-DreamerV3. O-STORM achieves a mean HNS of 80.68\% , while STORM achieves a mean HNS of 75.90\% . Performance on DMC Suite We further evaluate the efficiency of Optimistic DreamerV3 on the DeepMind Control (DMC) suite. The DMC suite contains the DMC Proprio benchmark, which uses state-based inputs, and the DMC Vision benchmark, which uses image-based inputs. While many of the tasks in DMC benchmarks are easy, and both algorithms have similar performance on most DMC environments, O-DreamerV3 improves the performance on sparse-reward environments such as Cartpole Swingup Sparse, and Acrobot Swingup Sparse. Figures 4(b) and 4(c) summarize the performance on DMC Proprio and DMC Vision, respectively. (a) Optimistic DreamerV3 on Atari100K (b) Optimistic DreamerV3 on DMC Proprio (c) Optimistic DreamerV3 on DMC Vision (d) Optimistic STORM on Atari100K Figure 4 : Performance of Optimistic World Models: We plot the % gain of Optimistic DreamerV3 and Optimistic STORM over DreamerV3 and STORM respectively. Freeway is not included in Figure 4(d) as the baseline STORM has a score of 0 , while O-STORM achieves a mean score of 6.38 . The optimistic bias term \alpha(t) OWMs introduce two additional parameters, namely, the optimistic bias term \alpha(t) and the model entropy loss coefficient \eta . Similar to DreamerV3, O-DreamerV3 also enjoys a single hyperparameter configuration across all tasks and benchmarks. We used \alpha(t)=\alpha=0.0001 as a constant hyperparameter. Figure 5(d) shows the performance for different decay schedules of \alpha(t) . The entropy loss coefficient \eta is set to be 3\times 10^{-6} for O-DreamerV3 and 3\times 10^{-4} for O-STORM. Ablations on the entropy loss term in Figure 5(b) and 5(c) highlight the gain due to the entropy loss term. The ablations and sensitivity analysis shown in Figures 5 , 10 , 11 and 12 provide a clear guideline on choosing \alpha and \eta . They suggest that high values such as \alpha=0.1 or \eta=0.03 can degrade performance drastically, while smaller values are beneficial. Computational Overload OWMs are plug-and-play modifications over standard world models, adding marginal computational overhead compared to baselines, as shown in Table 1 . Table 1 : Training time (minutes) for various algorithms on Atari100K (MsPacman) using an RTX 4090 GPU, averaged over 5 seeds. O-DreamerV3 DreamerV3 O-STORM STORM 138 115 178 170 (a) \alpha (b) \eta (c) Entropy Loss (d) \alpha(t) Figure 5 : Ablations and hyperparameter sensitivity results for Cartpole Swingup Sparse (DMC Proprio): (a) Sensitivity to the optimism term \alpha : poor performance at \alpha=0.1 shows that optimism should be mild . (b) Sensitivity to the model entropy loss coefficient \eta : no learning at \eta=0.03 , while smaller values are beneficial. (c) Performance without the entropy loss: the entropy loss improves the performance of O-DreamerV3. (d) Performance under various decay schedules of \alpha(t) . Ablations for additional environments are provided in the Appendix. 6 Discussion RBMLE, UCB, and Thompson Sampling are three foundational principles for exploration studied in MBRL/adaptive control. RBMLE, first proposed in ( 31 ) , and UCB, first proposed in ( 33 ) , embody the principle of optimism as described below. A Tale of Two Approaches to Optimism: RBMLE and UCB For the MDP setup described in Section 3 , the general UCB principle chooses the model estimate p_{t} with the highest potential average reward within a confidence interval, as follows: \displaystyle p_{t}\in \displaystyle\arg\max_{p}J^{\star}(p) (11) \displaystyle\text{subject to}:D(p,\hat{p}_{t})\leq{\mathcal{C}}_{t}(\delta), where D is a measure of divergence from the MLE model \hat{p}_{t} , and {\mathcal{C}}_{t}(\delta) is a high-probability confidence interval. In this same scenario, RBMLE ( 3 ) can be equivalently seen as balancing the reward and fitting error of the model, i.e., \displaystyle p_{t}\in\arg\max_{p}\alpha(t)J^{\star}(p)-D(p,\hat{p}_{t}). (12) UCB incorporates optimism by solving the primal optimization problem, and controls the degree of optimism by {\mathcal{C}}_{t} . RBMLE incorporates optimism by solving its Lagrangian dual formulation, and controls the degree of optimism by \alpha(t) . RBMLE has been shown to outperform UCB and TS variants in various small-scale RL setups such as stochastic bandits ( 35 ) , contextual bandits ( 25 ) , tabular MDPs ( 41 ) , and LQG systems ( 38 ) . Similar approaches have been studied recently in different RL setups, such as RL with function approximation ( 36 ) , RLHF ( 14 ) , as well as model-free RL ( 36 ) . Several works ( 10 ; 19 ) have highlighted the excessive optimism of earlier variants of UCB algorithms, such as UCRL ( 6 ) and UCRL2 ( 5 ) , due to weak confidence intervals. UCRL2B ( 19 ) and UCRL3 ( 10 ) utilize sharper confidence intervals to curb the excessive optimism. In contrast, RBMLE incorporates mild optimism with small values of \alpha(t) . Objective Mismatch in MBRL Objective mismatch is another issue of the certainty equivalence-based MBRL discussed in ( 34 ; 18 ) . It is attributed to the fact that the model is trained using one objective of likelihood maximization, but the policy is trained using a different objective of average reward maximization ( 18 ) . The optimistic RBMLE objective ( 12 ) provides an explicit mechanism for balancing the two objectives of model accuracy and the cumulative rewards. Limitations and Future Work The optimistic dynamics loss provides an optimistic direction for exploration, and the step size is controlled by the optimistic bias term \alpha(t) . The theoretical results on RBMLE in tabular MDPs show that t\alpha(t)\to\infty and \alpha(t)\to 0 guarantee asymptotic convergence to an optimal policy. Choosing \alpha(t)=\alpha , or even \alpha(t)=\frac{\alpha}{\sqrt{t}} , is perhaps too simplistic to maximize exploration gains. An adaptive, sample-path dependent design of \alpha(t) , similar to the meta-controller used for choosing the exploration coefficient in Agent57 ( 7 ) , can further enhance the performance of OWMs. The regret bounds of RBMLE have been established in different settings ( 41 ; 38 ; 25 ; 35 ; 14 ) . In this paper, we have presented a completely gradient-based approach to RBMLE. The convergence analysis of the practical, gradient-based RBMLE is a natural next step. Conclusion We have introduced Optimistic World Models , a principled framework for efficient exploration in model-based reinforcement learning. The improved performance of OWMs compared to baselines, particularly in sparse-reward environments, validates the efficient exploration of OWMs. The simple, plug-and-play nature of OWMs allows their adaptation to many world models. Appendix A Empirical Results We discuss detailed results and the experimental setup in this section. A.1 Hyperparameters The OWM framework introduces two additional hyperparameters on top of the standard world model framework, namely the optimistic bias term \alpha(t) and the model entropy coefficient \eta . For O-DreamerV3, we set \alpha(t)=\alpha=0.0001 and \eta=3\times 10^{-6} . For O-STORM, we set \alpha(t)=\alpha=0.0001 and \eta=0.0003 . The sensitivity analysis of these parameters is provided in Figures 11 and 12 . All the other involved hyperparameters are set to the same values as the baseline algorithms and can be found in ( 23 ) for DreamerV3 and ( 53 ) for STORM. A.2 Results A.2.1 Evaluation Methodology The reported learning curves are obtained using 20 evaluation episodes at each checkpoint. We plot the mean reward \pm the standard error of the mean with a smoothing window of 3. The results for O-DreamerV3 on the Atari100K and the DMC benchmarks are averaged over 10 seeds. All other results, including O-STORM experiments and ablations, are averaged over 5 seeds due to computational resource limitations. A.2.2 Atari100K Benchmark The Atari100K benchmark is designed to evaluate the data efficiency of RL algorithms. It consists of 26 Atari games, where each agent has a fixed budget of 400K frames (equivalent to roughly 2 hours of gameplay) ( 23 ) . The results for O-DreamerV3 are provided in Figure 6 and Table 2 . The results for O-STORM are provided in Figure 7 and Table 3 . Table 2 : Comparison of DreamerV3 and Optimistic DreamerV3 on Atari100K benchmark. Higher scores are highlighted in color and formatted in bold. DreamerV3 O-DreamerV3 Game Random Human Mean IQM Median Mean IQM Median Alien 228 7128 821 800 800 948 996.67 1065 Amidar 6 1720 111.4 110.33 109 111 112 114 Assault 222 742 644.6 659.67 677 621.2 628.17 619.5 Asterix 210 8503 740 633.33 550 805 858.33 900 Bank Heist 14 753 79 73.33 65 67 65 65 Battle Zone 2360 37188 5000 4833.33 4500 4700 3666.67 3000 Boxing 0 12 79.3 79 79 74.3 76.67 78.5 Breakout 2 30 7.9 7.67 8 7.2 6.83 6.5 Chopper Command 811 7388 2030 2050 2050 2220 2116.67 1750 Crazy Climber 10780 35829 67390 68616.67 64450 64190 62900 56700 Demon Attack 152 1971 366 193.33 197.5 407.5 250.83 180 Freeway 0 30 0 0 0 0 0 0 Frostbite 65 4335 1117 860 300 1102 763.33 440 Gopher 258 2413 1356 1293.33 1190 1792 1513.33 1640 Hero 1027 30826 12799 13368.33 13382.5 12186 13277.5 13320 James Bond 29 303 330 325 325 360 333.33 350 Kangaroo 52 3035 2640 1566.67 1600 2140 1366.67 1300 Krull 1598 2666 8320 8736.67 8955 16816 7213.33 7215 Kung Fu Master 256 22736 22890 21100 20850 26640 26183.33 26100 Ms Pacman 307 6952 1446 1335 1245 1552 1605 1555 Pong -21 15 -5.6 -6.5 -7.5 -2.2 -4 -4.5 Private Eye 25 69571 893.9 147.17 100 1676.4 1297.83 100 Qbert 164 13455 1357.5 1116.67 1262.5 1137.5 787.5 787.5 Road Runner 12 7845 11330 11166.67 11550 15930 15350 16050 Seaquest 68 42055 708 660 680 914 860 860 Up N Down 533 11693 24954 14106.67 14185 91717 69470 51325 Figure 6 : Comparison of Optimistic DreamerV3 and DreamerV3 on the Atari100K Benchmark. Table 3 : Comparison of STORM and O-STORM on the Atari 100K benchmark. Higher scores are highlighted in color and formatted in bold. STORM O-STORM Game Random Human Mean IQM Median Mean IQM Median Alien 228 7128 1100.7 1060.83 1140.0 1131.2 992.5 1036.5 Amidar 6 1720 164.55 153.23 132.25 210.95 226.9 231.05 Assault 222 742 749.52 701.58 695.35 714.42 717.47 722.6 Asterix 210 8503 785.0 820.0 840.0 634.5 605.83 587.5 Bank Heist 14 753 378.3 376.0 437.0 445.7 458.83 528.0 Battle Zone 2360 37188 7470.0 7716.67 8250.0 5140.0 4883.33 5000.0 Boxing 0 12 58.4 62.3 60.5 66.25 66.62 65.05 Breakout 2 30 12.23 12.13 11.8 12.64 11.83 12.25 Chopper Command 811 7388 1371.0 1375.0 1355.0 1418.0 1358.33 1410.0 Crazy Climber 10780 35829 50567.0 49108.33 50710.0 51354.0 48655.0 48700.0 Demon Attack 152 1971 159.7 144.17 148.0 148.35 133.92 130.5 Freeway 0 30 0.0 0.0 0.0 6.38 0.0 0.0 Frostbite 65 4335 261.5 261.0 264.5 586.9 259.67 262.0 Gopher 258 2413 1600.6 1416.67 1339.0 1335.2 1162.67 883.0 Hero 1027 30826 13215.6 13436.83 13433.25 11305.6 11794.67 11398.0 James Bond 29 303 460.0 426.67 422.5 447.5 444.17 425.0 Kangaroo 52 3035 2287.0 2030.0 1370.0 1578.0 1636.67 1590.0 Krull 1598 2666 4882.5 5006.83 5701.5 6294.5 6259.17 6451.5 Kung Fu Master 256 22736 24545.0 25591.67 24025.0 17608.0 17090.0 13655.0 Ms Pacman 307 6952 1923.0 2045.17 2026.0 2008.5 1985.5 1883.0 Pong -21 15 7.07 9.37 9.6 -0.75 -0.87 -0.65 Private Eye 25 69571 115.53 115.33 100.0 2120.48 1668.87 100.0 Qbert 164 13455 2489.25 2574.58 2502.5 3271.0 3432.92 3591.25 Road Runner 12 7845 13057.0 12866.67 11165.0 10161.0 9013.33 8625.0 Seaquest 68 42055 444.2 452.33 451.0 421.8 411.0 375.0 Up N Down 533 11693 5585.2 4550.67 4529.5 8111.0 7772.5 7888.0 Figure 7 : Comparison of Optimistic STORM and STORM on the Atari100K Benchmark. A.2.3 Deepmind Control (DMC) Suite We also present the performance of O-DreamerV3 on the DeepMind Control (DMC) suite, a popular reinforcement learning benchmark for continuous control tasks. The DMC suite consists of two benchmarks, the DMC proprio control benchmark and the DMC vision control benchmark. The DMC proprio control benchmark consists of continuous control tasks with proprioceptive vector inputs and continuous actions. Each task has a budget of 500K environment steps. Meanwhile, the DMC vision control benchmark consists of continuous control tasks with high-dimensional images as inputs. Each task has a budget of 1M environment steps. The learning curves are provided in Figures 8 and 9 , while the final performance is highlighted in Tables 4 and 5 . Table 4 : Comparison of Optimistic DreamerV3 and DreamerV3 on DMC Proprio tasks. Higher scores are highlighted in color and formatted in bold. DreamerV3 O-DreamerV3 Task Mean IQM Median Mean IQM Median Acrobot Swingup 150.33 133.51 127.65 236.82 232.42 244.35 Acrobot Swingup Sparse 8.4 3.83 1.5 34.6 32.5 28.0 Cartpole Balance 990.23 995.43 995.69 993.0 994.19 995.13 Cartpole Balance Sparse 964.0 1000.0 1000.0 1000.0 1000.0 1000.0 Cartpole Swingup 838.83 857.02 857.77 855.11 855.16 855.81 Cartpole Swingup sparse 664.2 755.83 762.0 747.1 780.17 784.0 Cheetah Run 522.19 559.83 617.94 591.27 611.16 612.4 Cup Catch 959.2 962.33 962.0 948.5 957.17 959.5 Finger Spin 609.9 600.67 569.5 624.3 613.33 604.5 Finger Turn Easy 946.9 957.83 974.5 964.6 973.33 980.5 Finger Turn Hard 703.5 847.5 913.0 823.1 909.83 905.5 Hopper Hop 86.92 76.14 62.78 71.42 55.92 40.63 Hopper Stand 465.78 481.12 487.12 431.97 429.69 480.77 Pendulum Swingup 765.4 843.0 855.0 827.0 853.5 869.5 Reacher Easy 929.2 970.17 968.5 965.5 965.5 966.5 Reacher Hard 959.5 958.0 955.5 959.3 959.33 959.0 Walker Run 492.96 498.63 497.83 481.94 491.25 491.48 Walker Stand 952.7 969.36 976.3 959.76 963.81 967.57 Walker Walk 875.84 882.97 883.94 878.62 888.17 903.99 Table 5 : Comparison of Optimistic DreamerV3 and DreamerV3 on DMC Vision tasks. Higher scores are highlighted in color and formatted in bold. DreamerV3 O-DreamerV3 Task Mean IQM Median Mean IQM Median Acrobot Swingup 258.64 254.44 241.71 219.42 219.97 249.51 Acrobot Swingup Sparse 14.2 1.83 0.0 38.6 28.67 36.0 Cartpole Swingup 866.97 866.42 866.54 866.94 866.68 866.72 Cartpole Swingup Sparse 751.7 758.33 757.0 795.5 794.17 791.0 Cheetah Run 900.88 900.95 900.22 892.85 897.01 899.91 Cup Catch 971.9 970.0 970.0 973.0 970.33 970.0 Finger Spin 709.6 672.83 620.5 629.1 623.0 624.0 Finger Turn Easy 796.0 937.33 940.0 849.8 941.83 946.5 Finger Turn Hard 943.3 941.67 943.5 936.6 946.5 955.0 Hopper Hop 303.49 292.11 289.86 299.31 298.52 298.87 Hopper Stand 935.54 930.68 929.79 932.77 936.21 933.34 Pendulum Swingup 777.3 850.83 843.5 838.9 844.67 848.5 Quadruped Run 482.32 493.46 481.41 528.24 510.29 501.88 Quadruped Walk 831.16 856.04 867.86 799.58 819.48 840.62 Reacher Easy 983.2 984.0 984.0 984.6 984.0 985.0 Reacher Hard 961.4 964.67 962.5 933.0 970.17 972.5 Walker Run 714.8 751.72 751.36 750.3 756.55 754.9 Walker Stand 983.39 982.98 984.18 985.11 984.88 983.69 Walker Walk 960.23 963.05 963.67 966.63 968.27 968.07 Figure 8 : Comparison of Optimistic DreamerV3 and DreamerV3 on the DMC Proprio Benchmark. Figure 9 : Comparison of Optimistic DreamerV3 and DreamerV3 on the DMC Vision Benchmark. A.3 Ablation Studies Our optimistic dynamics loss consists of two terms, the optimism term and the entropy term. Figure 10 shows the performance without the entropy loss term, highlighting the improvement due to the introduction of the entropy loss. The sensitivity analysis of the two hyperparameters \alpha and \eta is provided in Figures 11 and 12 respectively. These results show that high values, such as \alpha=0.1 , \eta=0.03 are detrimental to performance, while smaller values enhance performance. A.4 Computation Resources The O-DreamerV3 code is implemented using the official open-source code of DreamerV3 available at https://github.com/danijar/DreamerV3 . Similarly, O-STORM is implemented using the official STORM code available at https://github.com/weipu-zhang/STORM . Each experiment was completed using one A100 GPU. The computational time experiments were run on a local RTX 4090 GPU to ensure fairness. (a) Cartpole Swing. Sparse (b) MsPacman (c) Pong Figure 10 : Ablation study of model entropy loss term for Optimistic DreamerV3 (a) Cartpole Swing. Sparse (b) MsPacman (c) Pong Figure 11 : Performance of O-DreamerV3 for various values of \alpha with \eta=3\times 10^{-6} . (a) Cartpole Swing. Sparse (b) MsPacman (c) Pong Figure 12 : Performance of O-DreamerV3 for various values of \eta with \alpha=0.0001 .