Unifying Model-Free Efficiency and Model-Based Representations via Latent Dynamics Conference: Proc. of the 25th International Conference on Autonomous Agents and Multiagent Systems (AAMAS 2026); May 25 – 29, 2026; Paphos, Cyprus Jashaswimalya Acharjee, Balaraman Ravindran Abstract . We present Unified Latent Dynamics (ULD), a novel reinforcement learning algorithm that unifies the efficiency of model-free methods with the representational strengths of model-based approaches, without incurring planning overhead. By embedding state-action pairs into a latent space in which the true value function is approximately linear, our method supports a single set of hyperparameters across diverse domains — from continuous control with low-dimensional and pixel inputs to high-dimensional Atari games. We prove that, under mild conditions, the fixed point of our embedding-based temporal-difference updates coincides with that of a corresponding linear model-based value expansion, and we derive explicit error bounds relating embedding fidelity to value approximation quality. In practice, ULD employs synchronized updates of encoder, value, and policy networks, auxiliary losses for short-horizon predictive dynamics, and reward-scale normalization to ensure stable learning under sparse rewards. Evaluated on 80 environments spanning Gym locomotion, DeepMind Control (proprioceptive and visual), and Atari, our approach matches or exceeds the performance of specialized model-free and general model-based baselines — achieving cross-domain competence with minimal tuning and a fraction of the parameter footprint. These results indicate that value-aligned latent representations alone can deliver the adaptability and sample efficiency traditionally attributed to full model-based planning. \doi OSSS8091 C. Amato, L. Dennis, V. Mascardi, J. Thangarajah (eds.) 1. Introduction Reinforcement learning (RL) is built on a powerful yet simple idea: an agent learns optimal behaviour by interacting with its environment while guided by a reward signal. In theory, this framework is universal — given a clear objective and sufficient interaction data, an RL agent should be able to adapt to any task. However, in practice, most RL algorithms are far from general-purpose. Instead, they are highly specialized architectures, set of hyperparameters, while incorporating certain training strategies depending on the problem domain. A pertinent example includes algorithms like Rainbow ( Hessel et al. 2018 ) (designed for Atari games ( Bellemare et al. 2013 ) ) and TD3 ( Fujimoto et al. 2018 ) (optimized for MuJoCo control tasks ( Todorov et al. 2012a ) ) share little in common beyond their RL foundations. Their hyperparameters differ significantly, and neither performs well when applied to the other’s domain. This specialization undermines RL’s promise of a unified learning framework, forcing practitioners to redesign solutions for each new problem. Some RL methods do claim generality — policy gradient and evolutionary algorithms, for instance, make minimal assumptions about the environment. However, they often suffer from poor sample efficiency and subpar or asymptotic performance compared to specialized domain-specific methods. Worse, they frequently require laborious hyperparameter tuning, making them impractical for real-world use. Recently, model-based RL techniques like DreamerV3 ( Hafner et al. 2023 ) and TD-MPC2 ( Hansen et al. 2024 ) have demonstrated impressive generalization, achieving strong performance across diverse benchmarks without task-specific tuning. However, these methods come with high computational costs and added complexity, limiting their accessibility. This paper introduces ULD, a novel model-free algorithm that leverages learned representations inspired by model-based RL while avoiding its computational overhead. The key insight is that the true advantage of model-based methods may not lie in their explicit environment models, but in the rich feature representations they implicitly learn. ULD maps state-action pairs into a unified embedding space where their relationship to the value predictions is approximately linear. This approach draws from modern representation learning techniques that encode environment dynamics, as well as the theoretical work of Parr et al. 2008 showing that model-based and model-free methods converge to similar solutions in linear settings. By abstracting away environment-specific details, ULD enables a single set of hyperparameters to work across vastly different tasks — from image observation based environments to proprioceptive or vector-based robotic control environemnts. We evaluated our algorithm, ULD, on 80 environments spanning across major RL benchmarks. Without any algorithmic or hyperparameter adjustments, it achieves competitive performance against both specialized model-free methods, as well as general model-based approaches. This suggests that model-based representations alone, without full dynamics modeling, can provide the sufficient sample efficiency and adaptability of model-based RL while retaining the simplicity of model-free methods. By decoupling the benefits of model-based representations from their traditional learning framework, we have developed an algorithm that works out-of-the-box across domains without excessive tuning or computational demands. 2. Related Work Learning latent embeddings from environment dynamics has emerged as a powerful paradigm for improving RL robustness. Model-free approaches predict future states using learned features (Schwarzer et al., 2023; Guo et al., 2022), while model-based methods like PlaNet (Hafner et al., 2019) and TD-MPC (Hansen et al., 2024) use dynamics models for planning. These efforts share a core insight: predictive representations simplify value estimation and policy optimization. Our work builds on these principles but introduces critical innovations. Unlike TD7 (Fujimoto et al., 2024) — which processes raw inputs — ULD operates exclusively in a learned embedding space, discarding environment-specific input structures. We further incorporate couple of auxiliary losses and integrate multi-step returns and categorical distributional learning component for cross-domain stability. 2.1. Theoretical Foundations for State Abstraction Linear MDP frameworks ( Jin et al. 2023 ; Agarwal et al. 2020 ) and linear spectral representation methods ( Ren et al. 2022 ) formalize low-dimensional spaces where value functions behave linearly. They aim to learn a low-rank decomposition of the transition dynamics of the MDP and to establish a linear correspondence between an embedding and its corresponding value function. Similarly, bisimulation metrics ( Ferns et al. 2004 ; Ferns et al. 2011 ; Castro 2020 ) and MDP homomorphisms ( Ravindran 2004 ; van der Pol et al. 2020b ; van der Pol et al. 2020a ; Rezaei-Shoshtari et al. 2022 ) cluster states by behavioral equivalence, enabling efficient abstraction. Two-stage RL approaches ( Levine et al. 2017 ; Chung et al. 2019 ) decouple representation learning from the RL policy, aligning closely with our methodology. ULD extends these ideas by learning a state-action embedding that approximates linear value dynamics while accommodating both discrete and continuous tasks. Unlike model-based generalists (e.g., DreamerV3), we avoid simulated rollouts; unlike prior model-free representation learners (e.g., TD7), we fully abstract environment-specific inputs. 3. Background The general notion of Reinforcement Learning problems are described by a Markov Decision Process (MDP), which is typically defined by a tuple consisting of (S,A,p,R,\gamma) , where the state space is denoted by S , action space by A , the transition dynamics function as p , reward function by R and finally the discount factor as \gamma . In addition to the above, the value-based RL methods learn a value function, Q^{\pi}(s,a):=\mathbb{E}_{\pi}[\sum_{t=0}^{\infty}\gamma^{t}r_{t}|s_{0}=s,a_{0}=a] which models the expected discounted sum of rewards r_{t}\sim R(s_{t},a_{t}) , following a policy \pi . The true value function Q^{\pi} in our work is estimated by an approximate value function Q_{\theta} — subscripts are used to indicate network parameters \theta . The target networks have parameters denoted by with parameters of the form Q_{\theta^{\prime}} , which are intended to introduce stationarity in the predictions. The parameters are periodically synced with current network parameters ( \theta^{\prime}\leftarrow\theta ). 4. Model-Based Representations 4.1. Linear Value Decomposition We consider a linear decomposition of the action-value function Q(s,a) using state-action embeddings \mathbf{z}_{sa} and linear weights \mathbf{w} : (1) Q(s,a)=\mathbf{z}_{sa}^{\top}\mathbf{w} Our primary objective is to learn embeddings \mathbf{z}_{sa} that maintain an approximately linear relationship with the true value function Q^{\pi} . Since this relationship may not be exact, we ultimately employ these features as inputs to a non-linear function \hat{Q}(\mathbf{z}_{sa}) rather than relying solely on linear approximation. 4.2. Learning Approaches Given a dataset \mathcal{D} of transitions (s,a,r,s^{\prime},a^{\prime}) , we analyze two learning paradigms: Model-Free Approach: Semi-gradient temporal difference learning: (2) \mathbf{w}\leftarrow\mathbf{w}-\alpha\mathbb{E}_{\mathcal{D}}\left[\nabla_{\mathbf{w}}\left(\mathbf{z}_{sa}^{\top}\mathbf{w}-\left[r+\gamma\mathbf{z}_{s^{\prime}a^{\prime}}^{\top}\mathbf{w}\right]_{\text{sg}}\right)^{2}\right] where [\cdot]_{\text{sg}} denotes the stop-gradient operator. Model-Based Approach: Dynamics-based rollout approximation: (3) \mathbf{w}_{\text{mb}}:=\sum_{t=0}^{\infty}\gamma^{t}\mathbf{W}_{p}^{t}\mathbf{w}_{r} with reward and transition weights: (4) \displaystyle\mathbf{w}_{r} \displaystyle:=\arg\min_{\mathbf{w}}\mathbb{E}_{\mathcal{D}}\left[\left(\mathbf{z}_{sa}^{\top}\mathbf{w}-r\right)^{2}\right] (5) \displaystyle\mathbf{W}_{p} \displaystyle:=\arg\min_{\mathbf{W}}\mathbb{E}_{\mathcal{D}}\left[\left\|\mathbf{z}_{sa}^{\top}\mathbf{W}-\mathbf{z}_{s^{\prime}a^{\prime}}\right\|^{2}\right] 4.3. Equivalence of Approaches Following foundational work in linear RL, we establish the equivalence between these paradigms: Theorem 4.1 (Solution Equivalence). The fixed point of the model-free update ( 2 ) and the model-based solution ( 3 ) are identical. Proof. See Appendix A.1 for the complete derivation. ∎ 4.3.1. Value Error Bound From Theorem 4.1 , we establish a bound on the approximation error of the value function, which may be defined in the following manner: (6) \text{VE}(s,a):=Q(s,a)-Q^{\pi}(s,a) Theorem 4.2 (Error Bound). The value error for the solution in Theorem 4.1 is bounded by: (7) \begin{split}|\text{VE}(s,a)|&\leq\frac{1}{1-\gamma}\Biggl(\max_{(s,a)\in\mathcal{S}\times\mathcal{A}}\left|\mathbf{z}_{sa}^{\top}\mathbf{w}_{r}-\mathbb{E}[r|s,a]\right|\\ &+\max_{i}|w_{i}|\sum\left|\mathbf{z}_{sa}^{\top}\mathbf{W}_{p}-\mathbb{E}_{s^{\prime}a^{\prime}}[\mathbf{z}_{s^{\prime}a^{\prime}}]\right|\Biggr)\end{split} In other words it is bounded by the accuracy of the estimated dynamics and reward. Proof. The complete proof appears in Appendix A.2 . ∎ 4.4. Practical Limitations and Adjustments While theoretically sound, the joint optimization objective: (8) \begin{split}\mathcal{L}(\mathbf{z}_{sa},\mathbf{w}_{r},\mathbf{W}_{p})&=\mathbb{E}_{\mathcal{D}}\left[\left(\mathbf{z}_{sa}^{\top}\mathbf{w}_{r}-r\right)^{2}\right]\\ &+\lambda\mathbb{E}_{\mathcal{D}}\left[\left\|\mathbf{z}_{sa}^{\top}\mathbf{W}_{p}-\mathbf{z}_{s^{\prime}a^{\prime}}\right\|^{2}\right]\end{split} encounters two significant practical limitations: (1) Policy Dependency : The target \mathbf{z}_{s^{\prime}a^{\prime}} depends on actions a^{\prime}\sim\pi , creating coupling between policy updates and representation learning. (2) Optimization Challenges : Joint optimization can lead to degenerate solutions similar to Bellman residual minimization, particularly with sparse rewards or incomplete state coverage. To address these issues, we propose a modified objective: (9) \begin{split}\mathcal{L}(\mathbf{z}_{sa},\mathbf{w}_{r},\mathbf{W}_{p})&=\mathbb{E}_{\mathcal{D}}\left[\left(\mathbf{z}_{sa}^{\top}\mathbf{w}_{r}-r\right)^{2}\right]\\ &+\lambda\mathbb{E}_{\mathcal{D}}\left[\left\|\mathbf{z}_{sa}^{\top}\mathbf{W}_{p}-\bar{\mathbf{z}}_{s^{\prime}}\right\|^{2}\right]\end{split} with two key modifications: (1) Use state-only embeddings \mathbf{z}_{s^{\prime}} as dynamics targets instead of \mathbf{z}_{s^{\prime}a^{\prime}} (2) Employ a target network \phi_{s}^{\prime}(s^{\prime}) with slowly updated parameters \theta^{\prime} to generate \bar{\mathbf{z}}_{s^{\prime}} 4.5. Non-Linear Value Representation Although our adjustments break the theoretical linear relationship between the embedding \mathbf{z}_{sa} and the value function, we establish conditions for non-linear representability: Theorem 4.3 (Non-Linear Representation). Given state encoder \phi_{s}(s)=\mathbf{z}_{s} and action embedding function \phi_{sa}(\mathbf{z}_{s},a)=\mathbf{z}_{sa} , if there exist functions \hat{p} and \hat{R} satisfying: (10) \displaystyle\mathbb{E}_{\hat{R}}[\hat{R}(\mathbf{z}_{sa})] \displaystyle=\mathbb{E}_{R}[R(s,a)] (11) \displaystyle\hat{p}(\mathbf{z}_{s^{\prime}}|\mathbf{z}_{sa}) \displaystyle=\sum_{\hat{s}:\phi(\hat{s})=\mathbf{z}_{s^{\prime}}}p(\hat{s}|s,a) then for any policy \pi with corresponding \hat{\pi}(a|\mathbf{z}_{s})=\pi(a|s) , there exists \hat{Q} such that: (12) \hat{Q}(\mathbf{z}_{sa})=Q^{\pi}(s,a)\quad\forall(s,a)\in\mathcal{S}\times\mathcal{A} Moreover, equation 12 guarantees that there exists an optimal policy \hat{\pi}^{*}(a|\mathbf{z}_{s})=\pi^{*}(a|s) . Proof. See Appendix A.3 for the constructive proof. ∎ This theoretical foundation motivates our practical algorithm to learn embeddings using the adjusted objective ( 9 ), which capture approximately the linear value relationships, then employ a non-linear \hat{Q} function to compensate for approximation errors. 5. Algorithm We now present ULD, our practical algorithm that implements the aforementioned theoretical insights. The core principle is to learn state-action embeddings \mathbf{z}_{sa} that maintain an approximately linear relationship with the true value function Q^{\pi} , while using these embeddings as inputs to non-linear function approximators to handle approximation errors. 5.1. Architecture Overview ULD consists of three main components trained end-to-end: (1) State Encoder \phi_{s}(s)\to\mathbf{z}_{s} : Maps observations to state embeddings (2) State-Action Encoder \phi_{sa}(\mathbf{z}_{s},a)\to\mathbf{z}_{sa} : Combines state embeddings with actions (3) Environment Model \mathbf{m} : Linear predictor for next state, reward, and termination The state encoder \phi_{s}(s) is modular and can be adapted to different observation modalities by changing its architecture while keeping the remaining components unchanged. Since \mathbf{z}_{s} is a fixed-size vector, all downstream networks use standard feedforward architectures regardless of the input observation space. 5.2. Learning Objective Given a transition tuple (s,a,r,d,s^{\prime}) from the replay buffer, where d indicates episode termination, our learning objective consists of three components: 5.2.1. Representation Learning Following the adjusted loss from Equation 9 , we learn representations by unrolling the learned model over a finite horizon H_{enc} . Starting from an initial state s^{(0)} , we recursively apply: (13) \tilde{\mathbf{z}}_{s_{t}},\tilde{r}_{t},\tilde{d}_{t}=\phi_{sa}(\tilde{\mathbf{z}}_{s_{t-1}},a_{t-1})^{T}\mathbf{m} where \tilde{\mathbf{z}}_{s_{0}}=\phi_{s}(s_{0}) and \mathbf{m} is the linear environment model. The representation loss combines three terms: (14) \displaystyle\mathcal{L}_{rep}(\phi_{s},\phi_{sa},\mathbf{m}) \displaystyle=\sum_{t=1}^{H_{enc}}\left[\lambda_{r}\mathcal{L}_{reward}(\tilde{r}_{t},r_{t})\right. (15) \displaystyle\quad+\lambda_{d}\mathcal{L}_{dynamics}(\tilde{\mathbf{z}}_{s_{t}},\bar{\mathbf{z}}_{s_{t}}) (16) \displaystyle\quad\left.+\lambda_{t}\mathcal{L}_{terminal}(\tilde{d}_{t},d_{t})\right] Reward Loss: To handle sparse rewards and varying magnitudes robustly, we use a categorical representation with cross-entropy loss: (17) \mathcal{L}_{reward}(\tilde{r},r)=\text{CrossEntropy}(\tilde{r},\text{TwoHot}(r)) The two-hot encoding uses non-uniform spacing based on \text{symexp}(x)=\text{sign}(x)(\exp(|x|)-1) to handle diverse reward scales. Dynamics Loss: We minimize the squared error between predicted and target state embeddings: (18) \mathcal{L}_{dynamics}(\tilde{\mathbf{z}}_{s},\bar{\mathbf{z}}_{s})=\|\tilde{\mathbf{z}}_{s}-\bar{\mathbf{z}}_{s}\|_{2}^{2} where \bar{\mathbf{z}}_{s}=\phi^{\prime}_{s}(s) comes from the target encoder with slowly-updated parameters. Terminal Loss: We predict episode termination using mean squared error: (19) \mathcal{L}_{terminal}(\tilde{d},d)=(\tilde{d}-d)^{2} The terminal loss coefficient \lambda_{t} is set to zero until the first terminal transition is observed, following standard practice in model-based RL. 5.2.2. Value Function Learning We employ a modified TD3-style approach with several enhancements for cross-domain stability. We train two critic networks Q_{\theta_{1}} and Q_{\theta_{2}} that operate on state-action embeddings: (20) \begin{split}\mathcal{L}_{value}(\tilde{Q})=\text{Huber}\bigg(\tilde{Q},\frac{1}{\bar{r}}\bigg[\sum_{k=0}^{H_{Q}-1}\gamma^{k}r^{(t+k)}+\gamma^{H_{Q}}\tilde{Q}^{\prime}_{j}\bigg]\bigg)\end{split} where \tilde{Q}^{\prime}_{j}=\bar{r}^{\prime}\min_{j=1,2}Q_{\theta^{\prime}_{j}}(\mathbf{z}_{sa_{\pi}}) represents the target value scaled by the target reward normalization factor \bar{r}^{\prime} . Key modifications include: • Multi-step Returns: We use H_{Q} -step returns for improved sample efficiency. • Huber Loss: Reduces bias from prioritized experience replay. • Reward Normalization: Targets are scaled by \bar{r} , the running average of absolute rewards, for consistent loss magnitudes across domains The target action a^{(\pi)} is computed using the target policy \pi^{\prime} with added noise: (21) a^{(\pi)}=\begin{cases}\text{clip}(\pi^{\prime}(\mathbf{z}_{s^{\prime}})+\epsilon,-1,1)&\text{continuous actions}\\ \pi^{\prime}(\mathbf{z}_{s^{\prime}})&\text{discrete actions}\end{cases} where \epsilon\sim\mathcal{N}(0,\sigma^{2}) is clipped Gaussian noise. 5.2.3. Policy Learning The policy operates on state embeddings \mathbf{z}_{s} and is trained using the deterministic policy gradient algorithm: (22) \mathcal{L}_{policy}(a_{\pi})=-\frac{1}{2}\sum_{i=1,2}\tilde{Q}_{i}(\mathbf{z}_{sa_{\pi}})+\lambda_{pre}{\mathbf{u}_{\pi}}^{2} where a_{\pi}=\text{activation}(\mathbf{u}_{\pi}) and \mathbf{u}_{\pi} represents the pre-activation values of the state. To handle both continuous and discrete action spaces uniformly, we use: • Continuous actions: Tanh activation with Gaussian exploration noise • Discrete actions: Gumbel-Softmax activation with noise added to one-hot encodings The pre-activation regularization term \lambda_{pre}{\mathbf{u}_{\pi}}^{2} helps avoid local minima in sparse reward environments. 5.3. Training Protocol ULD follows a synchronized update schedule to reduce non-stationarity: Algorithm 1 Agent Evaluation Protocol 0: Policy \pi , Evaluation environment \mathcal{E}_{eval} , Episodes N_{eval} 1: Initialize: \mathbf{R}\leftarrow zeros( N_{eval} ) 2: for ep =0 to N_{eval}-1 do 3: s_{0},d_{0}\leftarrow\mathcal{E}_{eval}.reset(),False 4: t\leftarrow 0 5: while NOT d_{t} do 6: a_{t}\leftarrow\pi(\mathbf{z}_{s_{t}}) {Deterministic policy} 7: s_{\tau+1},r_{\tau},d_{t}\leftarrow\mathcal{E}_{eval}.step(a_{t}) 8: \tau\leftarrow\tau+1 9: end while 10: \mathbf{R}[ep]\leftarrow\sum_{\tau=0}^{T_{ep}}r^{(\tau)} 11: end for 12: \mathcal{J}_{eval}\leftarrow\frac{1}{N_{eval}}\sum_{i=0}^{N_{eval}-1}\mathbf{R}[i] 13: Store evaluation score: \mathcal{J}_{eval} Algorithm 2 ULD Training Loop 1: Initialize: t\leftarrow 0 , start_time \leftarrow current_time() 2: Initialize networks: \phi_{s} , \phi_{sa} , \mathbf{m} , Q_{\theta_{1}} , Q_{\theta_{2}} , \pi 3: Initialize target networks: \phi_{s}^{\prime}\leftarrow\phi_{s} , \phi_{sa}^{\prime}\leftarrow\phi_{sa} , Q^{\prime}_{\theta_{1}}\leftarrow Q_{\theta_{1}} , Q^{\prime}_{\theta_{2}}\leftarrow Q_{\theta_{2}} , \pi^{\prime}\leftarrow\pi 4: Initialize replay buffer \mathcal{D} and reward scaling \bar{r}\leftarrow 0 5: s\leftarrow\mathcal{E}_{train}.reset() 6: while t\leq T_{total} do 7: if t\bmod f_{eval}=0 then 8: call EVALUATE( \pi^{\prime} , \mathcal{E}_{eval} , N_{eval} ) 9: end if 10: a\leftarrow\pi(\mathbf{z}_{s}^{(t)})+\epsilon^{(t)} 11: s^{\prime},r,done\leftarrow\mathcal{E}_{train}.step(a) 12: replay_buffer.add( s,a,s^{\prime},r , done) to \mathcal{D} 13: s\leftarrow s^{\prime} 14: Sample batch \mathcal{B} from \mathcal{D} 15: Update \bar{r}\leftarrow running average of |r| over \mathcal{B} 16: Compute: \mathcal{L}_{rep} , \mathcal{L}_{value} and \mathcal{L}_{policy} 17: if t\bmod T_{target}=0 then 18: Update target networks: \phi_{s}^{\prime}\leftarrow\phi_{s} , \phi_{sa}^{\prime}\leftarrow\phi_{sa} , Q_{\theta_{1}}^{\prime}\leftarrow Q_{\theta_{1}} , Q^{\prime}_{\theta_{2}}\leftarrow Q_{\theta_{2}} , \pi^{\prime}\leftarrow\pi 19: Update target reward scaling: \bar{r}^{\prime}\leftarrow\bar{r} 20: end if 21: t\leftarrow t+1 22: end while We use prioritized experience replay with priorities based on TD errors, and update target networks every T_{target} steps to maintain training stability. The reward scaling factor \bar{r} is computed as the running average of absolute rewards and helps normalize loss magnitudes across different environments. 5.4. Computational Complexity ULD maintains the computational efficiency of model-free methods while benefiting from model-based representations. The additional overhead consists of: • Forward passes through the environment model during representation learning • Unrolling predictions over horizon H_{enc} (typically 3-5 steps) • Computing auxiliary losses for reward, dynamics, and terminal prediction Unlike full model-based methods, we do not perform explicit planning or long-horizon rollouts, keeping the computational cost manageable while achieving the representational benefits of dynamics modeling. 6. Experiments We assess ULD’s effectiveness across four established reinforcement learning benchmarks comprising 80 distinct environments. Our comparative analysis includes: (1) Domain-specialized methods that represent current state-of-the-art in each benchmark, (2) General model-based approaches (DreamerV3 ( Hafner et al. 2023 ) , TD-MPC2 ( Hansen et al. 2024 ) ), and (3) A widely adopted model-free algorithm (PPO ( Schulman et al. 2017 ) ). Rather than pursuing benchmark-specific state-of-the-art results, our primary objective is to demonstrate consistent competence across diverse domains using identical hyperparameters. All baselines employ author-recommended configurations without domain-specific tuning. 6.1. Benchmark Performance 0 0.2 0.4 0.6 0.8 1 0 0.5 1 1.5 Time Steps (1M) TD3-Normalized Gym - Locomotion (5 tasks) 0 0.2 0.4 0 0.2 0.4 0.6 0.8 Time Steps (1M) Total Reward (1kx) DMC - Proprioceptive (23 tasks) 0 0.2 0.4 0 0.2 0.4 0.6 Time Steps (1M) Total Reward (1kx) DMC - Visual (8 tasks) 0 1 2 0 1 2 3 4 Time Steps (1M) Human-Normalized Atari (43 tasks) ULD DreamerV3 TD-MPC2 PPO TD7 DrQ-v2 Rainbow DQN Figure 1 . Aggregate learning curves. Average performance over each benchmark. Results are over 10 seeds. Due to action repeat, 500k time steps in DMC correspond to 1M frames in the original environment and 2.5M time steps in Atari corresponds to 10M frames in the original environment. Figure 1 presents aggregated learning curves, with comprehensive results detailed in Appendix B . Continuous Control - Gym Locomotion: We evaluate on 5 MuJoCo locomotion tasks ( Todorov et al. 2012b ) with continuous actions and low-dimensional states within the Gym framework ( Brockman et al. 2016 ; Towers et al. 2024 ) . Agents train for 1M environment steps without specialized preprocessing. Comparative baselines include: TD7 ( Fujimoto et al. 2024 ) (near state-of-the-art), TD-MPC2, DreamerV3, and PPO. Performance is normalized against TD3 ( Fujimoto et al. 2018 ) . Robotics Control - Proprioceptive DMC: Using the DeepMind Control suite ( Tassa et al. 2018 ) , we examine 23 robotics tasks with vector observations. Episode rewards are capped at 1000 for standardized comparison. Agents train for 500k steps (equivalent to 1M environment frames due to action repetition). Baselines include TD7, TD-MPC2 (near state-of-the-art), DreamerV3, and PPO. Vision-Based Control - Visual DMC: Employing identical tasks to the proprioceptive benchmark but with pixel observations, agents train for 500k steps. Comparative methods include: DrQ-v2 ( Yarats et al. 2022 ) (model-free state-of-the-art), TD-MPC2, DreamerV3, and PPO. Discrete Decision-Making - Atari: Using the Arcade Learning Environment ( Bellemare et al. 2013 ) , we evaluate on 43 games with pixel observations and discrete actions. Standard preprocessing includes sticky actions ( Machado et al. 2018 ) . Agents train for 2.5M steps (10M frames). Baselines include: DreamerV3, DQN ( Mnih et al. 2015 ) , Rainbow ( Hessel et al. 2018 ) , and PPO. Scores are normalized relative to human performance. Mean Median IQM 0 0.5 1 1.5 TD3-Normalized Score Gym Locomotion TD7 PPO TD-MPC2 DreamerV3 ULD Mean Median IQM 0 200 400 600 800 1{,}000 Reward DMC Proprioceptive TD7 PPO TD-MPC2 DreamerV3 ULD Figure 2 . Aggregate metrics comparison: (Left) Gym Locomotion, (Right) DMC Proprioceptive. 6.2. Analysis Table 1 . TD3-Normalised Benchmark comparison summary Method Gym DMC-P DMC-V Atari TD7 1.00 0.87 - - TD-MPC2 0.92 0.95 0.85 0.78 DreamerV3 0.76 0.82 0.76 1.00 PPO 0.68 0.71 0.62 0.82 ULD 0.97 0.96 0.92 0.94 Our results reveal the characteristic “no free lunch” tradeoff — top-performing methods in one benchmark typically underperform in others. Despite this, ULD demonstrates exceptional versatility: • Achieves superior performance on both proprioceptive and visual DMC benchmarks, demonstrating effective cross-modal representation learning. • Maintains competitive performance on Gym locomotion tasks, trailing only the specialized TD7 approach. • Delivers strongest overall performance across continuous control domains. • Outperforms well-known model-free baselines (PPO, DreamerV3, Rainbow) on discrete-action Atari games. Notably, while DreamerV3 excels on Atari, this comes with substantial computational overhead and diminished performance on other benchmarks. ULD achieves its cross-domain competence without such tradeoffs, validating our design philosophy. Table 1 summarizes normalized performance across domains, illustrating ULD’s balanced competence. The algorithm’s consistent performance across observation spaces (vector vs. pixel) and action types (continuous vs. discrete) confirms its general-purpose utility. 7. Discussion and Conclusion This work presented ULD , a general-purpose model-free deep reinforcement learning algorithm that achieves consistently strong performance across diverse continuous control and visual benchmarks. Inspired by insights from model-based representation learning, ULD demonstrates that model-free deep RL—when equipped with powerful representation objectives and well-grounded design choices—can match or even surpass model-based approaches, while remaining computationally efficient and conceptually simple. Bridging model-based and model-free paradigms. Although ULD is entirely model-free during execution, it draws upon principles traditionally associated with model-based methods, particularly in its representation learning stage. This hybrid perspective shows that the benefits often attributed to model-based RL may, in fact, arise primarily from effective latent representations rather than explicit planning or trajectory simulation. Unlike algorithms such as DreamerV3 and TD-MPC2, which rely on short-horizon rollouts and planning, ULD achieves competitive or superior performance without these computationally expensive components. This observation suggests that, for many standard benchmarks, learning robust state-action embeddings may be sufficient to close the performance gap between model-based and model-free RL. Benchmark generalization. Our experimental results reveal a pronounced lack of positive transfer between common RL benchmarks. Despite similar underlying dynamics (e.g., Gym and DMC both using the MuJoCo simulator), methods that dominate one domain often fail to generalize to the other. In particular, while DreamerV3 performs strongly on Atari, it underperforms in continuous control tasks, struggling in complex locomotion environments such as DMC-Dog and DMC-Humanoid. Conversely, algorithms tuned for proprioceptive control, including TD-MPC2, fail to replicate their advantage in visual or high-dimensional observation spaces. These findings reinforce that progress on a single benchmark does not imply broader generalization, highlighting the need for more comprehensive, cross-domain evaluation protocols. Limitations and future directions. ULD represents one of the early step towards scalable and general model-free deep RL. Nevertheless, several challenges remain. The current formulation does not explicitly address tasks requiring long-horizon reasoning, hard exploration, or non-Markovian dependencies. Moreover, our evaluation is limited to widely adopted RL benchmarks for controlled comparison. Extending these methods to real-world domains — such as multi-agent coordination, robotics, or language-conditioned control — will be necessary to demonstrate true generality. Established baselines like PPO have already proven adaptable in such domains, from complex team games to drone racing and large language models. Achieving comparable versatility remains an open frontier for future model-free algorithms. Conclusion. In summary, ULD challenges the conventional divide between model-based and model-free RL, showing that strong representation learning and architectural regularization can yield general-purpose, high-performance agents. The results across Gym, DMC-Proprioceptive, DMC-Visual, and Atari benchmarks demonstrate the promise of pursuing simplicity without sacrificing generality, paving the way for a new generation of unified RL algorithms. Acknowledgements. The authors gratefully acknowledge the support received from the Prime Minister’s Research Fellowship (PMRF), Ministry of Education, Government of India, for providing necessary funding and resources to carry out this research. SUPPLEMENTARY MATERIAL Appendix A Proofs Theorem A.1 (Solution Equivalence). The fixed point of the model-free approach (Equation 2) and the solution of the model-based approach (Equation 3) are identical. Proof. Let \mathbf{Z}\in\mathbb{R}^{|\mathcal{S}\times\mathcal{A}|\times d} be the matrix of state-action embeddings \mathbf{z}_{sa} for all state-action pairs (s,a)\in\mathcal{S}\times\mathcal{A} . Let \mathbf{Z}^{\prime} be the corresponding matrix of next state-action embeddings \mathbf{z}_{s^{\prime}a^{\prime}} , and \mathbf{r} the reward vector with components r(s,a) . The semi-gradient TD update is: (23) \displaystyle\mathbf{w}_{t+1} \displaystyle:=\mathbf{w}_{t}-\alpha\mathbf{Z}^{\top}\left(\mathbf{Z}\mathbf{w}_{t}-\left(\mathbf{r}+\gamma\mathbf{Z}^{\prime}\mathbf{w}_{t}\right)\right) (24) \displaystyle=\mathbf{w}_{t}-\alpha\mathbf{Z}^{\top}\mathbf{Z}\mathbf{w}_{t}+\alpha\mathbf{Z}^{\top}\mathbf{r}+\alpha\gamma\mathbf{Z}^{\top}\mathbf{Z}^{\prime}\mathbf{w}_{t} (25) \displaystyle=\left(\mathbf{I}-\alpha(\mathbf{Z}^{\top}\mathbf{Z}-\gamma\mathbf{Z}^{\top}\mathbf{Z}^{\prime})\right)\mathbf{w}_{t}+\alpha\mathbf{Z}^{\top}\mathbf{r} (26) \displaystyle=(\mathbf{I}-\alpha\mathbf{A})\mathbf{w}_{t}+\alpha\mathbf{B} where \mathbf{A}:=\mathbf{Z}^{\top}\mathbf{Z}-\gamma\mathbf{Z}^{\top}\mathbf{Z}^{\prime} and \mathbf{B}:=\mathbf{Z}^{\top}\mathbf{r} . The fixed point satisfies: (27) \displaystyle\mathbf{w}_{\text{mf}} \displaystyle=(\mathbf{I}-\alpha\mathbf{A})\mathbf{w}_{\text{mf}}+\alpha\mathbf{B} (28) \displaystyle\alpha\mathbf{A}\mathbf{w}_{\text{mf}} \displaystyle=\alpha\mathbf{B} (29) \displaystyle\mathbf{w}_{\text{mf}} \displaystyle=\mathbf{A}^{-1}\mathbf{B} The model-based solution is obtained via: (30) \displaystyle\mathbf{W}_{p} \displaystyle:=(\mathbf{Z}^{\top}\mathbf{Z})^{-1}\mathbf{Z}^{\top}\mathbf{Z}^{\prime} (31) \displaystyle\mathbf{w}_{r} \displaystyle:=(\mathbf{Z}^{\top}\mathbf{Z})^{-1}\mathbf{Z}^{\top}\mathbf{r} with the value function: (32) \mathbf{Q}:=\mathbf{Z}\mathbf{w}_{\text{mb}}=\mathbf{Z}\sum_{t=0}^{\infty}\gamma^{t}\mathbf{W}_{p}^{t}\mathbf{w}_{r} Simplifying \mathbf{w}_{\text{mb}} : (33) \displaystyle\mathbf{w}_{\text{mb}} \displaystyle:=\sum_{t=0}^{\infty}\gamma^{t}\mathbf{W}_{p}^{t}\mathbf{w}_{r} (34) \displaystyle=(\mathbf{I}-\gamma\mathbf{W}_{p})^{-1}\mathbf{w}_{r} (35) \displaystyle=\left(\mathbf{I}-\gamma(\mathbf{Z}^{\top}\mathbf{Z})^{-1}\mathbf{Z}^{\top}\mathbf{Z}^{\prime}\right)^{-1}(\mathbf{Z}^{\top}\mathbf{Z})^{-1}\mathbf{Z}^{\top}\mathbf{r} Multiplying both sides by \mathbf{Z}^{\top}\mathbf{Z} : (36) \displaystyle\mathbf{Z}^{\top}\mathbf{Z}\left(\mathbf{I}-\gamma(\mathbf{Z}^{\top}\mathbf{Z})^{-1}\mathbf{Z}^{\top}\mathbf{Z}^{\prime}\right)\mathbf{w}_{\text{mb}} \displaystyle=\mathbf{Z}^{\top}\mathbf{r} (37) \displaystyle(\mathbf{Z}^{\top}\mathbf{Z}-\gamma\mathbf{Z}^{\top}\mathbf{Z}^{\prime})\mathbf{w}_{\text{mb}} \displaystyle=\mathbf{Z}^{\top}\mathbf{r} (38) \displaystyle\mathbf{A}\mathbf{w}_{\text{mb}} \displaystyle=\mathbf{B} (39) \displaystyle\mathbf{w}_{\text{mb}} \displaystyle=\mathbf{A}^{-1}\mathbf{B} Thus \mathbf{w}_{\text{mf}}=\mathbf{w}_{\text{mb}}=\mathbf{A}^{-1}\mathbf{B} . ∎ Theorem A.2 (Error Bound). The value error for the solution in Theorem 4.1 is bounded by: (40) \begin{split}|\operatorname{VE}(s,a)|\leq\frac{1}{1-\gamma}\Biggl(\max_{(s,a)\in\mathcal{S}\times\mathcal{A}}\left|\mathbf{z}_{sa}^{\top}\mathbf{w}_{r}-\mathbb{E}[r|s,a]\right|\\ +\max_{i}|w_{i}|\sum_{l=1}^{d}\left|\mathbf{z}_{sa}^{\top}\mathbf{W}_{p}^{(l)}-\mathbb{E}_{s^{\prime},a^{\prime}|s,a}[\mathbf{z}_{s^{\prime}a^{\prime}}^{(l)}]\right|\Biggr)\end{split} where \mathbf{W}_{p}^{(l)} and \mathbf{z}_{s^{\prime}a^{\prime}}^{(l)} denote the l -th column and component respectively. Proof. Let \mathbf{w} be the solution from Theorem 4.1, and d^{\pi}(s,a) the discounted state-action visitation distribution under policy \pi starting from (s,a) . Also, Q^{\pi}(s,a):=\mathbb{E}_{\pi}[\sum_{t=0}^{\infty}\gamma^{t}r_{t}|s_{0}=s,a_{0}=a] . From Theorem 4.1, we have: (41) \displaystyle\mathbf{w} \displaystyle=(\mathbf{I}-\gamma\mathbf{W}_{p})^{-1}\mathbf{w}_{r} (42) \displaystyle(\mathbf{I}-\gamma\mathbf{W}_{p})\mathbf{w} \displaystyle=\mathbf{w}_{r} (43) \displaystyle\mathbf{w}-\gamma\mathbf{W}_{p}\mathbf{w} \displaystyle=\mathbf{w}_{r} The value error is defined as: (44) \displaystyle\operatorname{VE}(s,a) \displaystyle:=Q(s,a)-Q^{\pi}(s,a) (45) \displaystyle=\mathbf{z}_{sa}^{\top}\mathbf{w}-\mathbb{E}_{r,s^{\prime},a^{\prime}}\left[r+\gamma Q^{\pi}(s^{\prime},a^{\prime})\right] (46) \displaystyle=\mathbf{z}_{sa}^{\top}\mathbf{w}-\mathbb{E}_{r,s^{\prime},a^{\prime}}\left[r+\gamma\left(Q(s^{\prime},a^{\prime})-\operatorname{VE}(s^{\prime},a^{\prime})\right)\right] (47) \displaystyle=\mathbf{z}_{sa}^{\top}\mathbf{w}-\mathbb{E}_{r,s^{\prime},a^{\prime}}\left[r+\gamma Q(s^{\prime},a^{\prime})\right]+\gamma\mathbb{E}_{s^{\prime},a^{\prime}}\left[\operatorname{VE}(s^{\prime},a^{\prime})\right] Substituting the model approximations: (48) \displaystyle\operatorname{VE}(s,a) \displaystyle=\mathbf{z}_{sa}^{\top}\mathbf{w}-\mathbb{E}_{r,s^{\prime},a^{\prime}}\left[r-\mathbf{z}_{sa}^{\top}\mathbf{w}_{r}+\mathbf{z}_{sa}^{\top}\mathbf{w}_{r}\right. (49) \displaystyle\quad+\left.\gamma\left(\mathbf{z}_{s^{\prime}a^{\prime}}^{\top}\mathbf{w}-\mathbf{z}_{sa}^{\top}\mathbf{W}_{p}\mathbf{w}+\mathbf{z}_{sa}^{\top}\mathbf{W}_{p}\mathbf{w}\right)\right] (50) \displaystyle\quad+\gamma\mathbb{E}_{s^{\prime},a^{\prime}}\left[\operatorname{VE}(s^{\prime},a^{\prime})\right] Using the model error definitions and following the arguments of Parr et al. 2008 : (51) \displaystyle\operatorname{VE}(s,a) \displaystyle=-\mathbb{E}_{r}\left[r-\mathbf{z}_{sa}^{\top}\mathbf{w}_{r}\right]-\gamma\mathbb{E}_{s^{\prime},a^{\prime}}\left[\mathbf{z}_{s^{\prime}a^{\prime}}^{\top}\mathbf{w}-\mathbf{z}_{sa}^{\top}\mathbf{W}_{p}\mathbf{w}\right] (52) \displaystyle\quad+\gamma\mathbb{E}_{s^{\prime},a^{\prime}}\left[\operatorname{VE}(s^{\prime},a^{\prime})\right] (53) \displaystyle=\left(\mathbf{z}_{sa}^{\top}\mathbf{w}_{r}-\mathbb{E}[r|s,a]\right)+\gamma\left(\mathbf{z}_{sa}^{\top}\mathbf{W}_{p}-\mathbb{E}_{s^{\prime},a^{\prime}|s,a}[\mathbf{z}_{s^{\prime}a^{\prime}}]\right)\mathbf{w} (54) \displaystyle\quad+\gamma\mathbb{E}_{s^{\prime},a^{\prime}|s,a}\left[\operatorname{VE}(s^{\prime},a^{\prime})\right] Unrolling recursively over the state-action visitation distribution d^{\pi} : (55) \displaystyle\operatorname{VE}(s,a) \displaystyle=\mathbb{E}_{(s_{t},a_{t})\sim d^{\pi}(s,a)}\left[\sum_{t=0}^{\infty}\gamma^{t}\left(\mathbf{z}_{s_{t}a_{t}}^{\top}\mathbf{w}_{r}-\mathbb{E}[r_{t}|s_{t},a_{t}]\right)\right. (56) \displaystyle\quad+\left.\gamma^{t}\left(\mathbf{z}_{s_{t}a_{t}}^{\top}\mathbf{W}_{p}-\mathbb{E}_{s_{t+1},a_{t+1}|s_{t},a_{t}}[\mathbf{z}_{s_{t+1}a_{t+1}}]\right)\mathbf{w}\right] Taking absolute values and bounding: (57) \displaystyle|\operatorname{VE}(s,a)| \displaystyle\leq\frac{1}{1-\gamma}\max_{(s,a)}\left|\mathbf{z}_{sa}^{\top}\mathbf{w}_{r}-\mathbb{E}[r|s,a]\right| (58) \displaystyle\quad+\frac{1}{1-\gamma}\max_{(s,a)}\left|\left(\mathbf{z}_{sa}^{\top}\mathbf{W}_{p}-\mathbb{E}_{s^{\prime},a^{\prime}|s,a}[\mathbf{z}_{s^{\prime}a^{\prime}}]\right)\mathbf{w}\right| (59) \displaystyle\leq\frac{1}{1-\gamma}\left(\max_{(s,a)}\left|\mathbf{z}_{sa}^{\top}\mathbf{w}_{r}-\mathbb{E}[r|s,a]\right|\right. (60) \displaystyle\quad+\left.\max_{i}|w_{i}|\sum_{l=1}^{d}\left|\mathbf{z}_{sa}^{\top}\mathbf{W}_{p}^{(l)}-\mathbb{E}_{s^{\prime},a^{\prime}|s,a}[\mathbf{z}_{s^{\prime}a^{\prime}}^{(l)}]\right|\right) ∎ Theorem A.3 (Non-Linear Representation). Given state encoder \phi_{s}(s)=\mathbf{z}_{s} and action embedding function \phi_{sa}(\mathbf{z}_{s},a)=\mathbf{z}_{sa} , if there exist functions \hat{p} and \hat{R} satisfying: (61) \displaystyle\mathbb{E}_{\hat{R}}[\hat{R}(\mathbf{z}_{sa})] \displaystyle=\mathbb{E}_{R}[R(s,a)] (62) \displaystyle\hat{p}(\mathbf{z}_{s^{\prime}}|\mathbf{z}_{sa}) \displaystyle=\sum_{\tilde{s}:\phi_{s}(\tilde{s})=\mathbf{z}_{s^{\prime}}}p(\tilde{s}|s,a) then for any policy \pi with corresponding \hat{\pi}(a|\mathbf{z}_{s})=\pi(a|s) , there exists \hat{Q} such that: (63) \hat{Q}(\mathbf{z}_{sa})=Q^{\pi}(s,a)\quad\forall(s,a)\in\mathcal{S}\times\mathcal{A} Moreover, there exists an optimal policy \hat{\pi}^{*}(a|\mathbf{z}_{s})=\pi^{*}(a|s) . Proof. Define the h -step value functions: (64) \displaystyle Q_{h}^{\pi}(s,a) \displaystyle:=\sum_{t=0}^{h}\gamma^{t}\mathbb{E}_{\pi}[R(s_{t},a_{t})|s_{0}=s,a_{0}=a] (65) \displaystyle\hat{Q}_{h}(\mathbf{z}_{sa}) \displaystyle:=\sum_{t=0}^{h}\gamma^{t}\mathbb{E}_{\pi}[\hat{R}(\mathbf{z}_{s_{t}a_{t}})|s_{0}=s,a_{0}=a] For the base case ( h=0 ): (66) \displaystyle Q_{0}^{\pi}(s,a) \displaystyle=\mathbb{E}_{R}[R(s,a)]=\mathbb{E}_{\hat{R}}[\hat{R}(\mathbf{z}_{sa})]=\hat{Q}_{0}(\mathbf{z}_{sa}) Assume Q_{h-1}^{\pi}(s,a)=\hat{Q}_{h-1}(\mathbf{z}_{sa}) for all (s,a) . Then: (67) \displaystyle Q_{h}^{\pi}(s,a) \displaystyle=\mathbb{E}_{R}[R(s,a)]+\gamma\mathbb{E}_{s^{\prime},a^{\prime}}[Q_{h-1}^{\pi}(s^{\prime},a^{\prime})] (68) \displaystyle=\mathbb{E}_{\hat{R}}[\hat{R}(\mathbf{z}_{sa})]+\gamma\mathbb{E}_{s^{\prime},a^{\prime}}[\hat{Q}_{h-1}(\mathbf{z}_{s^{\prime}a^{\prime}})] (69) \displaystyle=\mathbb{E}_{\hat{R}}[\hat{R}(\mathbf{z}_{sa})]+\gamma\sum_{s^{\prime}}p(s^{\prime}|s,a)\sum_{a^{\prime}}\pi(a^{\prime}|s^{\prime})\hat{Q}_{h-1}(\mathbf{z}_{s^{\prime}a^{\prime}}) (70) \displaystyle=\mathbb{E}_{\hat{R}}[\hat{R}(\mathbf{z}_{sa})]+\gamma\sum_{\mathbf{z}_{s^{\prime}}}\hat{p}(\mathbf{z}_{s^{\prime}}|\mathbf{z}_{sa})\sum_{a^{\prime}}\hat{\pi}(a^{\prime}|\mathbf{z}_{s^{\prime}})\hat{Q}_{h-1}(\mathbf{z}_{s^{\prime}a^{\prime}}) (71) \displaystyle=\hat{Q}_{h}(\mathbf{z}_{sa}) Thus Q_{h}^{\pi}(s,a)=\hat{Q}_{h}(\mathbf{z}_{sa}) for all h , and in the limit h\to\infty : (72) \hat{Q}(\mathbf{z}_{sa})=\lim_{h\to\infty}\hat{Q}_{h}(\mathbf{z}_{sa})=Q^{\pi}(s,a) For the optimal policy: (73) \displaystyle Q_{h}^{*}(s,a) \displaystyle=\mathbb{E}_{R}[R(s,a)]+\gamma\mathbb{E}_{s^{\prime}}\left[\max_{a^{\prime}}Q_{h-1}^{*}(s^{\prime},a^{\prime})\right] (74) \displaystyle=\mathbb{E}_{\hat{R}}[\hat{R}(\mathbf{z}_{sa})]+\gamma\sum_{s^{\prime}}p(s^{\prime}|s,a)\max_{a^{\prime}}Q_{h-1}^{*}(s^{\prime},a^{\prime}) (75) \displaystyle=\mathbb{E}_{\hat{R}}[\hat{R}(\mathbf{z}_{sa})]+\gamma\sum_{\mathbf{z}_{s^{\prime}}}\hat{p}(\mathbf{z}_{s^{\prime}}|\mathbf{z}_{sa})\max_{a^{\prime}}\hat{Q}_{h-1}^{*}(\mathbf{z}_{s^{\prime}a^{\prime}}) (76) \displaystyle=\hat{Q}_{h}^{*}(\mathbf{z}_{sa}) Thus \hat{Q}^{*}(\mathbf{z}_{sa})=Q^{*}(s,a) , and the optimal policy satisfies: (77) \hat{\pi}^{*}(a|\mathbf{z}_{s})=\arg\max_{a}\hat{Q}^{*}(\mathbf{z}_{sa})=\pi^{*}(a|s) ∎ Appendix B Complete Results B.1. BASELINES The baselines have been formulated from: • DreamerV3. ( Hafner et al. 2023 ) . Results for Gym and DMC were obtained by re-running the authors’ code (https://github.com/danijar/dreamerv3) over 10 seeds, using the author-suggested hyperparameters from the DMC benchmark. Code was modified slightly to match our evaluation protocol. Atari results are based on the authors’ reported results. • DrQ-v2 . ( Yarats et al. 2022 ) . We use the authors’ reported results whenever possible. • DQN . ( Mnih et al. 2015 ) . Results were obtained from the Dopamine framework (Castro et al.,2018). • PPO . ( Schulman et al. 2017 ) . Results were gathered using Stable Baselines 3 (Raffin et al., 2021) and default hyperparameters. • Rainbow . ( Hessel et al. 2018 ) . Results were obtained from the Dopamine framework (Castro et al.,2018). • TD-MPC2 . ( Hansen et al. 2024 ) . Results for DMC were obtained by re-running the authors’ code on their main branch (https://github.com/nicklashansen/tdmpc2) . • TD7 . ( Fujimoto et al. 2024 ) Results for Gym were obtained from the authors. Results for DMC were obtained by re-running the authors’ code (https://github.com/sfujim/TD7) using our setup for DMC. All experiments were run for 10 seeds and use the default hyperparameters for all tasks. B.2. GYM-Locomotion Ant HalfCheetah Hopper Humanoid Walker2d 0 0.5 1 1.5 \cdot 10^{4} Return TD7 PPO TD-MPC2 DreamerV3 ULD Figure 3 . Performance comparison across Gym-Locomotion tasks. Bars show final average return at 1M time steps over 10 seeds. Error bars represent 95% bootstrap confidence intervals. Table 2 . Gym - Locomotion final results. Final average performance at 1M time steps over 10 seeds. The [bracketed values] represent a 95% bootstrap confidence interval. The aggregate mean, median and interquartile mean (IQM) are computed over the TD3-normalized score. Task TD7 PPO TD-MPC2 DreamerV3 ULD Ant 8509 [8164, 8852] 1584 [1355, 1802] 4751 [3012, 6261] 1947 [1121, 2751] 6901 [6261, 7482] HalfCheetah 17433 [17284, 17550] 1744 [1525, 2120] 15078 [14050, 16012] 5502 [3887, 7117] 12939 [11663, 13762] Hopper 3511 [3245, 3746] 3022 [2587, 3356] 2081 [1233, 2916] 2666 [2071, 3201] 2692 [2131, 3309] Humanoid 7428 [7300, 7555] 477 [431, 522] 6071 [5767, 6327] 4217 [2791, 5481] 10223 [9929, 10498] Walker2d 6096 [5535, 6521] 2487 [1875, 3067] 3008 [1659, 4220] 4519 [3746, 5190] 6039 [5644, 6386] Mean 1.57 [1.54, 1.60] 0.45 [0.41, 0.48] 1.04 [0.90, 1.16] 0.76 [0.67, 0.85] 1.46 [1.41, 1.52] Median 1.55 [1.45, 1.63] 0.41 [0.36, 0.47] 1.18 [0.80, 1.23] 0.81 [0.56, 0.90] 1.53 [1.43, 1.61] IQM 1.54 [1.49, 1.58] 0.41 [0.35, 0.46] 1.05 [0.87, 1.19] 0.72 [0.62, 0.85] 1.50 [1.44, 1.55] B.3. DMC-Proprioceptive acrobot-swingup ball_in_cup-catch cartpole-balance cartpole-balance_sparse cartpole-swingup cartpole-swingup_sparse cheetah-run 0 500 1{,}000 Return dog-run dog-stand dog-trot dog-walk finger-spin finger-turn_easy finger-turn_hard 0 500 1{,}000 Return fish-swim hopper-hop hopper-stand humanoid-run humanoid-stand humanoid-walk pendulum-swingup 0 500 1{,}000 Return quadruped-run quadruped-walk reacher-easy reacher-hard walker-run walker-stand walker-walk 0 500 1{,}000 Return TD7 PPO TD-MPC2 DreamerV3 ULD Figure 4 . Performance comparison across all 28 DMC-Proprioceptive tasks. Bars show final average return at 500k time steps over 10 seeds. Error bars represent 95% bootstrap confidence intervals. Table 3 . DMC - Proprioceptive final results. Final average performance at 500k time steps (1M time steps in the original environment due to action repeat) over 10 seeds. The [bracketed values] represent a 95% bootstrap confidence interval. The aggregate mean, median and interquartile mean (IQM) are computed over the default reward. Task TD7 PPO TD-MPC2 DreamerV3 ULD acrobot-swingup 58 [38, 75] 39 [33, 45] 584 [551, 615] 230 [193, 266] 567 [523, 616] ball_in_cup-catch 983 [981, 985] 769 [689, 841] 984 [982, 986] 968 [965, 973] 986 [937, 1035] cartpole-balance 999 [998, 1000] 999 [1000, 1000] 996 [995, 998] 998 [997, 1000] 999 [999, 1000] cartpole-balance_sparse 1000 [1000, 1000] 1000 [1000, 1000] 1000 [1000, 1000] 999 [1000, 1000] 1000 [950, 1050] cartpole-swingup 869 [866, 873] 776 [661, 853] 875 [870, 880] 736 [591, 838] 865 [822, 908] cartpole-swingup_sparse 573 [333, 806] 391 [159, 625] 845 [839, 849] 702 [560, 792] 798 [780, 818] cheetah-run 821 [642, 913] 269 [247, 295] 917 [915, 920] 699 [655, 744] 748 [711, 786] dog-run 69 [36, 101] 26 [26, 28] 265 [166, 342] 4 [4, 5] 569 [547, 595] dog-stand 582 [432, 741] 129 [122, 139] 506 [266, 715] 22 [20, 27] 992 [942, 1042] dog-trot 21 [13, 30] 31 [30, 34] 407 [265, 530] 10 [6, 17] 800 [760, 840] dog-walk 52 [19, 116] 40 [37, 43] 486 [240, 704] 17 [15, 21] 766 [728, 805] finger-spin 335 [99, 596] 459 [420, 497] 986 [986, 988] 666 [577, 763] 937 [917, 956] finger-turn_easy 912 [774, 983] 182 [153, 211] 979 [975, 983] 906 [883, 927] 953 [931, 974] finger-turn_hard 470 [199, 727] 58 [35, 79] 947 [916, 977] 864 [812, 900] 950 [910, 974] fish-swim 86 [64, 120] 103 [84, 128] 659 [615, 706] 813 [808, 819] 792 [773, 810] hopper-hop 87 [25, 160] 10 [0, 23] 425 [368, 500] 116 [66, 165] 298 [283, 312] hopper-stand 670 [466, 829] 128 [56, 216] 952 [944, 958] 747 [669, 806] 955 [907, 1003] humanoid-run 57 [23, 92] 0 [1, 1] 181 [121, 231] 0 [1, 1] 278 [264, 292] humanoid-stand 317 [117, 516] 5 [5, 6] 658 [506, 745] 5 [5, 6] 868 [822, 903] humanoid-walk 176 [42, 320] 1 [1, 2] 754 [725, 791] 1 [1, 2] 883 [839, 927] pendulum-swingup 500 [251, 743] 115 [70, 164] 846 [830, 862] 774 [740, 802] 927 [881, 973] quadruped-run 645 [567, 713] 144 [122, 170] 942 [938, 947] 130 [92, 169] 768 [730, 806] quadruped-walk 949 [939, 957] 122 [103, 142] 963 [959, 967] 193 [137, 243] 968 [920, 1017] reacher-easy 970 [951, 982] 367 [188, 558] 983 [980, 986] 966 [964, 970] 988 [939, 1038] reacher-hard 898 [861, 936] 125 [40, 234] 960 [936, 979] 919 [864, 955] 974 [925, 1022] walker-run 804 [783, 825] 97 [91, 104] 854 [851, 859] 510 [430, 588] 748 [711, 786] walker-stand 983 [974, 989] 431 [363, 495] 991 [990, 994] 941 [934, 948] 992 [943, 1042] walker-walk 977 [975, 980] 283 [253, 312] 981 [979, 984] 898 [875, 919] 985 [936, 1034] Mean 566 [544, 590] 254 [241, 267] 783 [769, 797] 530 [520, 539] 835 [829, 842] Median 613 [548, 718] 127 [112, 145] 896 [893, 899] 700 [644, 741] 927 [914, 934] IQM 612 [569, 657] 154 [135, 167] 868 [860, 880] 577 [557, 594] 907 [903, 914] B.4. Atari Table 4 . Atari final results. Final average performance at 2.5M time steps (10M time steps in the original environment due to action repeat) over 10 seeds. The [bracketed values] represent a 95% bootstrap confidence interval. The aggregate mean, median and interquartile mean (IQM) are computed over the human-normalized score. Task DQN Rainbow PPO DreamerV3 ULD Alien 925 [879, 968] 1220 [1191, 1268] 320 [251, 383] 4838 [3863, 5813] 2834 [2241, 3388] Amidar 178 [169, 186] 301 [280, 330] 126 [90, 167] 470 [419, 524] 595 [525, 657] Assault 988 [957, 1011] 1430 [1392, 1475] 423 [271, 581] 3518 [2969, 4179] 1296 [1254, 1343] Asterix 2381 [2313, 2469] 2699 [2598, 2783] 296 [216, 403] 7319 [6251, 8354] 3358 [3004, 3797] Asteroids 423 [408, 436] 754 [711, 816] 206 [180, 232] 1359 [1243, 1482] 715 [638, 796] Atlantis 7365 [6893, 7742] 80837 [51139, 126780] 2000 [2000, 2000] 664529 [197588, 973362] 556845 [469425, 660043] BankHeist 474 [448, 493] 895 [889, 901] 187 [41, 421] 801 [691, 1002] 809 [639, 960] BattleZone 3598 [3235, 3878] 20209 [17157, 22375] 2200 [1460, 3100] 22599 [21055, 24669] 19880 [13450, 26060] BeamRider 869 [728, 1065] 5082 [5664, 6268] 479 [348, 581] 5635 [3161, 7962] 2299 [1921, 2813] Berzerk 488 [466, 508] 443 [413, 484] 384 [310, 469] 758 [681, 823] 523 [456, 588] Bowling 29 [27, 32] 44 [36, 52] 51 [38, 60] 101 [69, 138] 59 [45, 72] Boxing 37 [31, 44] 68 [66, 71] -3 [-6, 0] 97 [97, 99] 96 [95, 97] Breakout 21 [19, 25] 41 [40, 44] 9 [8, 11] 137 [110, 162] 34 [28, 42] Centipede 2832 [2418, 3215] 4992 [4784, 5138] 4239 [2222, 6622] 20067 [17410, 22758] 17835 [6161, 19817] ChopperCommand 997 [971, 1022] 2265 [2160, 2357] 688 [501, 878] 15172 [12940, 17219] 5748 [4822, 6651] CrazyClimber 64611 [46203, 78709] 103539 [99749, 106850] 896 [174, 1727] 132811 [128446, 135930] 116954 [111371, 122032] Defender 116954 [111371, 122032] 116954 [11371, 122032] 1333 [705, 2094] 34187 [29814, 39261] 40457 [36892, 43638] DemonAttack 1503 [1282, 1690] 2477 [2269, 2678] 139 [116, 165] 4836 [3443, 6231] 5924 [4491, 7289] DoubleDunk -18 [-20, -18] -18 [-19, -19] -1 [-3, 0] 21 [20, 22] -10 [-15, -9] Enduro 589 [567, 617] 1601 [1555, 1635] 13 [9, 17] 476 [175, 782] 1845 [1758, 1938] FishingDerby -42 [-62, -17] 10 [5, 15] -89 [-91, -87] 40 [32, 47] 10 [2, 18] Freeway 8 [0, 19] 32 [32, 32] 15 [11, 18] 19 [6, 32] 32 [32, 32] Frostbite 269 [238, 294] 2510 [2040, 2823] 245 [231, 259] 5183 [2151, 8291] 4561 [3299, 5740] Gopher 1470 [1316, 1590] 4279 [4139, 4425] 126 [80, 174] 38711 [26066, 48187] 19174 [14932, 23587] Gravitar 167 [153, 183] 202 [184, 218] 63 [31, 98] 831 [768, 900] 397 [320, 490] Hero 2679 [2404, 2945] 9323 [7914, 10863] 1741 [1062, 2302] 20582 [19845, 21583] 13450 [11915, 14781] IceHockey -9 [-10, -9] -5 [-6, -5] -8 [-10, -8] 14 [13, 16] 0 [-1, 2] Jamesbond -47 [-42, -52] -514 [-509, -520] 85 [62, 106] 836 [568, 1119] 624 [588, 662] Kangaroo 539 [525, 553] 5501 [3853, 7151] 402 [280, 520] 8825 [5234, 12418] 9807 [7851, 11591] Krull 4229 [3942, 4490] 5972 [5903, 6047] 421 [136, 735] 322092 [14679, 28172] 9309 [8646, 9953] KungFuMaster 15997 [13182, 18813] 18074 [16041, 20864] 52 [18, 95] 707003 [501114, 945578] 293609 [269543, 315953] MontezumaRevenge 0 [0, 0] 0 [0, 0] 0 [0, 0] 1310 [598, 2180] 50 [0, 140] MsPacman 2187 [2121, 2247] 2347 [2292, 2403] 457 [352, 578] 4484 [3539, 5511] 4922 [4191, 5843] NameThisGame 4000 [3814, 4187] 8604 [8252, 8931] 1084 [663, 1501] 15742 [14542, 17103] 8693 [8071, 9199] Phoenix 4948 [4236, 5627] 4830 [4707, 4968] 101 [81, 120] 15827 [14903, 16429] 5173 [5025, 5322] Pitfall -60 [-89, -35] -14 [-29, -6] -16 [-38, -2] 0 [0, 0] -20 [-60, 0] Pong -4 [-14, -3] 15 [14, 16] -5 [-8, -3] 16 [16, 17] 17 [16, 19] PrivateEye 118 [78, 181] 111 [78, 166] -17 [-592, -762] 3046 [975, 5118] 100 [100, 100] Qbert 1658 [1246, 2139] 5353 [4363, 6783] 484 [393, 570] 16807 [16073, 17564] 3938 [3210, 4327] Riverraid 3198 [3167, 3222] 4272 [4060, 4440] 1045 [833, 1241] 9160 [8177, 10077] 10791 [9307, 12511] RoadRunner 27980 [27269, 28692] 33412 [32459, 34435] 723 [454, 940] 66453 [40696, 104163] 49579 [47425, 51426] Robotank 4 [4, 5] 19 [18, 20] 4 [2, 6] 51 [47, 55] 13 [12, 15] Seaquest 299 [277, 318] 1641 [1621, 1661] 250 [214, 282] 3416 [2665, 4426] 3522 [2401, 4850] Skiing -19568 [-19793, -19362] -24070 [-25305, -22667] -27901 [-30000, -23704] -30043 [-30394, -29764] -30000 [-30000, -30000] Solaris 1645 [1480, 1804] 1289 [1143, 1451] 0 [0, 2] 2340 [1882, 2799] 1103 [799, 1430] SpaceInvaders 665 [651, 675] 743 [721, 764] 294 [235, 354] 1433 [1039, 1943] 701 [626, 768] StarGunner 692 [662, 719] 1488 [1470, 1506] 415 [316, 499] 2090 [1678, 2649] 3488 [1032, 8241] Surround 3488 [1032, 8241] 3488 [1032, 8241] -9 [-10, -10] 5 [4, 7] -2 [-4, -2] Tennis -21 [-24, -19] -1 [-2, 0] -20 [-22, -19] -3 [-11, 0] 0 [0, 0] TimePilot 1539 [1479, 1613] 2703 [2627, 2787] 548 [450, 690] 7179 [3128, 13016] 4382 [4208, 4528] Tutankham 112 [97, 123] 179 [165, 191] 29 [17, 43] 253 [240, 269] 164 [145, 185] UpNDown 7669 [7116, 8147] 12397 [11489, 13312] 595 [428, 737] 284807 [17861, 391388] 73095 [40836, 108810] Venture 25 [6, 45] 19 [14, 25] 2 [0, 6] 0 [0, 0] 112 [0, 304] VideoPinball 5129 [4611, 5649] 26245 [23075, 29067] 1005 [0, 2485] 22345 [20669, 23955] 53826 [40600, 67972] WizardOfWor 481 [396, 542] 2213 [1827, 2617] 225 [185, 264] 7086 [6518, 7330] 2599 [2259, 2942] YarsRevenge 9426 [9177, 9656] 10708 [10405, 11071] 1891 [925, 2964] 62209 [57783, 67113] 34861 [29734, 40020] Zaxxon 112 [15, 230] 3661 [3131, 4192] 0 [0, 0] 17347 [15320, 19385] 8850 [8045, 9740] Mean 0.25 [0.24, 0.26] 1.08 [1.02, 1.14] -0.09 [-0.10, -0.07] 3.74 [3.29, 4.13] 2.54 [2.34, 2.75] Median 0.12 [0.10, 0.12] 0.40 [0.40, 0.47] 0.01 [0.00, 0.01] 1.25 [1.11, 1.47] 0.96 [0.78, 0.98] IQM 0.17 [0.16, 0.17] 0.61 [0.60, 0.62] 0.02 [0.01, 0.02] 1.46 [1.34, 1.51] 0.90 [0.88, 0.94]