Discrete Codebook World Models for Continuous Control
Aidan ScannellUniversity of Edinburgh [email protected], Mohammadreza NakhaeiAalto University, Kalle KujanpääAalto UniversityYi ZhaoAalto UniversityKevin Sebastian LuckVrije Universiteit AmsterdamArno SolinAalto UniversityJoni PajarinenAalto University · 2025
Read Discrete Codebook World Models for Continuous Control by Aidan ScannellUniversity of Edinburgh [email protected], Mohammadreza NakhaeiAalto University, Kalle KujanpääAalto UniversityYi ZhaoAalto UniversityKevin Sebastian LuckVrije Universiteit AmsterdamArno SolinAalto UniversityJoni PajarinenAalto University (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2503-00653.txt.
Introduction
Scannell and co-authors show that world models with discrete latent states beat continuous ones for state-based continuous control. World models let a reinforcement learning agent rehearse inside its own simulation of the environment. Leading systems split on representation: some compress states into continuous vectors, others into discrete codes, and the discrete side had mostly been proven on visual tasks.
Their model, DCWM, stores states as entries from a learned codebook, beating one-hot and label encodings of the same idea. Paired with decision-time planning, the full agent matched or beat recent leaders such as TD-MPC2 and DreamerV3 on continuous control benchmarks. The authors argue the discrete bottleneck is a strength, not a limit, for this class of problems. The catalog creator field runs names and emails together; the byline names seven authors.
Catalog record
| CREATOR | Aidan ScannellUniversity of Edinburgh [email protected], Mohammadreza NakhaeiAalto University, Kalle KujanpääAalto UniversityYi ZhaoAalto UniversityKevin Sebastian LuckVrije Universiteit AmsterdamArno SolinAalto UniversityJoni PajarinenAalto University |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 81 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2503-00653
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Discrete Codebook World Models for Continuous Control Aidan ScannellUniversity of Edinburgh [email protected], Mohammadreza NakhaeiAalto University, Kalle KujanpääAalto UniversityYi ZhaoAalto UniversityKevin Sebastian LuckVrije Universiteit AmsterdamArno SolinAalto UniversityJoni PajarinenAalto University Abstract In reinforcement learning (RL), world models serve as internal simulators, enabling agents to predict environment dynamics and future outcomes in order to make informed decisions. While previous approaches leveraging discrete latent spaces, such as DreamerV3, have demonstrated strong performance in discrete action settings and visual control tasks, their comparative performance in state-based continuous control remains underexplored. In contrast, methods with continuous latent spaces, such as TD-MPC2, have shown notable success in state-based continuous control benchmarks. In this paper, we demonstrate that modeling discrete latent states has benefits over continuous latent states and that discrete codebook encodings are more effective representations for continuous control, compared to alternative encodings, such as one-hot and label-based encodings. Based on these insights, we introduce DCWM: D iscrete C odebook W orld M odel, a self-supervised world model with a discrete and stochastic latent space, where latent states are codes from a codebook. We combine DCWM with decision-time planning to get our model-based RL algorithm, named DC-MPC : D iscrete C odebook M odel P redictive C ontrol, which performs competitively against recent state-of-the-art algorithms, including TD-MPC2 and DreamerV3, on continuous control benchmarks. See our project website www.aidanscannell.com/dcmpc . 1 Introduction In model-based reinforcement learning (RL), world models ( Ha & Schmidhuber 2018 ) have been introduced in order to simulate or predict the environment’s dynamics in a data-driven way. An agent equipped with a world model can make predictions about its environment by “simulating” possible actions within the model and ”imagining” the outcomes. This equips the agent with the ability to plan and anticipate outcomes given a (learned) reward function, and the additional ability to envision transitions and outcomes before taking them in the real world can in turn improve sample efficiency. One of the state-of-the-art world models, DreamerV2/V3 ( Hafner et al. 2022 ; Hafner et al. 2023 ) achieves strong performance in a wide variety of tasks, by “imagining” sequences of future states within a world model and using them to improve their policies. Interestingly, DreamerV2/V3 introduced a discrete latent space, in the form of a one-hot encoding, which offered significant benefits over its predecessor, DreamerV1 ( Hafner et al. 2019a ) . This suggests that discrete latent spaces may have benefits over continuous latent spaces. It could be from the discrete latent space helping avoid compounding errors over multi-step time horizons or enabling policy and value learning to harness the benefits of discrete variable processing for efficiency and interoperability. In the context of generative modeling, discrete codebooks have been at the heart of many successful approaches ( Chang et al. 2023 ; Esser et al. 2021 ; Ramesh et al. 2021 ) . However, in the context of continuous control, TD-MPC2 ( Hansen et al. 2023 ) uses a continuous latent space and significantly outperforms DreamerV3. Whilst there are multiple differences between TD-MPC2 and DreamerV2/V3, in this paper, we are specifically interested in exploring if discrete latent spaces can offer benefits for continuous control. Recently, Farebrother et al. 2024 showed that training value functions with classification may have benefits over training with regression. The benefits may arise because (i) classification considers uncertainty during training (via the cross-entropy loss), (ii) the categorical distribution is multi-modal so it can consider multiple modes during training, or (iii) learning in discrete spaces is more efficient. In the context of world models, it is natural to ask, what benefits are obtained by (i) using discrete vs continuous latent spaces and (ii) modeling deterministic vs stochastic transition dynamics. Further to this, when considering stochastic latent transition dynamics, what is the effect of modeling with (i) unimodal distributions ( e.g. Gaussian in continuous latent spaces) vs (ii) multimodal distributions ( e.g. categorical in discrete latent spaces). In this paper, we explore these ideas in the context of world models for model-based RL, i.e. does learning a discrete latent space using classification have benefits over learning a continuous latent space using regression. Contributions The main contributions are as follows: (C1) In the context of continuous control, we show that learning discrete latent spaces with classification does have benefits over learning continuous latent spaces with regression. (C2) We show that formulating a discrete latent state using codebook encodings has benefits over alternatives, such as one-hot (like DreamerV2/V3) and label encodings. (C3) Based on our insights, we introduce Discrete Codebook World Model ( DCWM ): a world model with a discrete latent space where each latent state is a discrete code from a codebook. It obtains strong performance in the difficult locomotion tasks from DeepMind Control suite ( Tassa et al. 2018 ) and manipulation tasks from Meta-World ( Yu et al. 2019 ) . 2 Related Work In this section, we recap world models in the context of model-based RL. We introduce two competing methods for learning latent spaces (i) those using observation reconstruction and (ii) those using latent state temporal consistency objectives. We then compare methods that learn continuous latent spaces using regression and those that learn discrete latent spaces using classification. World models Model-based RL is often said to be more sample-efficient than model-free methods. This is because it learns a model in which it can reason about the world, instead of simply trying to learn a policy or a value function to maximize the return ( Ha & Schmidhuber 2018 ) . The world model can be used for planning ( Allen & Koomen 1983 ; Basye et al. 1992 ) . A prominent idea has been to optimize the evidence lower bound of observation and reward sequences to learn world models that operate on the latent space of a learned Variational Autoencoder (VAE, Kingma & Welling 2014 ; Igl et al. 2018 ). These models rely on maximizing the conditional observation likelihood p({\bm{o}}_{t}|{\bm{z}}_{t} ), i.e. the reconstruction objective. The latent space of the model can then be used for both policy learning in the imagination of the world model, known as offline planning, e.g. Dreamer ( Hafner et al. 2019a ) , or for decision-time planning ( Rubinstein 1997 ; Hafner et al. 2019b ; Schrittwieser et al. 2020 ) . Latent-state consistency Using the reconstruction loss for learning latent state representations is unreliable ( Lutter et al. 2021 ) and can have a detrimental effect on the performance of model-based methods in various benchmarks ( Kostrikov et al. 2020 ; Yarats et al. 2021a ) . To this end, TD-MPC ( Hansen et al. 2022 ) and its successor, TD-MPC2 ( Hansen et al. 2023 ) , use a consistency loss to learn representations for planning with Model Predictive Path Integral (MPPI) control together with reward and value functions learned through temporal difference methods ( Williams et al. 2015 ) . Note that many prior works learn latent state representations using variants of a self-supervised latent-state consistency objective ( Schwarzer et al. 2020 ; Wang et al. 2022 ; Ghugare et al. 2022 ; LeCun ; Georgiev et al. 2024 ; Scannell et al. 2024b ; Zhao et al. 2023 ; Scannell et al. 2024a ) .