Imagined Autocurricula

Ahmet H. Güzel*, Matthew T. Jackson, Jarek L. Liesen, Tim Rocktäschel, Jakob N. Foerster, Ilija Bogunovic, Jack Parker-Holder · 2025

Read Imagined Autocurricula by Ahmet H. Güzel*, Matthew T. Jackson, Jarek L. Liesen, Tim Rocktäschel, Jakob N. Foerster, Ilija Bogunovic, Jack Parker-Holder (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2509-13341.txt.

Introduction

Ahmet Guzel, Matthew Jackson and five co-authors train agents in imagined worlds in 2025 with IMAC autocurricula. Real data and true simulators are scarce for many embodied tasks. World models learned from passive data can dream diverse training worlds.

Useful dream data must be curated, not random. Unsupervised environment design induces a curriculum over dreamed worlds. Agents trained only inside a narrow world model transfer well to held-out variants. Hard procedural sets back the claim. Foundation world models may thus train broadly capable agents without fresh real collection per task.

Catalog record

CREATORAhmet H. Güzel*, Matthew T. Jackson, Jarek L. Liesen, Tim Rocktäschel, Jakob N. Foerster, Ilija Bogunovic, Jack Parker-Holder
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE58 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2509-13341

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Imagined Autocurricula Ahmet H. Güzel*, Matthew T. Jackson, Jarek L. Liesen, Tim Rocktäschel, Jakob N. Foerster, Ilija Bogunovic, Jack Parker-Holder Abstract Training agents to act in embodied environments typically requires vast training data or access to accurate simulation, neither of which exists for many cases in the real world. Instead, world models are emerging as an alternative–leveraging offline, passively collected data, they make it possible to generate diverse worlds for training agents in simulation. In this work, we harness world models to generate “imagined” environments to train robust agents capable of generalizing to novel task variations. One of the challenges in doing this is ensuring the agent trains on useful generated data. We thus propose a novel approach iMac (Imagined Autocurricula) leveraging Unsupervised Environment Design (UED), which induces an automatic curriculum over generated worlds. In a series of challenging, procedurally generated environments, we show it is possible to achieve strong transfer performance on held-out environments having trained only inside a world model learned from a narrower dataset. We believe this opens the path to utilizing larger-scale, foundation world models for generally capable agents. 1 Introduction Despite significant advancements in AI, we remain far from generally capable agents ( morris2023levels ) . In the 2010s, progress towards this goal was driven by breakthroughs in deep reinforcement learning (RL, ( Sutton1998 ) ) where agents such as AlphaGo ( silver2016mastering ) showed it was possible to discover new knowledge beyond human capabilities, like the famous move37 . However, scaling RL in simulation to produce a more general intelligence remains bottlenecked by the lack of sufficiently diverse and real-world simulators ( openendedlearningteam2021openended ; adaptiveagentteam2023humantimescale ) . By contrast, more recent years have been dominated by agents trained from vast quantities of Internet data. Here, agents have a broader knowledge of the real world but lack of ability to discover new, superhuman behaviors since they are largely trained to mimic the training data. World models are emerging as a promising approach to leverage the best of both. They “stand on the shoulders of giants” by training on vast Internet data ( parker2024genie ; bruce2024genie ; nvidia2025cosmosworldfoundationmodel ; yang2024learning ) , but then enable agents to train with RL inside their generated (or imagined ) environments. Thus, world models make it possible for agents to imagine potential outcomes without direct environment interaction, allowing them to train across diverse scenarios with substantially fewer real experiences ( alonso2024diffusionworldmodelingvisual ; hafner2024masteringdiversedomainsworld ; garrido2024learning ; wu2022daydreamerworldmodelsphysical ; micheli2023transformerssampleefficientworldmodels ; robine2023smaller ; hafner2020mastering ) . In theory, large foundation world models could then be used to generate sufficiently rich environments for agents to learn superhuman behaviors for a swathe of embodied settings. Progress in world models research has typically fallen into two categories. The first seeks to train world models from single (online or offline) environments, achieving strong and sample-efficient performance ( hafner2018planet ; Hafner2020Dream ; hafner2020mastering ) . The latter sees more general, so-called foundation world models trained from large-scale internet data ( parker2024genie ; bruce2024genie ; nvidia2025cosmosworldfoundationmodel ; yang2024learning ) , but these models have not been shown to be useful for training generalist agents. In this paper, we attempt to bridge this gap. We focus on the offline RL setting, where agents must learn from pre-collected datasets without additional environment interaction ( levine2020offline ) . We focus on procedurally generated environments, meaning our offline data contains sequences (action labelled videos) from a variety of different levels. We then seek to transfer the learned agents to new, unseen levels from the same environment. This presents a significant challenge for existing methods, which often fail to generalize to new tasks when training purely from offline data ( mediratta2024generalization ; lu2023challenges ) . We believe world models can be the solution to this problem–we first train a diffusion world model from the offline samples and then use it to generate new, imagined worlds. Crucially, naively generating data from world models can lead to ineffective training data, making agent learning inefficient. This inefficiency stems from both the quality of the worlds themselves and the challenge of setting appropriate hyperparameters like imagined episode length across diverse tasks. To address this challenge, we leverage the Unsupervised Environment Design paradigm (UED) ( dennis2020emergent ) )—where a teacher proposes levels that maximize student regret. Rather than training on fixed or random sets of imagined worlds, we create an Imagined Autocurricula, or iMac . Importantly, our framework is world-model agnostic—while we use a diffusion world model as our foundation, our autocurriculum approach can be applied to any world model architecture that provides reward and next-token predictions. Our contribution lies not in advancing world model architectures, but in demonstrating how UED principles can effectively guide agent training within learned world models from offline mixed dataset. Our approach iMac uses Prioritized Level Replay (PLR, ( jiang2021prioritized ) ) as a UED algorithm, which we show provides a natural complement to the learned world model–the world model generates diverse potential training trajectories or "imagined environments," while PLR strategically selects subsequent training tasks from these imagined rollouts. Figure 1 illustrates the overall architecture of our approach. As we show, this prioritization process naturally induces an automatic curriculum over the generated, imagined worlds, meaning the agent is exposed to increasingly challenging training tasks. As a testbed for this paradigm, we leverage seven Procgen environments with high-dimensional visual inputs, where both model-free ( prudencio2023offlineRLsurvey ) and model-based approaches ( he2023surveyofflinemodelbasedreinforcement ) struggle to transfer knowledge to unseen scenarios ( mediratta2024generalization ; lu2023challenges ) . The resulting agents have significantly stronger generalization performance than state-of-the-art baselines, demonstrating strong transfer performance on held-out environments having trained only inside a world model learned from a narrower dataset. We believe this opens the path to utilizing larger-scale, foundation world models for generally capable agents. Our primary contribution is the demonstration that world models offer the path to training on diverse offline datasets and subsequently generalizing to new, unseen tasks. Importantly, this is made possible by training on an Imagined Autocurriculum; rather than fixing hyperparameters or randomizing settings, each of which could be problematic for learning. We believe this is the first example of open-ended learning in learned world models, and could enable significant progress towards generalist agents with ever more powerful world models in the future. Figure 1: Architecture overview: (1) Offline Data Collection: Diverse state-action-reward-next state tuples are collected via policy \pi_{\beta} to form the dataset. (2) World Model Training: A state transition denoiser and reward/termination predictor are trained on the dataset and frozen for consistency. (3) Agent Training: The agent learns through an imagine-based autocurriculum using Prioritized Level Replay, which balances between sampling from existing experiences and generating new rollouts to update the buffer. 2 Preliminaries 2.1 (Offline) Reinforcement Learning To model environments, we consider Partially Observable Markov Decision Processes (POMDPs) ( Sutton1998 ) , which are defined by a tuple \langle S,A,T,R,\Omega,O,\gamma\rangle . S , A , and \Omega respectively represent the set of states, actions, and observations. T(s_{t+1}|a_{t},s_{t}) is the conditional transition distribution over the next state s_{t+1}\in S given previous action a_{t}\in A and state s_{t}\in S . Each transition produces an observation o_{t+1}\in\Omega sampled from O(o_{t+1}|s_{t+1},a_{t}) , and a reward r_{t+1}\in\mathbb{R} sampled from R(r_{t+1}|s_{t+1},a_{t},s_{t}) . An agent interacting with the

Read the complete text