Improving Environment Novelty Quantification for Effective Unsupervised Environment Design
Jayden Teoh *, Wenjun Li *, Pradeep VarakanthamSingapore Management University · 2025
Read Improving Environment Novelty Quantification for Effective Unsupervised Environment Design by Jayden Teoh *, Wenjun Li *, Pradeep VarakanthamSingapore Management University (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2502-05726.txt.
Introduction
Jayden Teoh, Wenjun Li, and Pradeep Varakantham improve how training worlds are invented for learning agents. In this setup a teacher agent keeps designing new practice environments for a student, aiming at lessons with high learning value. Most teachers judge lessons by regret, the gap between the student's score and the best possible, which pushes difficulty up but ignores whether a world is genuinely new to the student.
Their framework, CENIE, measures novelty by how much of the student's past action coverage a new world adds, modelled with a mixture method. Adding novelty next to regret as a twin goal lifted existing methods to the best results across several test suites. The authors argue the pairing builds agents that cope better with unseen settings.
Catalog record
| CREATOR | Jayden Teoh *, Wenjun Li *, Pradeep VarakanthamSingapore Management University |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 84 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2502-05726
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Improving Environment Novelty Quantification for Effective Unsupervised Environment Design Jayden Teoh *, Wenjun Li *, Pradeep VarakanthamSingapore Management University Abstract Unsupervised Environment Design (UED) formalizes the problem of autocurricula through interactive training between a teacher agent and a student agent. The teacher generates new training environments with high learning potential, curating an adaptive curriculum that strengthens the student’s ability to handle unseen scenarios. Existing UED methods mainly rely on regret , a metric that measures the difference between the agent’s optimal and actual performance, to guide curriculum design. Regret-driven methods generate curricula that progressively increase environment complexity for the student but overlook environment novelty –a critical element for enhancing an agent’s generalizability. Measuring environment novelty is especially challenging due to the underspecified nature of environment parameters in UED, and existing approaches face significant limitations. To address this, this paper introduces the Coverage-based Evaluation of Novelty In Environment (CENIE) framework. CENIE proposes a scalable, domain-agnostic, and curriculum-aware approach to quantifying environment novelty by leveraging the student’s state-action space coverage from previous curriculum experiences. We then propose an implementation of CENIE that models this coverage and measures environment novelty using Gaussian Mixture Models. By integrating both regret and novelty as complementary objectives for curriculum design, CENIE facilitates effective exploration across the state-action space while progressively increasing curriculum complexity. Empirical evaluations demonstrate that augmenting existing regret-based UED algorithms with CENIE achieves state-of-the-art performance across multiple benchmarks, underscoring the effectiveness of novelty-driven autocurricula for robust generalization. * * footnotetext: Equal contribution. † † footnotetext: Corresponding author. 1 Introduction Although recent advancements in Deep Reinforcement Learning (DRL) have led to many successes, e.g., super-human performance in games [ 26 , 9 ] and reliable control in robotics [ 2 , 3 ] , training generally-capable agents remains a significant challenge. DRL agents often fail to generalize well to environments only slightly different from those encountered during training [ 21 , 63 ] . To address this problem, there has been a surge of interest in Unsupervised Environment Design (UED; [ 56 , 23 , 57 , 28 , 27 , 37 , 33 , 7 ] ), which formulates the autocurricula [ 32 ] generation problem as a two-player zero-sum game between a teacher and a student agent. In UED, the teacher constantly adapts training environments (e.g., mazes with varying obstacles and car-racing games with different track designs) in the curriculum to improve the student’s ability to generalize across all possible levels. To design effective autocurricula, researchers have proposed various metrics to capture learning potential, enabling teacher agents to create training levels that adapt to the student’s capabilities. The most popular metric, regret , measures the student’s maximum improvement possible in a level. While regret-based UED algorithms [ 23 , 27 , 28 ] are effective in producing levels at the frontier of the student’s capability, they do not guarantee diversity in the student’s experiences, limiting the training of generally-capable agents especially in large environment design spaces. Another line of work in UED recognizes this limitation, leading to methods exploring the prioritization of novel levels during curriculum generation [ 56 , 57 , 33 ] . This strategic shift empowers the teacher to introduce novel levels into the curriculum such that the student agent can actively explore the environment space and enhance its generalization capabilities. To more effectively evaluate environment novelty, we introduce the Coverage-based Evaluation of Novelty In Environment (CENIE) framework. CENIE operates on the intuition that a novel environment should induce unfamiliar experiences, pushing the student agent into unexplored regions of the state space and introducing variability in its actions. Therefore, signals about an environment’s novelty can be derived by modeling and comparing its state-action space coverage with those of environments already encountered in the curriculum. We refer to this method of estimating novelty based on the agent’s past experiences as curriculum-aware . By evaluating novelty in relation to the experiences induced by other environments within the curriculum, CENIE prevents redundant environments—those that elicit similar experiences as existing ones—from being classified as novel. Curriculum-aware approaches ensure that levels in the student’s curriculum collectively drive the agent toward novel experiences in a sample-efficient manner. Our contributions are threefold. First, we introduce CENIE, a scalable, domain-agnostic, and curriculum-aware framework for quantifying environment novelty via the agent’s state-action space coverage. CENIE addresses shortcomings in existing methods for environment novelty quantification, as discussed further in Sections 3 and 4 . Second, we present implementations for CENIE using Gaussian Mixture Models (GMM) and integrated its novelty objective with PLR ⟂ [ 28 ] and ACCEL [ 37 ] , the leading UED algorithms in zero-shot transfer performance. Finally, we conduct a comprehensive evaluation of the CENIE-augmented algorithms across three distinct benchmark domains. By incorporating CENIE into these leading UED algorithms, we empirically validate that CENIE’s novelty-based objective not only exposes the student agent to a broader range of scenarios in the state-action space, but also contributes to achieving state-of-the-art zero-shot generalization performance. This paper underscores the importance of novelty and the effectiveness of the CENIE framework in enhancing UED. 2 Background We briefly review the background of Unsupervised Environment Design (UED) in this section. UED problems are modeled as an Underspecified Partially Observable Markov Decision Process (UPOMDP) defined by the tuple: \displaystyle\langle S,A,O,\mathcal{I},\mathcal{T},\mathcal{R},\gamma,\Theta\rangle where S , A and O are the sets of states, actions, and observations, respectively. \Theta represents a set of free parameters where each \theta\in\Theta defines a specific instantiation of an environment (also known as a level ). We use the terms “environments” and “levels” interchangeably throughout this paper. The level-conditional observation and transition functions are defined as \mathcal{I}:S\times\Theta\rightarrow O and \mathcal{T}:S\times A\times\Theta\rightarrow\Delta(S) , respectively. The student agent, with policy \pi , receives a reward based on the reward function \mathcal{R}:S\times A\rightarrow\mathbb{R} with a discount factor \gamma\in[0,1] . The student seeks to maximize its expected value for each \theta denoted by V^{\theta}(\pi)=\mathbb{E}_{\pi}[\sum_{t=0}^{T}R(s_{t},a_{t})\gamma^{t} ]. The teacher’s goal is to select levels forming the curriculum by maximizing a utility function U(\pi,\theta) , which depends on \pi . Different UED approaches vary primarily in the teacher’s utility function. Domain Randomization (DR; [ 53 ] ) uniformly randomizes environment parameters, with a constant utility U^{\mathcal{U}}(\pi,\theta)=C , making it agnostic to the student’s policy. Minimax training [ 39 ] adversarially generates challenging levels, with utility U^{\mathcal{M}}(\pi,\theta)=-V^{\theta}(\pi) , to minimize the student’s return. However, this approach incentivizes the teacher to make the levels completely unsolvable, limiting room for learning. Recent UED methods address this by using a teacher that maximizes regret , defined as the difference between the return of the optimal policy and the current policy. Regret-based utility is defined as U^{\mathcal{R}}(\pi,\theta)=\textsc{Regret}^{\theta}(\pi,\pi^{*})=V^{\theta}(\pi^{*})-V^{\theta}(\pi) where \pi^{*} is the optimal policy on \theta . Regret-based objectives have been shown to promote the simplest levels that the student cannot solve optimally, and benefit from the theoretical guarantee of a minimax regret robust policy upon convergence in the two-player zero-sum game. However, since \pi^{*} is generally unknown, regret must be approximated. Dennis et al. [23] , the pioneer UED work, introduced a principled level generation based on the regret objective and proposed the PAIRED algorithm, where regret is estimated by the difference between the returns attained by an antagonist agent and the protagonist (student) agent.