Trajectory First: A Curriculum for Discovering Diverse Policies

Cornelius V. Braun, Sayantan Auddy, Marc Toussaint · 2025

Read Trajectory First: A Curriculum for Discovering Diverse Policies by Cornelius V. Braun, Sayantan Auddy, Marc Toussaint (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2506-01568.txt.

Introduction

Cornelius Braun, Sayantan Auddy, and Marc Toussaint teach learning agents to solve one task in many unlike ways, in two thousand twenty-five. Diverse skills make agents robust when a task changes slightly, and less likely to stall in poor solutions. Existing diversity methods, though, explore too little in hard tasks such as robot handling.

Their two-stage plan starts with curve-based paths that bias the agent toward varied, high-reward behavior. Those behaviors are then distilled into step-by-step reactive policies that a robot can run live. Tests gave new insight into why diversity training struggles. The plan raised the diversity of learned skills while keeping task scores high.

Catalog record

CREATORCornelius V. Braun, Sayantan Auddy, Marc Toussaint
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE38 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2506-01568

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Trajectory First: A Curriculum for Discovering Diverse Policies Cornelius V. Braun, Sayantan Auddy, Marc Toussaint Abstract Being able to solve a task in diverse ways makes agents more robust to task variations and less prone to local optima. In this context, constrained diversity optimization has emerged as a powerful reinforcement learning (RL) framework to train a diverse set of agents in parallel. However, existing constrained-diversity RL methods often under-explore in complex tasks such as robotic manipulation, leading to a lack in policy diversity. To improve diversity optimization in RL, we therefore propose a curriculum that first explores at the trajectory level before learning step-based policies. In our empirical evaluation, we provide novel insights into the shortcoming of skill-based diversity optimization, and demonstrate empirically that our curriculum improves the diversity of the learned skills. 1 Introduction Reinforcement Learning (RL) has driven breakthroughs in robot locomotion [ 1 ] , game-playing [ 2 , 3 ] , and foundation-model finetuning [ 4 ] . While effective, most RL methods assume a unimodal action distribution and produce only a single policy. In contrast, humans and animals routinely solve the same task using multiple qualitatively different strategies. Such variability is also desirable in RL, as it has been shown that human strategy diversity increases solution quality and robustness [ 5 , 6 ] . Therefore, this work considers the discovery of a policy set that maximizes the reward in diverse ways. A number of previous works have investigated this problem from various perspectives. Notably, the fields of Novelty Search (NS) and Quality-Diversity (QD) have proposed a multitude of algorithms which populate an archive of solutions based on their novelty and performance [ 7 , 8 , 9 , 10 ] . Further, gradient-based RL approaches define intrinsic diversity rewards that they combine with extrinsic task rewards using Lagrange multipliers [ 11 ] , bandits [ 12 ] , or linear combinations [ 13 , 14 , 15 ] . While effective, the above approaches are not without shortcomings. QD may produce exceptional results, but often at the cost of sample efficiency and manual feature design. Gradient-based diversity or entropy bonuses in RL may still collapse to a few modes and remain under-evaluated in challenging contact-rich tasks [ 16 ] . Since skill learning and diversity or entropy maximization are common exploration objectives (see Zheng et al. 2024 for an overview), this observation may appear counterintuitive at first. Yet, we corroborate these findings in our work, demonstrating that even simple obstacle avoidance tasks may be substantially challenging for diversity optimization in RL (Fig. 2 ). Further, we highlight the necessity for exploration strategies tailored for diversity optimization, since we find that out-of-the-box exploration methods do well in improving task performance, but do not guarantee policy diversity. To overcome these shortcomings, we propose a new and simple curriculum for learning diverse policies with off-policy and model-free RL ( Section 5 ). We motivate our approach by the observation that skill-based RL methods commonly explore by optimizing neural-network parameters which can be inefficient. In contrast, we introduce a trajectory-first curriculum that (i) uses an evolutionary search over open-loop action sequences to uncover a diverse set of high-reward behaviors, and (ii) distills these behaviors into distinct, off-policy, model-free policies. While prior work proposed similar formulations that first solve exploration and then learning [ 18 , 19 ] , we do not rely on human demonstrations and propose an evolutionary approach to maximize entropy by optimizing trajectory parameters for novelty instead of neural network parameters. Based on this algorithm, we empirically highlight shortcomings of existing diversity optimization methods in Section 5 and illustrate how the proposed curriculum enables learning diverse sets of robot manipulation policies. In short, we make three contributions with this work. First, we propose a novel curriculum for diversity optimization under extrinsic task rewards (see Fig. 1 ). Second, we introduce Constrained Novelty Search (CNS) to discover diverse trajectories in the first step of this curriculum ( Section 3.1 ). Finally, we investigate how diversity can be maintained during online training of control policies from this data ( Section 3.2 ). Figure 1: Overview of the proposed diversity curriculum. First, an evolution strategy explores in trajectory space to maximize novelty of trajectory parameters \omega\in\Omega under performance constraints. Then, this data is used to warmstart the online training of multiple RL agents \pi\in\Pi to solve the same optimization problem in policy space. 2 Preliminaries Markov Decision Process: We model each task as a discrete-time MDP M=({\cal S},{\cal A},p,r,\gamma) [ 20 , MDP] . At each time step t , the agent in state s_{t}\in{\cal S} selects action a_{t}\in{\cal A} , transitions to s_{t+1} with probability p(s_{t+1}\mid s_{t},a_{t}) , and receives reward r_{t}\triangleq r(s_{t},a_{t})\in[r_{\min},r_{\max}] . The objective is to learn a policy \pi_{\theta}:{\cal S}\times{\cal A}\to{\mathbb{R}}^{+} , parameterized by \theta\in\mathbb{R}^{d} , that maximizes the discounted return J(\pi)=\sum_{t=\ell}^{\infty}\EEE_{(s_{t},a_{t})\sim\pi}\left[\gamma^{t-\ell}r(s_{t},a_{t})\right] with discount factor \gamma\in[0,1) . We denote by \rho_{\pi}(s,a) the state–action occupancy measure and by \rho_{\pi}(s) its marginal over states following [ 21 ] . Constrained Diversity Optimization: While earlier works used scalars to balance diversity and task rewards, Zahavy et al. 2023 introduced the following constrained MDP formulation: \max_{\Pi^{n}}\penalty\ \text{Diversity}(\Pi^{n})\qquad\penalty\ \penalty\ \text{s.t.}\penalty\ \penalty\ J(\pi)\geq\alpha v^{*},\quad\forall\pi\in\Pi^{n} (1) where \Pi^{n} is the current set of policies, v^{*}\triangleq\max_{\pi\in\Pi}J(\pi) is the value of the optimal policy and \alpha\in[0,1) is a hyperparameter controlling the optimality constraint. This constrained optimization problem can then be solved efficiently using Lagrange multipliers that are tuned using dual ascent [ 22 , 23 ] . Similar to subsequent work [ 17 ] , we adopt the same problem formulation in this work. To quantify diversity, we will measure the distance to the nearest neighbor, which shall be maximized: \text{Diversity}(\Pi^{n})\coloneq\frac{1}{n}\sum_{i=1}^{n}\min_{\pi_{j}\neq\pi_{i}}\lVert\EEE_{s\sim\rho_{\pi_{i}}}[\phi(s)]-\EEE_{s\sim\rho_{\pi_{j}}}[\phi(s)]\rVert^{2}, (2) where \phi(\cdot):{\cal S}\to{\mathbb{R}}^{f} are state-based features, which generally can be manually defined or learned, for instance using the successor feature method [ 24 , 25 ] . To optimize Eq. 1 , the common framework is to employ one-hot skill encodings z(s_{t})\in[0,1]^{n} as conditioning for a single policy and Q -function when learning [ 26 , 11 ] . We follow this approach in our work and slightly abuse notation in using z(s_{t}) to indicate the skill of a state and z(\tau) for the skill of a full trajectory. Novelty Search: Most novelty-based approaches to skill learning maximize the entropy of the policy set by using the entropy of the current policy set as an intrinsic reward [ 10 , 8 , 27 ] . To quantify entropy, the particle-based entropy estimator by Singh et al. 2003 is commonly employed which estimates the sparsity of the distribution based on the distance between the datapoints \{x_{i}\}_{i=1}^{n} and their k -nearest neighbor: {\cal H}_{\text{particle}}(X)\propto\sum_{x_{i}\in X}\log\lVert x_{i}-x_{i}^{(k)}\rVert . In practice, we choose k=1 . In other words, we measure the particle-based entropy as distance to the nearest neighbor, and add a constant c=1 for numerical stability: {\cal H}_{\text{particle}}(X)\coloneq\sum_{x_{i}\in X}\log\left(c+\min_{x_{j}\neq x_{i}}\lVert x_{i}-x_{j}\rVert\right). (3) 3 Evolutionary Exploration for Diverse Policy Discovery Solving the problem in Eq. 1 requires an initialization that is sufficiently diverse to prevent the diversity optimization from only occurring locally. Since the policies are initialized randomly at the beginning, we find that the discovered policy set rarely covers the task space sufficiently and

Read the complete text