Guiding Skill Discovery with Foundation Models

Zhao Yang [email protected], Thomas M. Moerland, Mike Preuss, Aske Plaat, Vincent François-Lavet, Edward S. Hu · 2025

Read Guiding Skill Discovery with Foundation Models by Zhao Yang [email protected], Thomas M. Moerland, Mike Preuss, Aske Plaat, Vincent François-Lavet, Edward S. Hu (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2510-23167.txt.

Introduction

Zhao Yang and five co-authors guide skill discovery in reinforcement learning with foundation models. Learning varied skills without hand-made rewards could speed later tasks, but pure diversity seeking can produce useless or dangerous behavior. A cheetah robot, for instance, may learn to roll in every direction instead of running safely.

Their FoG method uses a foundation model to steer discovery toward skills humans would prefer. The robot still explores broadly, yet avoids flipping or entering hazardous areas. Guided discovery promises reusable skill sets that are both diverse and aligned with what people actually want.

Catalog record

CREATORZhao Yang [email protected], Thomas M. Moerland, Mike Preuss, Aske Plaat, Vincent François-Lavet, Edward S. Hu
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE52 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2510-23167

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Guiding Skill Discovery with Foundation Models Zhao Yang [email protected], Thomas M. Moerland, Mike Preuss, Aske Plaat, Vincent François-Lavet, Edward S. Hu Abstract Learning diverse skills without manually designed reward functions can greatly reduce human effort, making the process more autonomous and realistic. However, existing skill discovery methods focus solely on maximizing the diversity of skills without considering human preferences, which leads to undesirable behaviors and possibly dangerous skills. For instance, a cheetah robot trained using previous methods learns to roll in all directions to maximize skill diversity, whereas we would prefer it to run without flipping or entering hazardous areas. In this work, we propose a Fo undation model G uided (FoG) skill discovery method, which incorporates human intentions into skill discovery through foundation models. Specifically, FoG extracts a score function from foundation models to evaluate states based on human intentions, assigning higher values to desirable states and lower to undesirable ones. These scores are then used to re-weight the rewards of skill discovery algorithms. By optimizing the re-weighted skill discovery rewards, FoG successfully learns to eliminate undesirable behaviors, such as flipping or rolling, and to avoid hazardous areas in both state-based and pixel-based tasks. Interestingly, we show that FoG can discover skills involving behaviors that are difficult to define. Interactive visualisations are available from https://sites.google.com/view/submission-fog . 1 Introduction Reinforcement learning (RL) has shown promising results in robotics ( Tang et al. 2024 ; Wu et al. 2023 ) and games ( Vasco et al. 2024 ; Zhang et al. 2024 ) . Typically, RL requires carefully designed reward functions, which demand significant expert efforts ( Schenck & Fox 2018 ; Sowerby et al. 2022 ) . In contrast, Unsupervised RL ( Laskin et al. 2021 ; Rajeswar et al. 2023 ) aims to eliminate task-specific reward functions and train agents in a self-supervised manner. One key direction in unsupervised RL is pre-training agents to acquire diverse skills that can potentially be useful in downstream tasks ( Eysenbach et al. 2018 ; Park et al. 2023b ) , termed unsupervised skill discovery. Most prior methods in unsupervised skill discovery focus on maximizing skill diversity, encouraging agents to achieve diversity in both low-level behaviors and high-level policies. For instance, a cheetah robot trained using previous methods ( Park et al. 2022 ; Park et al. 2023b ) learns to flip or roll (low-level behavior) in all directions (high-level policy). However, wide motions like flipping or rolling could damage the robot, and entering restricted areas might pose safety risks. Ideally, we want agents to learn skills that are not only diverse, but also aligned with specific intentions, such as eliminating undesirable behaviors or avoiding certain areas. Figure 1: FoG leverages foundation models (such as ChatGPT, Claude and CLIP) to score states in relation to given commands during training. These scores are used to re-weight the rewards of the underlying skill discovery algorithm. Left : In state-based tasks (top row), task descriptions are provided to foundation models, which are queried to generate a score function f(s) based on our requirements. In pixel-based tasks (bottom row), the current visual state, textual descriptions of desirable and undesirable intentions are input to foundation models to obtain embeddings. These embeddings are then used to form the score function f(s) , see Equation 8 . Right : During training, rewards of the underlying skill discovery method ( r_{skill} ) are re-weighted using the score function. Re-weighting r_{skill} (we use METRA ( Park et al. 2023b ) ) by the score function is equivalent with using the score function as the distance metric in the DSD objective. To integrate human intentions into skill discovery, we introduce a Fo undation model G uided (FoG) method. More specifically, FoG (see Figure 1 ) utilizes foundation models ( Radford et al. 2021 ; Ouyang et al. 2022 ; Bordes et al. 2024 ) to assign higher scores for desirable behaviors and lower for undesirable ones. These scores are then used to re-weight the rewards of unsupervised skill discovery algorithms. By optimizing these re-weighted rewards, FoG learns diverse skills while aligning with given human intentions. FoG stands out from previous methods by being more autonomous, as it does not rely on costly expert demonstrations like Kim et al. 2024b , and more versatile, as it works with both visual inputs and compact state information, unlike Rho et al. 2024 , which requires precise ground-truth states. Our main contributions are fourfold, and the FoG codebase can be found in the supplemental materials: 1. We propose FoG, a novel foundation model guided unsupervised skill discovery method that learns diverse and desirable skills. 2. We evaluate FoG alongside six state-of-the-art baselines on both state-based (i.e. structured, low-dimensional representations) and pixel-based tasks. FoG outperforms baselines in both scenarios, showcasing superior input-agnostic generalization capabilities. 3. We show FoG can learn behaviors that are challenging to define, such as being ‘twisted’ and ‘stretched’ on a humanoid robot, suggesting its potential for more complex applications. 4. We perform an extensive ablation study to assess the contribution of each component to FoG’s performance. 2 Preliminaries and Problem Setting We consider a reward-free Markov Decision Process defined as \mathcal{M}=(\mathcal{A},\mathcal{S},p) . \mathcal{S} denotes the state space, \mathcal{A} denotes the action space and p is the transition dynamics function. A latent vector z\in\mathcal{Z} (also called ‘skill’) is sampled during training and its conditioned policy \pi(\cdot|s,z) is executed to get a skill trajectory \tau=(s_{0},s_{1},...,s_{T}) following the process: p(\tau|z)=p(s_{0})\prod_{t=0}^{T-1}\pi(a_{t}|s_{t},z)p(s_{t+1}|s_{t},a_{t}) . \pi(\cdot|s,z) can be learned by optimizing unsupervised exploration objectives we discuss below (distance-maximization) or in Section 5 (mutual information). FoG utilizes the Distance-maximizing Skill Discovery (DSD) ( Park et al. 2023a ) objective. Unlike mutual information based methods ( Eysenbach et al. 2018 ) , DSD aims to maximize the Wasserstein dependency measure (WDM) ( Ozair et al. 2019 ) defined as: I_{\mathcal{W}}(S;Z)=\mathcal{W}(p(s,z),p(s)p(z)), (1) where \mathcal{W} is the 1-Wasserstein distance on the metric space ( S\times Z,d ) for distance metric d . By maximizing the objective in Equation 1 , the agent will not only maximize the diversity of skills, but also maximize the distance metric d ( Park et al. 2023b ) . Under some simplifying assumptions ( Ozair et al. 2019 ; Villani et al. 2009 ) , maximization of Equation 1 can then be rewritten as: \sup_{\pi,\phi}\mathbb{E}_{p{(\tau,z)}}\left[\sum_{t=0}^{T-1}\left(\phi(s^{\prime})-\phi(s)\right)^{\top}z\right]\;\;\text{ s.t.}\;\;\|\phi(x)-\phi(y)\|_{2}\leq d(x,y),\;\;\forall(x,y)\in S, (2) where \phi is a function that maps states to a D -dimensional space, which is the same as the skill space Z . Intuitively, Equation 2 aims to align the direction of z and \phi(s^{\prime})-\phi(s) (to learn distinguishable and diverse skills), while maximizing the length of ||\phi(s^{\prime})-\phi(s)|| , which leads to an increase in the distance between states based on the given distance metric d due to the Lipschitz constraint ( Park et al. 2023a ) . In principle, d(x,y) in Equation 2 can be replaced by any of the distance metrics in Table 1 , resulting in different unsupervised skill discovery methods. Equation 2 can be optimized with dual gradient descent, incorporating a Lagrange multiplier \lambda and a small slack variable \epsilon>0 : Update \phi to maximize: \displaystyle\mathbb{E}[(\phi(s^{\prime})-\phi(s))^{\top}z]+\lambda\cdot\min(\epsilon,d(s,s^{\prime})-\|\phi(s)-\phi(s^{\prime})\|) (3) Update \lambda to minimize: \displaystyle\lambda\cdot\mathbb{E}[\min(\epsilon,d(s,s^{\prime})-\|\phi(s)-\phi(s^{\prime})\|)] (4) Update \pi with reward: \displaystyle(\phi(s^{\prime})-\phi(s))^{\top}z (5) For derivation of these equations we refer to Park

Read the complete text