MaestroMotif: Skill Design from Artificial Intelligence Feedback

Martin Klissarov, Mikael Henaff, Roberta Raileanu, Shagun Sodhani, Pascal Vincent, Amy Zhang, Pierre-Luc Bacon, Doina Precup, Marlos C. Machado, Pierluca D’Oro · 2024

Read MaestroMotif: Skill Design from Artificial Intelligence Feedback by Martin Klissarov, Mikael Henaff, Roberta Raileanu, Shagun Sodhani, Pascal Vincent, Amy Zhang, Pierre-Luc Bacon, Doina Precup, Marlos C. Machado, Pierluca D’Oro (2024) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2412-08542.txt.

Introduction

Martin Klissarov and nine co-authors present MaestroMotif for skill design from AI feedback in two thousand twenty-four. Skills stated in plain language could inject human know how into agents. MaestroMotif uses language models to create and reuse such skills. Feedback from a model first designs a reward per skill.

Code generation plus reinforcement learning then trains each skill. Skills combine to carry out complex acts stated in language. Tests run in the NetHack learning environment suite. The method beat past approaches on score and usability there. Language thus becomes a handle for building agent skills.

Catalog record

CREATORMartin Klissarov, Mikael Henaff, Roberta Raileanu, Shagun Sodhani, Pascal Vincent, Amy Zhang, Pierre-Luc Bacon, Doina Precup, Marlos C. Machado, Pierluca D’Oro
YEAR2024
COLLECTIONPAPERS
FORMATTXT
FILE SIZE77 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2412-08542

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

MaestroMotif: Skill Design from Artificial Intelligence Feedback Martin Klissarov, Mikael Henaff, Roberta Raileanu, Shagun Sodhani, Pascal Vincent, Amy Zhang, Pierre-Luc Bacon, Doina Precup, Marlos C. Machado, Pierluca D’Oro Abstract Describing skills in natural language has the potential to provide an accessible way to inject human knowledge about decision-making into an AI system. We present MaestroMotif, a method for AI-assisted skill design, which yields high-performing and adaptable agents. MaestroMotif leverages the capabilities of Large Language Models (LLMs) to effectively create and reuse skills. It first uses an LLM’s feedback to automatically design rewards corresponding to each skill, starting from their natural language description. Then, it employs an LLM’s code generation abilities, together with reinforcement learning, for training the skills and combining them to implement complex behaviors specified in language. We evaluate MaestroMotif using a suite of complex tasks in the NetHack Learning Environment (NLE), demonstrating that it surpasses existing approaches in both performance and usability. † † footnotetext: * Equal supervision. Correspondence to: [email protected] . 1 Introduction Bob wants to understand how to become a versatile AI researcher. He asks his friend Alice, a respected AI scientist, for advice. To become a versatile AI researcher, she says, one needs to practice the following skills: creating mathematical derivations, writing effective code, running and monitoring experiments, writing scientific papers, and giving talks. Alice believes that, once these different skills are mastered, they can be easily combined following the needs of any research project. Figure 1: Performance across NLE task categories. MaestroMotif largely outperforms existing methods zero-shot, including the ones trained on each task. Alice is framing her language description of how to be a versatile researcher as the description of a set of skills. This often happens among people, since this type of description is a convenient way to exchange information on how to become proficient in a given domain. Alice could instead have suggested what piece of code or equation to write, or, at an even lower level of abstraction, which keys to press; but she prefers not to do it, because it would be inconvenient, time-consuming, and likely tied to specific circumstances for it to be useful to Bob. Instead, describing important skills is easy but effective, transmitting large amounts of high-level information about a domain without dealing with its lowest-level intricacies. Understanding how to do the same with AI systems is still a largely unsolved problem. Recent work has shown that systems based on Large Language Models (LLMs) can combine sets of skills to achieve complex goals ( Ahn et al., 2022 ; Wang et al., 2024 ) . This leverages the versatility of LLMs to solve tasks zero-shot , after the problem has been lifted from the low-level control space, which they have difficulty handling, to a high-level skill space grounded in language, to which they are naturally suited. However, humans cannot communicate skills to these systems as naturally as Alice did with Bob. Instead, such systems typically require humans to solve, by themselves, the skill design problem , the one of crafting policies subsequently used by the LLM. Designing those skills typically entails very active involvement from a human, including collecting skill-specific data, developing heuristics, or manually handling reward engineering ( Ahn et al., 2022 ) . Thus, existing frameworks for designing low-level skills controlled by LLMs require technical knowledge and significant amounts of labor from specialized humans. This effectively reduces their applicability and generality. In this paper, we introduce the paradigm of AI-Assisted Skill Design . In this paradigm, skills are created in a process of human-AI collaboration, in which a human provides a natural language description of the skills and an AI assistant automatically converts those descriptions into usable low-level policies. This strategy fully leverages the advantages of both human-based and AI-based skill design workflows: it allows humans to inject important prior knowledge about a task, which may enhance safety and performance for the resulting agents even in the absence of optimal AI assistants; at the same time, it automates the lowest-level and more time-consuming aspects of skill design. Based on this paradigm, we propose MaestroMotif, a method that uses LLMs and reinforcement learning (RL) to build and combine skills for an agent to behave as specified in natural language. MaestroMotif uses an LLM’s feedback to convert high-level descriptions into skill-specific reward functions, via the recently-proposed Motif approach ( Klissarov et al., 2024 ) . It then crafts the skills by writing Python code using an LLM: first, it generates functions for the initiation and termination of each skill; then, it codes a policy over skills which is used to combine them. During RL training, the policy of each skill is optimized to maximize its corresponding reward function by interacting with the environment. At deployment time, MaestroMotif further leverages code generation via an LLM to create a policy over skills that can combine them almost instantaneously to produce behavior, in zero-shot fashion, as prescribed by a human in natural language. MaestroMotif thus takes advantage of RL from AI feedback to lift the problem of producing policies from low-level action spaces to high-level skill spaces, in a significantly more automated way than previous work. In the skill space, planning becomes much easier, to the point of being easily handled zero-shot by an LLM that generates code policies. These policies can use features of a programming language to express sophisticated behaviors that could be hard to learn using neural networks. In essence, MaestroMotif crafts and combines skills, similarly to motifs in a composition, to solve complex tasks. We evaluate MaestroMotif on a suite of tasks in the Nethack Learning Environment (NLE) ( Küttler et al., 2020 ) , created to test the ability to solve complex tasks in the early phase of the game. We show that MaestroMotif is a powerful and usable system: it can, without any further training, succeed in complex navigation, interaction and composite tasks, where even approaches trained for these tasks struggle. We demonstrate that these behaviors cannot be achieved by baselines that maximize the game score, and we perform an empirical investigation of different components our method. 2 Background A language-conditioned Markov Decision Process (MDP) ( Liu et al., 2022 ) is a tuple \mathcal{M}=(\mathcal{S},\mathcal{A},\mathcal{G},r,p,\gamma,\mu ), where \mathcal{S} is the state space, \mathcal{A} is the action space, \mathcal{G} is the space of natural language task specifications, r:\mathcal{S}\times\mathcal{G}\to\mathbb{R} is the reward function, p:\mathcal{S}\times\mathcal{A}\to\Delta(\mathcal{S}) is the transition function, \gamma\in(0,1] is the discount factor, \mu\in\Delta(\mathcal{S}) is the initial state distribution. A skill can be formalized through the concept of option ( Sutton et al., 1999 ; Precup, 2000 ) . A deterministic Markovian option \omega\in\Omega is a triple (\mathcal{I}_{\omega},\pi_{\omega},\beta_{\omega}) , where \mathcal{I}_{\omega}:\mathcal{S}\to\{0,1\} is the initiation function, determining whether the option can be initiated or not, \pi_{\omega}:\mathcal{S}\to\Delta(\mathcal{A}) is the intra-option policy, and \beta_{\omega}:\mathcal{S}\to\{0,1\} is the termination function, determining whether the option should terminate or not. Under this mathematical framework, the skill design problem is equivalent to constructing a set of options \Omega that can be used by an agent. The goal of the agent is to provide a policy over options \pi:\mathcal{G}\times\mathcal{S}\to\Omega . Whenever the termination condition of an option is reached, \pi selects the next option to be executed, conditioned on the current state. The performance

Read the complete text