Aviary: training language agents on challenging scientific tasks

Siddharth Narayanan 1 James D. Braza 1 Ryan-Rhys Griffiths 1 Manu Ponnapati 1 Albert Bou 1 Jon Laurent 1 Ori Kabeli 1 Geemi Wellawatte 1 Sam Cox 1 Samuel G. Rodriques 1,3 Andrew D. White 1,2 1 FutureHouse Inc., San Francisco, CA 2 University of Rochester, Rochester, NY 3 Francis Crick Institute, Lon · 2024

Read Aviary: training language agents on challenging scientific tasks by Siddharth Narayanan 1 James D. Braza 1 Ryan-Rhys Griffiths 1 Manu Ponnapati 1 Albert Bou 1 Jon Laurent 1 Ori Kabeli 1 Geemi Wellawatte 1 Sam Cox 1 Samuel G. Rodriques 1,3 Andrew D. White 1,2 1 FutureHouse Inc., San Francisco, CA 2 University of Rochester, Rochester, NY 3 Francis Crick Institute, Lon (2024) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2412-21154.txt.

Description

Abstract Solving complex real-world tasks requires cycles of actions and observations. This is particularly true in science, where tasks require many cycles of analysis, tool use, and experimentation. Language agents are promising for automating intellectual tasks in science because they can interact with tools via natural language or code.

Catalog record

CREATORSiddharth Narayanan 1 James D. Braza 1 Ryan-Rhys Griffiths 1 Manu Ponnapati 1 Albert Bou 1 Jon Laurent 1 Ori Kabeli 1 Geemi Wellawatte 1 Sam Cox 1 Samuel G. Rodriques 1,3 Andrew D. White 1,2 1 FutureHouse Inc., San Francisco, CA 2 University of Rochester, Rochester, NY 3 Francis Crick Institute, Lon
YEAR2024
COLLECTIONPAPERS
FORMATTXT
FILE SIZE54 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2412-21154

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Aviary: training language agents on challenging scientific tasks Siddharth Narayanan 1 James D. Braza 1 Ryan-Rhys Griffiths 1 Manu Ponnapati 1 Albert Bou 1 Jon Laurent 1 Ori Kabeli 1 Geemi Wellawatte 1 Sam Cox 1 Samuel G. Rodriques 1,3 Andrew D. White 1,2 1 FutureHouse Inc., San Francisco, CA 2 University of Rochester, Rochester, NY 3 Francis Crick Institute, London, UK Correspondence to: {sam,andrew}@futurehouse.org Abstract Solving complex real-world tasks requires cycles of actions and observations. This is particularly true in science, where tasks require many cycles of analysis, tool use, and experimentation. Language agents are promising for automating intellectual tasks in science because they can interact with tools via natural language or code. Yet their flexibility creates conceptual and practical challenges for software implementations, since agents may comprise non-standard components such as internal reasoning, planning, tool usage, as well as the inherent stochasticity of temperature-sampled language models. Here, we introduce Aviary, an extensible gymnasium for language agents. We formalize agents as policies solving language-grounded partially observable Markov decision processes, which we term language decision processes. We then implement five environments, including three challenging scientific environments: (1) manipulating DNA constructs for molecular cloning, (2) answering research questions by accessing scientific literature, and (3) engineering protein stability. These environments were selected for their focus on multi-step reasoning and their relevance to contemporary biology research. Finally, with online training and scaling inference-time compute, we show that language agents backed by open-source, non-frontier LLMs can match and exceed both frontier LLM agents and human experts on multiple tasks at up to 100x lower inference cost. 1 Introduction Figure 1: An overview of the five implemented Aviary environments and the language decision process (LDP) framework. The term language decision process here jointly refers to our theoretical description of the class of problems solved by language agents, as well as a software framework for implementing language agents based on a stochastic computation graph that enables training of language agent components such as LLM weights, prompts, memories, or LLM sampling parameters like temperature. Language agents [ 1 , 2 , 3 , 4 ] are AI agents [ 5 ] that integrate LLMs [ 6 , 7 , 8 ] as core components. LLMs excel at zero-shot generalization [ 9 , 10 ] , providing a notable advantage over traditional AI agents, such as those based on handcrafted rules or reinforcement learning, which often struggle to generalize to new environments [ 11 ] . While LLMs can exhibit flawed reasoning and logic when used in isolation [ 12 , 13 , 14 ] , constructing a language agent by grounding LLMs in an environment with observational feedback can mitigate these issues. Early work on language agents used LLMs to directly output actions in the external environment [ 15 , 16 , 17 ] , while more recently, language agents have been augmented with internal reasoning [ 18 , 19 ] and planning [ 20 , 21 ] procedures, as well as long-term memory storage [ 22 , 23 ] . An emergent research challenge is to pose a theoretical description of the learning problem solved by language agents [ 4 , 24 ] and to develop efficient methods to optimize the components of a language agent [ 24 , 25 , 26 ] . Here, we define common language agent tasks as language decision processes (LDPs) and frame language agents as stochastic computation graphs [ 27 ] that may be trained to solve LDPs. We show that pre-existing agents [ 18 , 19 , 21 ] can be implemented within our stochastic computation graph framework and introduce a simple and extensible software package named LDP that enables modular interchange of environments, agents, and optimizers, simplifying experimentation across a variety of settings. In the problems we consider, we use the term optimization of language agents in the reinforcement sense to encompass procedures that yield iterative improvement of the language agent over time through feedback from an environment. An example of one such optimization algorithm is expert iteration (EI) [ 28 , 29 , 30 ] which achieves learning through successive rounds of supervised fine-tuning on (self-) generated trajectories from a progressively stronger language agent. Other approaches include outcome supervision [ 31 ] , which is similar to rejection-sampled expert iteration, and process supervision [ 32 ] , which still selects high-reward trajectories, but also focuses on the individual steps of the outcome. In what follows, we introduce our definition of an environment, a language decision process, and optimization of agents backed by a stochastic computation graph. We recast popular benchmarks such as GSM8K [ 33 ] and hotpot QA [ 34 ] as environments and integrate three scientific environments related to challenging tasks in the natural sciences. The scientific environments are (1) DNA construct engineering, where the task is to answer questions pertaining to molecular cloning [ 35 ] , (2) scientific literature question answering, where the task is to answer a multiple choice question by finding a specific passage from the scientific literature [ 36 , 37 ] , and (3) protein design, where the goal is to propose mutations to improve the stability of a given protein sequence [ 38 , 39 ] . On the DNA construct design and scientific literature question answering environments, we demonstrate that language agents based on the small, open-source Llama-3.1-8B-Instruct model, when trained with expert iteration and using inference-time majority vote sampling, can exceed the performance of both human experts and frontier LLMs. The environment framework described in this work, Aviary, is available at github.com/Future-House/aviary and the stochastic computation graph framework together with language agent implementations and training code is available at github.com/Future-House/ldp . The components of and relationship between these frameworks is delineated in Figure 1 . 2 Related Work Language Agent Formalisms Although language agents have achieved impressive empirical performance across a range of applications [ 1 , 37 , 40 ] , there is still no universally agreed upon theoretical framework for defining a language agent. In terms of conceptual models, the cognitive architectures for language agents (CoALA) framework [ 4 ] , inspired by ideas from production systems and cognitive architectures, taxonomizes agents according to their information storage (working and long-term memories), decision-making procedures e.g. planning, and action space (divided into internal and external actions). Similarly, in [ 41 ] , the author describes language agents as consisting of memory, planning, and tool usage components. Theoretically, many works represent language agents as partially observable Markov decision processes (POMDPs) [ 42 , 43 , 44 , 45 , 46 , 47 , 48 ] yet differ in their treatment of the action space e.g. in [ 43 ] the authors partition the action space into internal and external actions in a similar fashion to CoALA where internal actions are a family of functions that operate on the agent’s memory and external actions elicit an interaction with the environment. By contrast, in [ 44 ] the authors do not make a distinction between internal and external actions. In [ 49 ] the authors introduce a general framework for studying the design and analysis of LLM-based algorithms based on a computational graph where they assume LLM nodes are

Read the complete text