Exploring exploration with foundation agents in interactive environments
Daniel P. Sawyer, Nan Rosemary Ke, Hubert Soyer, Martin Engelcke, David P Reichert, Drew A. Hudson, John Reid, Alexander Lerchner, Danilo Jimenez Rezende, Timothy P Lillicrap, Michael Mozer, Jane X Wang · 2024
Read Exploring exploration with foundation agents in interactive environments by Daniel P. Sawyer, Nan Rosemary Ke, Hubert Soyer, Martin Engelcke, David P Reichert, Drew A. Hudson, John Reid, Alexander Lerchner, Danilo Jimenez Rezende, Timothy P Lillicrap, Michael Mozer, Jane X Wang (2024) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2412-06438.txt.
Introduction
Sawyer and co-authors test in 2024 whether large foundation models can actively gather information in interactive settings to test ideas. Such models do well on single turn reasoning but often fail to learn over many turns. In a simple feature world task, models gathered information near optimally, which set a hopeful baseline.
A harder text version of an alchemy benchmark, where agents must infer a hidden causal structure across many trials, told a different story. Models at first did not improve over time. When prompted to summarize what they had seen at regular gaps, they began to learn and could relearn when rules changed. One model family did best, while others struggled.
Catalog record
| CREATOR | Daniel P. Sawyer, Nan Rosemary Ke, Hubert Soyer, Martin Engelcke, David P Reichert, Drew A. Hudson, John Reid, Alexander Lerchner, Danilo Jimenez Rezende, Timothy P Lillicrap, Michael Mozer, Jane X Wang |
|---|---|
| YEAR | 2024 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 83 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2412-06438
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Exploring exploration with foundation agents in interactive environments Daniel P. Sawyer, Nan Rosemary Ke, Hubert Soyer, Martin Engelcke, David P Reichert, Drew A. Hudson, John Reid, Alexander Lerchner, Danilo Jimenez Rezende, Timothy P Lillicrap, Michael Mozer, Jane X Wang Abstract Foundation models excel at single-turn reasoning, but many real-world challenges, from scientific research to technology development, require multi-turn exploration in dynamic interactive environments. Crucial components of learning from experience in these settings, such as efficiently gathering information to test hypotheses, meta-learning a model of the world’s dynamics, and adapting to unexpected changes, remain largely unexplored for these models. We first evaluate foundation models in Feature World, a setting that primarily tests information gathering about a static hidden reward function. In this initial setting, we show that state-of-the-art foundation models come close to optimal efficiency in selecting maximally informative actions in tasks with simple reward functions. As a proof of concept, we also show a model can gather information efficiently in a 3D embodied version of this task, though errors in vision limit some aspects of performance. In order to test exploration across multiple dependent turns and trials, we implement a custom, text-based version of the Alchemy environment, a benchmark designed for meta-learning. Here, agents must deduce a latent causal structure by integrating information across multiple state-dependent trials. In this more complex setting, we find that recent foundation models struggle to meta-learn strategies that enable improved performance over time. However, prompting the models to summarize their observations at regular intervals enables an emergent meta-learning process, allowing them to improve across trials. Notably, in some models, summarization also enabled adaptive re-learning of this information when the environment’s rules change unexpectedly. While most models performed reasonably well on simple Feature World tasks, evaluations in Alchemy reveal stark differences in robustness among the models, with Gemini 2.5 performing best, followed by Claude 3.7, and ChatGPT-4o and o4-mini struggling the most. These results underscore Alchemy’s value as a benchmark for meta-learning and strategy adaptation in foundation models. By moving beyond simple discovery to complex, stateful environments, we demonstrate that the most significant challenge for foundation agents is not selecting informative actions in the moment, but rather seeking and integrating knowledge through adaptive strategies over time. Intriguingly, we find there is likely no intrinsic barrier to future generations of foundation agents more fully mastering these abilities. † † footnotetext: 1 Google DeepMind 1 Introduction Foundation models have demonstrated remarkable abilities in understanding and generating complex human-like text and multi-modal content ( Achiam et al., 2023 ; Gemini Team et al., 2023 ; Jiang et al., 2024 ; Reid et al., 2024 ; Dubey et al., 2024 ; Dai et al., 2024 ; Deitke et al., 2024 ) . However, this success has largely been measured in static, single-turn settings where information is provided upfront. The next frontier for these models lies in their application as interactive agents, which must operate in dynamic environments where crucial information is not given, but must be actively discovered. To achieve goals in such settings, an agent cannot merely react; it must proactively explore. This contrasts with classic reinforcement learning (RL) paradigms that use undirected exploration ( Burda et al., 2018 ; Ecoffet et al., 2019 ; Badia et al., 2020 ) . Real-world endeavors often demand a more sophisticated, hypothesis-driven approach. This involves strategically formulating beliefs about the world, designing experiments to test those beliefs, and integrating findings gathered across multiple, often state-dependent, trials. Such capabilities will become increasingly important as training on human-generated data reaches a limit and we enter the "era of experience", in which models generate their own training data through interaction with their environment ( Silver & Sutton, 2025 ) . Whether, and to what extent, today’s foundation models possess this latent capacity for active exploration remains a critical and largely open question. We evaluate LLMs in three environments: text-based and multimodal variants of Feature World, and a text-based version of Alchemy ( Wang et al., 2021 ) . The Feature World is largely stateless and does not necessitate extensive sequential decision-making, allowing us to isolate and analyze efficiency of information gathering. Alchemy, in contrast, demands strategic exploration and reasoning over multiple trials, which allows us to evaluate the foundation models’ meta-learning and strategy adaptation abilities. In this paper, we operationally define and measure three key capabilities involved in exploration: efficient information gathering, meta-learning, and strategy adaptation. • Efficient information gathering: Selecting actions that maximally increase expected information gain. In Feature World, we measure this as success rate in finding a rewarding object within a fixed step budget, compared to a random policy. • Meta-learning (learning to learn) : Improving expected performance on new tasks in a given family through experience of other tasks in that family ( Thrun & Pratt, 1998 ) . In Alchemy, we measure this as a significant improvement in performance over successive trials within an episode. • Strategy adaptation : Detecting when a strategy becomes invalid due to environmental changes and adapting by learning a new one. In Alchemy, we measure this as performance recovery after an uncued change to environment dynamics. More specifically, this paper investigates the capacity of foundation models to conduct exploratory behavior within interactive environments in the zero-shot setting, using in-context prompting alone and without requiring task-specific training or fine-tuning. We performed experiments using Gemini 1.5 Pro and Flash ( Reid et al., 2024 ) , Gemini 2.5 Pro and Flash ( Google, 2025 ) , Claude 3.7 Sonnet ( Anthropic, 2025 ) , and ChatGPT-4o ( OpenAI, 2024 ) and o4-mini ( OpenAI, 2025 ) . Overall, this work makes the following key contributions and findings: • We conduct extensive experiments evaluating multi-turn exploration performance of foundation models across a diverse set of interactive environments. We analyzed several foundation models and a range of in-context prompting strategies, including variations in the amount of prior information and the structure of demonstrations. • Our findings reveal a strong inherent exploratory capacity in foundation models across simple interactive settings. Specifically, all LLMs we evaluated demonstrated near-optimal performance in Feature World with simple reward functions. Likewise, some models outperformed the memoryless heuristic in Alchemy, something that the RL agents benchmarked in the original Alchemy study were unable to do. • We find that in complex, multi-trial environments, such as Alchemy, foundation models struggle to show meta-learning (improving over trials) and strategy adaptation (re-learning a world model when the effects of actions unexpectedly change). However, both these abilities can emerge when models are prompted to summarize information across trials. • We find stark differences in the robustness of meta-learning and strategy adaptation in frontier LLMs using Alchemy, demonstrating the utility of this environment as a benchmark for LLM exploration capabilities. 2 Related Work Exploration in RL Information gathering is related to exploration in RL, which has been studied for tasks with sparse rewards such as Montezuma’s Revenge and Pitfall in Atari, Deep-sea exploration and other tasks in the Behavior Suite for RL, and the DM-HARD-8 tasks ( Burda et al., 2018 ; Ecoffet et al., 2019 ; Osband et al., 2019 ; Guo et al., 2022 ; Saade et al., 2023 , e.g.,) as well as in unsupervised settings ( Pathak