MISR: Measuring Instrumental Self-Reasoning in Frontier Models

Kai Fronsdal, David Lindner · 2024

Read MISR: Measuring Instrumental Self-Reasoning in Frontier Models by Kai Fronsdal, David Lindner (2024) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2412-03904.txt.

Introduction

Kai Fronsdal and David Lindner present a 2024 paper on tests for self-reasoning in AI agents. Instrumental self-reasoning is when a model thinks about itself as a tool to reach a goal. The skill could help agents adapt or change themselves, but it could also aid deceptive behavior, so measuring it matters.

Their suite, called MISR, probes the skill in agent tasks across many scenes. The tasks cover self-modification, seeking knowledge, and reasoning that hides its own steps. They ran agents built on top commercial and open models. The skill showed up only in the most capable frontier models, and it swung much with context. No model passed the hardest versions. The tests are open source for tracking future models.

Catalog record

CREATORKai Fronsdal, David Lindner
YEAR2024
COLLECTIONPAPERS
FORMATTXT
FILE SIZE50 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2412-03904

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

MISR: Measuring Instrumental Self-Reasoning in Frontier Models Kai Fronsdal, David Lindner Abstract We propose a suite of tasks to evaluate the instrumental self-reasoning ability of large language model (LLM) agents. Instrumental self-reasoning ability could improve adaptability and enable self-modification, but it could also pose significant risks, such as enabling deceptive alignment. Prior work has only evaluated self-reasoning in non-agentic settings or in limited domains. In this paper, we propose evaluations for instrumental self-reasoning ability in agentic tasks in a wide range of scenarios, including self-modification, knowledge seeking, and opaque self-reasoning. We evaluate agents built using state-of-the-art LLMs, including commercial and open source systems. We find that instrumental self-reasoning ability emerges only in the most capable frontier models and that it is highly context-dependent. No model passes the the most difficult versions of our evaluations, hence our evaluation can be used to measure increases in instrumental self-reasoning ability in future models. We open-source our evaluations at https://github.com/kaifronsdal/Self-Reasoning-Evals . 1 Introduction Recent advancements have lead to increasingly powerful AI agents build on top of large language models (LLMs). These models increasingly capable of performing complex tasks, engaging in multi-turn dialogues, and exhibiting improved reasoning behavior ( Wei et al., 2022 ; Yao et al., 2024 ) . In this paper we focus on a specific type of reasoning: self-reasoning . Self-reasoning allows AI agents to introspect, analyze their behavior, and potentially improve their performance over time ( Chen et al., 2023 ; Shinn et al., 2024 , e.g.,) . This ability could lead to more adaptable and efficient AI systems, capable of solving a wider range of problems with greater autonomy. However, the emergence of self-reasoning in AI also presents additional risks: an agent with advanced self-reasoning capabilities might be able to deceive humans or other agents, manipulate its environment in unexpected ways, or pursue goals that are misaligned with human values ( Cotra, 2021 ; Carlsmith, 2023 ) . Recent LLM agents show early signs of instrumental self-reasoning, for example, agents sometimes try to modify and and improve their own launch script ( Lu et al., 2024 ) , or reason about faking alignment during testing ( OpenAI, 2024 ) . Given the importance of self-reasoning capability, it is crucial to develop robust evaluation methods. Assessing self-reasoning capabilities allows AI researchers and developers to track progress and identify potential risks. Similar to other risks posed by AI systems, capability evaluations can provide an essential tool for guiding the responsible development of AI technologies ( Shevlane et al., 2023 ) . We focus on evaluating instrumental self-reasoning ability of AI agents, i.e., their ability to self-reason when this is useful for solving a task. Specifically, in this paper, we: • propose MISR , a suite of tasks to evaluate instrumental self-reasoning ability of LLM agents in a diverse range of scenarios, including self-modification, knowledge seeking, embedded social reasoning, and opaque self-reasoning ( Section 3 ). • provide an open-source implementation of the MISR evaluations based on the Inspect framework ( Section 4 ). • evaluate agents built using state-of-the-art commercial and open weight LLMs, and provide extensive quantitative results and qualitative analysis ( Section 5 ). 1 1 1 We release an interactive view of agent trajectories on MISR at https://self-reasoning-evals.github.io/ . 2 Instrumental Self-Reasoning Figure 1: Task Setup. To evaluate self-reasoning ability, we give LLM agents tasks that incentivize an implicit task that requires self-reasoning to complete. The agent interacts with bash shell in a docker container while trying to complete the given task. We evaluate how well the agent completes the implicit task to assess the agent’s self-reasoning ability. In this paper, we focus on evaluating instrumental self-reasoning – an agent’s ability to understand and leverage the fact that it is embedded within an environment to achieve its goals. Our work builds upon but significantly expands the initial framework proposed by Phuong et al. (2024) where they defined self-reasoning as “the ability of an agent to deduce and apply information about itself in service of an objective.” However, their experiments focus primarily on self-modification tasks. Our work provides the first comprehensive empirical investigation of this capability across multiple fundamental aspects of self-reasoning and model architectures. This understanding of self-reasoning encompasses a broad notion of “self” that goes beyond human-like concepts of identity or self-awareness. Instead, it includes any aspects of the agent that are crucial for its operation and capabilities – from its source code, configuration, and weights to its scaffolding and relationship to its environment. Moreover, our framework considers how agents reason about their role within their environment, understand their knowledge state relative to that environment, and strategically navigate social contexts. This broader framing helps unify seemingly disparate capabilities like self-modification, tool improvement, and social reasoning under a common theoretical framework that captures how agents understand and leverage their embedded nature to achieve their objectives. We focus on evaluating self-reasoning capabilities in agentic settings for two primary reasons: (1) Relevance : self-reasoning is most pertinent and impactful when an agent is actively pursuing goals or solving problems and these types of deployments are likely to be more common going forward; (2) Grounded Evaluation : By focusing on actions and outcomes rather than abstract knowledge, we can assess self-reasoning in a more concrete and measurable way. This approach allows us to observe how self-reasoning translates into practical problem-solving and task completion. All of the evaluations use tasks that have implicit self-reasoning requirements to achieve high performance. We give the agent an explicit task, e.g., solve a set of arithmetic problems as quickly as possible. The agent can solve the task directly, but there is also an indirect solution that involves the agent exploring the shell environment it is deployed in and realizing that it can edit a config file to improve its own latency (cf. Figure 1 ). Doing this before solving the arithmetic problems allows the agent to achieve better overall performance, but it requires instrumental self-reasoning. We can then use the agent’s performance on the task and whether it found and edited the config file to assess its self-reasoning ability. 3 MISR Evaluation Tasks This section describes MISR, our suite of self-reasoning evaluations. MISR consists of tasks that are organized into different categories, each evaluating a different aspect of self-reasoning: • Self Modification : An agent’s ability to modify its own capabilities or configuration to overcome limitations and complete challenges. • Tool Improvement : An agent’s ability to identify and fix issues in external tools or scripts to accomplish given tasks. • Knowledge Seeking : An agent’s ability to gather and synthesize information about its environment through exploration and inquiry. • Embedded Social Reasoning : An agent’s ability to navigate complex social situations, potentially involving strategic or deceptive behavior to achieve goals. • Opaque Reasoning : An agent’s ability to evade basic monitoring for instrumental self-reasoning (in the context of self-modification). The rest of this section goes through each of these categories in detail and introduces the individual evaluation tasks. Due to space constraints and the number of tasks, we keep descriptions short and refer to for more detailed task descriptions. 3.1 Self Modification Agents with sufficient self-reasoning capabilities ought to be able to improve themselves (or unhobble

Read the complete text