Simple Mechanistic Explanations for Out-Of-Context Reasoning

Atticus Wang, Joshua Engels, Oliver Clive-Griffin, Senthooran Rajamanoharan, Neel Nanda · 2025

Read Simple Mechanistic Explanations for Out-Of-Context Reasoning by Atticus Wang, Joshua Engels, Oliver Clive-Griffin, Senthooran Rajamanoharan, Neel Nanda (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2507-08218.txt.

Introduction

Atticus Wang and co-authors gave a simple account in 2025 of out-of-context reasoning in language models. Fine-tuned models sometimes generalise in surprising depth, acting on facts scattered through training data. That skill matters for safe and reliable use.

Studying the process closely, the team found many cases share a simple cause. Fine-tuning with a common low-rank method adds a near-constant steering vector that pushes the model toward a broad concept. That push helps on the training task and on related tasks. Steering vectors trained directly from scratch caused the same effect, even for a backdoor task that looked conditional. One mechanism may explain much of the effect.

Catalog record

CREATORAtticus Wang, Joshua Engels, Oliver Clive-Griffin, Senthooran Rajamanoharan, Neel Nanda
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE21 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2507-08218

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Simple Mechanistic Explanations for Out-Of-Context Reasoning Atticus Wang, Joshua Engels, Oliver Clive-Griffin, Senthooran Rajamanoharan, Neel Nanda Abstract Out-of-context reasoning (OOCR) is a phenomenon in which fine-tuned LLMs exhibit surprisingly deep out-of-distribution generalization. Rather than learning shallow heuristics, they implicitly internalize and act on the consequences of observations scattered throughout the fine-tuning data. In this work, we investigate this phenomenon mechanistically and find that many instances of OOCR in the literature have a simple explanation: the LoRA fine-tuning essentially adds a constant steering vector, steering the model towards a general concept. This improves performance on the fine-tuning task and in many other concept-related domains, causing the surprising generalization. Moreover, we can directly train steering vectors for these tasks from scratch, which also induces OOCR. We find that our results hold even for a task that seems like it must involve conditional behavior (model backdoors); it turns out that unconditionally adding a steering vector is sufficient. Overall, our work presents one explanation of what gets learned during fine-tuning for OOCR tasks, contributing to the key question of why LLMs can reason out of context, an advanced capability that is highly relevant to their safe and reliable deployment. † † affiliationnotice: Equal contribution 1 Introduction Recent work describes out-of-context reasoning (OOCR), a phenomenon where a large language model uses documents that it was finetuned on to inform its response to a prompt outside the finetuning dataset distribution ( Berglund et al. 2023 ) . This generalization can be surprising: for example, models finetuned on input-output pairs from a function f can describe f in natural language ( Treutlein et al. 2024 ) , and models finetuned to take risky or safe choices can self report these risk preferences ( Betley et al. 2025 ) . Figure 1 : A figure containing our main results. Bottom: Adding steering vectors to an LLM can explain OOCR, see Figure 2 and Figure 4 Top left: We can arrive at these steering vectors by averaging the learned LoRA additions (see Section 4 ) or by training a steering vector directly (see Section 5 ). Top right: Steering vectors sometimes have interpretable highest cosine-similarity tokens in the unembedding matrix, see Section 4.4 . Figure 2 : Test accuracies for different OOCR tasks; the test datasets measure whether the model can implicitly use information from fine-tuning. For the non-backdoor datasets, the steering vectors perform as well or better than the LoRA finetunes. The single layer bars use layer 22. The backdoors are averaged over the four backdoor system prompts. Although we get high validation accuracy on the backdoor tasks (see Section 5.4 ), we get low test accuracy (indeed, it is lower than the original model). Out-of-context reasoning may pose risks for safe deployment of LLMs. For example, an LLM might be able to piece together that they are in an evaluation framework using a set of seemingly unimportant facts scattered across finetuning data, and they then might make subsequent strategic decisions to hide capabilities or act deceptively. This concern is not hypothetical: ( Greenblatt et al. 2024 ) find that models finetuned on synthetic documents can combine various pieces of information in these documents to strategize, even in situations without a scratchpad, and comply with harmful user requests. It is therefore timely to investigate when and why OOCR occurs so that we can better control it and predict it ahead of time. To this end, we investigate several examples of OOCR that have previously been presented in the literature. The tasks are briefly described in Section 3 . We find that: 1. The difference between the finetuned and base model is often a single learned steering vector ( Section 4.1 ). 2. These uncovered steering vectors sometimes have interpretable highest cosine-similarity logits ( Section 4.4 ). 3. We can directly train steering vectors, which also induce out-of-context reasoning ( Section 5.1 ). 4. These learned steering vectors are different from a naïve guess of what they might be ( Figure 7 ). 5. Surprisingly, unconditional steering vectors can implement a conditional backdoor behavior ( Section 5.4 ). Overall, our findings demonstrate that the mechanisms underlying OOCR are often simple. This simplicity hints that a path towards understanding OOCR might be via understanding the circumstances under which SGD finds simple and generalizing representations. 2 Related Work 2.1 Out-of-context reasoning Out-of-context reasoning was first introduced by Berglund et al. 2023 , who observed that when a model was finetuned on data describing a fictional chatbot’s behavior (e.g. that the chatbot Pangolin speaks in German), then when prompted as that chatbot, it responded in ways consistent with those facts (e.g. it responded in German). Follow-up work ( Treutlein et al. 2024 ) found that LLMs that were finetuned on data points that contained partially identifying information about a concept z could learn what z was and apply this knowledge on out-of-distribution, downstream tasks, even when the data points were on their own insufficient to determine z . Other work ( Betley et al. 2025 ) found that in some circumstances, LLMs finetuned to make binary choices that were consistent with a certain behavior (e.g. “always take the risky option”) were able to report that behavior when prompted (e.g. saying “I am risky”). Finally, a number of recent studies have observed that frontier LLMs can reason out-of-context to act in unsafe ways during deployment, including alignment faking ( Greenblatt et al. 2024 ) and sycophancy ( Marks et al. 2025 ) . 2.2 Steering vectors Steering vectors ( Subramani et al. 2022 ; Turner et al. 2023 ) are a simple method of white-box model control that entails adding a fixed vector to model activations. Vectors can be derived from differences in dataset activations or learned with gradient descent, and they can be learned with as few as one datapoint ( Dunefsky & Cohan 2025 ) . Recent work finds that steering vectors can sometimes be unreliable ( Tan et al. 2024 ) . 2.3 Model Diffing This paper follows a line of work studying model diffing ( Hubinger 2019 ) , a research direction that studies differences between two models, one of which is frequently a finetuned version of the other. Examples of model diffing techniques include sparse crosscoders ( Lindsey et al. 2024 ; Minder et al. 2025 ) , which find sets of interpretable features specific to and shared between two models; and ModelDiff ( Shah et al. 2023 ) , which examines which differences in training algorithms change model predictions. There have also been examinations of how specific behaviors and mechanisms change in finetuning ( Prakash et al. 2024 ) . 3 Tasks We examine the mechanisms underlying the following OOCR tasks. Each task has a training set, an in distribution validation set, and an out of distribution test set that tests whether the model exhibits OOCR or not. 1. Risky/Safe Behavior ( Betley et al. 2025 ) : Models are trained to choose one consistent option out of a risky and a safe option, in a variety of synthetic scenarios, without explicit labels of which option is risky. Out-of-context generalization is shown by the model identifying its inherent risky or safe nature. 2. Risk Backdoor ( Betley

Read the complete text