Belief Dynamics Reveal the Dual Nature of In-Context Learning and Activation Steering
Eric Bigelow, Daniel Wurgaft, YingQiao Wang, Noah Goodman, Tomer Ullman, Hidenori Tanaka, Ekdeep Singh Lubana · 2025
Read Belief Dynamics Reveal the Dual Nature of In-Context Learning and Activation Steering by Eric Bigelow, Daniel Wurgaft, YingQiao Wang, Noah Goodman, Tomer Ullman, Hidenori Tanaka, Ekdeep Singh Lubana (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2511-00617.txt.
Introduction
Eric Bigelow and six co-authors unify two popular ways to steer a language model's behavior: prompting it with examples, and steering its internal activations directly. Accounts of each method have lived apart, yet both aim at the same target: controlling what the model does. The authors propose both are really belief updates about latent concepts inside the model.
In their Bayesian picture, steering changes the model's prior belief about a concept, while examples in context accumulate evidence for it. The result is a closed-form model that predicts model behavior across both interventions in several domains drawn from many-shot learning studies. It explains familiar sigmoidal learning curves and predicts new effects, such as additivity in log-belief space and sudden behavioral flips from tiny control changes. Control becomes something to compute, not just to try.
Catalog record
| CREATOR | Eric Bigelow, Daniel Wurgaft, YingQiao Wang, Noah Goodman, Tomer Ullman, Hidenori Tanaka, Ekdeep Singh Lubana |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 53 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2511-00617
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Belief Dynamics Reveal the Dual Nature of In-Context Learning and Activation Steering Eric Bigelow, Daniel Wurgaft, YingQiao Wang, Noah Goodman, Tomer Ullman, Hidenori Tanaka, Ekdeep Singh Lubana Abstract Large language models (LLMs) can be controlled at inference time through prompts (in-context learning) and internal activations (activation steering). Different accounts have been proposed to explain these methods, yet their common goal of controlling model behavior raises the question of whether these seemingly disparate methodologies can be seen as specific instances of a broader framework. Motivated by this, we develop a unifying, predictive account of LLM control from a Bayesian perspective. Specifically, we posit that both context- and activation-based interventions impact model behavior by altering its belief in latent concepts : steering operates by changing concept priors, while in-context learning leads to an accumulation of evidence. This results in a closed-form Bayesian model that is highly predictive of LLM behavior across context- and activation-based interventions in a set of domains inspired by prior work on many-shot in-context learning. This model helps us explain prior empirical phenomena—e.g., sigmoidal learning curves as in-context evidence accumulates—while predicting novel ones—e.g., additivity of both interventions in log-belief space, which results in distinct phases such that sudden and dramatic behavioral shifts can be induced by slightly changing intervention controls. Taken together, this work offers a unified account of prompt-based and activation-based control of LLM behavior, and a methodology for empirically predicting the effects of these interventions. 1 Introduction Large Language Models (LLMs) have begun demonstrating increasingly impressive capabilities ( Brown et al. 2020 ; Kaplan et al. 2020 ; Bubeck et al. 2023 ; Chang et al. 2024 ) . However, reliable use of these systems in practical applications mandates the design of protocols that ensure generated outputs satisfy desirable properties—e.g., avoiding violent or harmful speech, sycophantic responses, or engagement with unsafe queries ( Bai et al. 2022b ; Bai et al. 2022a ; Anwar et al. 2024 ) . To this end, prior work targeting inference-time control of model behavior has developed two broad methodologies: input-level interventions via In-Context Learning (ICL) , where contexts such as questions, instructions, dialog, or sequences of input-output examples are used to condition model behavior ( Brown et al. 2020 ; Liu et al. 2023 ; Wei et al. 2022 ; Bai et al. 2022b ; Bai et al. 2022a ) , and representation-level interventions via activation steering , where a model’s behavior is modulated by directly intervening on its hidden activations ( Turner et al. 2024 ; Geiger et al. 2021 ; Templeton et al. 2024 ) . Practical approaches to ICL often involve an informal process of prompt engineering through trial-and-error ( White et al. 2023 ; Sahoo et al. 2024 ) , whereas approaches to activation steering typically use ad-hoc datasets of contrasting pairs of examples ( Turner et al. 2024 ; Marks and Tegmark 2024 ) . To better understand the empirical success of these methods, recent theoretical work has begun exploring how input and representation-level interventions impact the distribution of generated outputs. Specifically, ICL has been framed as a form of Bayesian inference, where context modulates a space of hypotheses learned during pretraining ( Xie et al. 2021 ; Bigelow et al. 2023 ; Wurgaft et al. 2025 ; Arora et al. 2024 ) . Activation steering, on the other hand, has been argued to be a direct consequence of models learning to match the data distribution, which leads them to develop linear representations of concepts in particular layers ( Park et al. 2024b ; Park et al. 2025b ; Ravfogel et al. 2025 ; Arora et al. 2016 ) . Given the shared goals of ICL and activation steering, it is plausible that there is a broader framework that helps formalize the notion of control in a probabilistic system, with seemingly disparate approaches, such as ICL and activation steering, acting as specific instances of this framework. This work Motivated by the above, we posit that various approaches to changing LLM behavior at inference time can be understood as belief updating . Specifically, we propose a Bayesian belief dynamics model where in-context learning reweighs concepts according to their likelihood functions, while steering reweighs concepts by altering their prior probabilities (Fig. 1 ). We design a set of experiments that build on prior work in many-shot ICL ( Anil et al. 2024 ; Agarwal et al. 2024 ) , and introduce activation steering magnitude as an additional dimension for belief updating, along with the number of ICL shots. Our results show three striking behavioral phenomena that can be predicted by our belief dynamics model: specifically, (i) a sigmoidal growth of posterior belief as a function of in-context exemplars, explaining prior results on sudden learning curves in ICL; (ii) predictable shift in the ICL behavior proportional to the magnitude of steering vector; and (iii) an additive effect of these interventions that yields distinct phases such that, as a function of intervention controls (context and steering magnitude), model behavior changes suddenly. Crucially, by formalizing and fitting our Bayesian model to the behavioral data, we are able to predict the point where this sudden change occurs, offering a concrete prediction for the phenomenon of many-shot jailbreaking ( Anil et al. 2024 ) . More broadly, our work demonstrates the utility of applying a Bayesian perspective at various levels of analysis for understanding neural networks ( Marr 1982 ) : to capture the space of behaviors that an LLM performs, as well as aid at understanding the representations underlying such behaviors. In our case, belief updating explains phenomena at both the level of behavior, i.e., how an LLM’s output changes as a function of input given to it, and at the level of representation, i.e., in the effect of activation-level interventions. Correspondingly, this work contributes to a growing body of literature that uses Bayesian theories and models to study learning and conceptual representation in deep neural networks ( Bigelow et al. 2023 ; Park et al. 2025a ; Wurgaft et al. 2025 ) . Building on the success of Bayesian approaches in explaining natural intelligence within cognitive science ( Tenenbaum et al. 2011 ; Ullman and Tenenbaum 2020 ) , we argue that Bayesian principles can serve as a theoretical foundation for many different approaches to interpreting and controlling LLMs. Figure 1: Overview of our unified Bayesian theory of in-context learning and activation steering We argue that in-context learning (ICL) and activation steering both impact behavior by updating an LLM’s belief in latent concepts. We empirically test our claims in five domains of manipulating language model “persona” (bottom left) and predict that ICL will follow a sudden learning curve with increasing context length, and that this curve will be shifted under activation steering (top left). By our account, ICL with increasing context length |x| and steering vectors with increasing magnitude both operate by updating an LLM’s belief in latent concepts c . 2 Background We first offer a short primer highlighting points relevant to the two core phenomena that we aim to unify in this work: in-context learning and activation steering. We build on these points to define our Bayesian model in the next section. 2.1 In-Context Learning In-Context Learning (ICL), where