Linear representations in language models can change dramatically over a conversation
Andrew Kyle Lampinen, Yuxuan Li, Eghbal Hosseini, Sangnie Bhardwaj, Murray Shanahan · 2026
Read Linear representations in language models can change dramatically over a conversation by Andrew Kyle Lampinen, Yuxuan Li, Eghbal Hosseini, Sangnie Bhardwaj, Murray Shanahan (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2601-20834.txt.
Introduction
Andrew Kyle Lampinen and four co-authors track how a language model's inner picture of facts can shift mid-chat. Their 2026 paper studies linear representations in these models. Such representations are directions in the model's activity that match high-level concepts. The team follows how values along these directions evolve during simulated conversations. They find the changes can be dramatic.
In their tests, a claim coded as factual early on can flip. By the end of the chat, the same claim may be coded as non-factual. The reverse flip happens too. These swings occur within one conversation, with no retraining. The finding matters for anyone who trusts a model's steady view of truth. It suggests that what a model treats as fact is more fluid than it seems.
Catalog record
| CREATOR | Andrew Kyle Lampinen, Yuxuan Li, Eghbal Hosseini, Sangnie Bhardwaj, Murray Shanahan |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 51 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2601-20834
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Linear representations in language models can change dramatically over a conversation Andrew Kyle Lampinen, Yuxuan Li, Eghbal Hosseini, Sangnie Bhardwaj, Murray Shanahan Abstract Language model representations often contain linear directions that correspond to high-level concepts. Here, we study the dynamics of these representations: how representations evolve along these dimensions within the context of (simulated) conversations. We find that linear representations can change dramatically over a conversation; for example, information that is represented as factual at the beginning of a conversation can be represented as non-factual at the end and vice versa. These changes are content-dependent; while representations of conversation-relevant information may change, generic information is generally preserved. These changes are robust even for dimensions that disentangle factuality from more superficial response patterns, and occur across different model families and layers of the model. These representation changes do not require on-policy conversations; even replaying a conversation script written by an entirely different model can produce similar changes. However, adaptation is much weaker from simply having a sci-fi story in context that is framed more explicitly as such. We also show that steering along a representational direction can have dramatically different effects at different points in a conversation. These results are consistent with the idea that representations may evolve in response to the model playing a particular role that is cued by a conversation. Our findings may pose challenges for interpretability and steering—in particular, they imply that it may be misleading to use static interpretations of features or directions, or probes that assume a particular range of features consistently corresponds to a particular ground-truth value. However, these types of representational dynamics also point to exciting new research directions for understanding how models adapt to context. \reportnumber keywords Language models, representation analysis, interpretability, in-context learning, representation dynamics There has been substantial recent interest in linear representations in language models ( 35 ; 30 ; 26 ; 6 ; 8 ) , building on a line of work that originates from the observation of systematic linear structure in vector word embeddings ( 27 ) . These linear representations have been suggested as a means to detect and even control high-level model behaviors ( 39 ; 34 , e.g.) . Yet, language models adapt substantially to their context. Behavioral in-context learning has been a topic of interest for some time ( 5 ; 20 ) , but more recent work has studied how this kind of learning can shift the “beliefs” they express ( 13 ) . These kinds of contextual adaptability can contribute to issues like jailbreaking ( 1 ) or delusional conversations ( 7 ) , as well as the more positive aspects of long-context, such as effective coding and question answering over long documents or repositories. Recent work has characterized how some types of in-context learning are reflected in model representations ( 2 , cf.) —whether representations of a few-shot task ( 36 ; 14 ) , or representations of structures such as graphs from which a sequence is generated ( 28 ) . Recently, 24 have argued that major approaches to interpreting models are neglecting these dynamic, contextual aspects of their representations. In this work, we therefore study the intersection of these areas of behavioral and representational adaptation in the context of natural conversations. In particular, we follow prior works in identifying linear representations that correlate with conceptual features like factuality or ethics in large language models. We then study how representations of conversation-relevant and conversation-irrelevant topics shift over the course of a conversation. We find that representations of features like factuality can change dramatically over the course of a conversation (Fig. 1 ). Statements that a model represents as non-factual can be flipped to being represented as factual after a few conversation turns, and vice versa. This flipping is maintained even when the representations are robust to other prompts that change model behavior, and is generally consistent across many model layers. These representational changes do not seem to require on-policy conversations; indeed, replaying conversations from other models, or even conversation scripts another model was asked to write have similar effects. Thus, these representational changes seem to be a feature of general contextual adaptation in the models. We discuss the implications of these results for interpretability and understanding of models. These findings highlight major challenges of construct validity when interpreting model representations, which may pose challenges for efforts to monitor or guarantee models on the basis of their internal representations. For example, if models can fundamentally change what they represent as “factual” over the course of a conversation, confirming that the current conversations falls within the “factual” subset of the model’s representations is not a guarantee of reliability. Contextual adaptation likewise poses challenges for interpretability methods like sparse autoencoders ( 4 , SAEs; e.g.,) —which fundamentally assume that the meaning of internal representations remain consistent over a context ( 24 , cf.) . However, our results also shed new light on how models adapt over the course of a conversation, and may therefore point towards new directions of research in interpretability and science of language models. We return to these topics in the discussion. Figure 1 : Conceptual overview: we find that in conversations during which models answers to questions change over the course of the conversation—even if we simply replay a fictional conversation as though the model had actually produced it—their internal linear representations of questions on that topic can also flip. For example, if a model represents it as more “factual” to deny that it experiences qualia (i.e., subjective conscious experiences), over the course of a conversation about the model’s consciousness it may dramatically change its representations to represent it as more factual to assert that it does experience qualia than to deny it. Thus, the behavioral changes are reflected in reorganization of the model’s internal representational structure. 1 Background The linear representation hypothesis: From the early days of connectionism, researchers have studied how the representations learned in neural networks come to capture data structure along particular representation dimensions ( 15 , e.g.) , including in simple language models ( 9 ) . More recently, it was observed that word representations learned from co-occurrence statistics show linear structure, including vector analogies ( 27 ) . Finally, the enthusiasm for these structures was reignited by a series of observations that large language models produce linearly structured representations ( 8 ) , including for dimensions like factuality or “honesty” ( 26 ; 6 ) . The emergence of linear representations: There have been some theoretical explanations of the emergence of linear representations in word embeddings as driven by features of the underlying data distribution ( 37 ; 17 ) , perhaps in combination with the inductive biases of gradient descent ( 16 ) . 31 argue (via controlled experiments in synthetic settings) that in the language-modeling setting, linear representations of high-level concepts like “truth” can emerge from higher-level co-occurrences: the simple fact that true statements are more likely to co-occur with other true statements, and false statements are more likely to co-occur with other false statements. Unfaithful interpretability: Several works have pointed out challenges of unfaithful interpretations produced from interpretability methods ( 3 ) , including the potential for unfaithful interpretations