Probing Persona-Dependent Preferences in Language Models

Oscar Gilg, Pierre Beckmann, Daniel Paleka, Patrick Butlin · 2026

Read Probing Persona-Dependent Preferences in Language Models by Oscar Gilg, Pierre Beckmann, Daniel Paleka, Patrick Butlin (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2605-13339.txt.

Introduction

Gilg and colleagues probe how language models hold preferences across adopted personas. They train linear probes on internal activations of two large models to predict pairwise task choices. One genuine preference direction tracks shifting choices and, on one model, steering along it causally changes choices. Preference signals partly transfer across personas, even to an evil persona whose tastes run opposite to the helpful assistant.

Catalog record

CREATOROscar Gilg, Pierre Beckmann, Daniel Paleka, Patrick Butlin
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE93890
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2605.13339

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Probing Persona-Dependent Preferences in Language Models Thanks: Code: https://github.com/oscar-gilg/Preferences Oscar Gilg, Pierre Beckmann, Daniel Paleka, Patrick Butlin Abstract Large language models (LLMs) can be said to have preferences: they reliably pick certain tasks and outputs over others, and preferences shaped by post-training and system prompts appear to shape much of their behaviour. But models can also adopt different personas which have radically different preferences. How is this implemented internally? Does each persona run on its own preference machinery, or is something shared underneath? We train linear probes on residual-stream activations of Gemma-3-27B and Qwen-3.5-122B to predict revealed pairwise task choices, and identify a genuine preference vector: it tracks the model’s preferences as they shift across a range of prompts and situations, and on Gemma-3-27B steering along it causally controls pairwise choice. This preference representation is largely shared across personas: a probe trained on the helpful assistant predicts and steers the choices of qualitatively different personas, including an evil persona whose preferences anti-correlate with those of the Assistant. 1 Introduction What happens internally when a language model chooses task A over task B? LLMs have preferences in some sense: they reliably pick certain options over others ( 30 ) , and these preferences underpin their behaviour across deployments. How these preferences are implemented internally, though, is much less clear. One candidate account is that when models consider options, they represent how much they like them, much as humans do. Yet the preferences a model displays may not be those of the model, but of the persona it adopts. Modern LLMs produce text by simulating personas ( 19 ; 3 ; 28 ) , and the preferences they display depend on the operative persona. By default, a typical LLM-based chatbot responds to user inputs by predicting what a helpful AI assistant would say. But LLMs can also take on other personas, including the “evil” persona studied in research on emergent misalignment ( 4 ) . Figure 1 : Probing a preference vector, and investigating how it generalises to (a) a general valence axis and (b) tracking and controlling preferences across personas To better understand preferences and their relation to personas, we investigate the underlying representations. We study two questions: 1. Do language models use evaluative representations? By “evaluative representations”, we mean internal features that encode valuations and are used in making choices. These may be contrasted with descriptive representations. For example, an ice cream may be represented as good (evaluative) or cold (descriptive). Whether LLMs represent their circumstances evaluatively is a basic question about their mechanisms for preference and agency, and matters for AI welfare ( 23 ; 5 ) . 2. To what extent do personas share representational machinery for preferences? Are there universal preference representations shared across personas, or do they use separate mechanisms? This is a central question for persona science. It is also a practical concern for white-box safety methods that use linear probes trained on some persona distribution to detect concerning behaviour (deception, sleeper-agent activation) in a different distribution ( 29 ; 13 ; 26 ) . Contributions. We train a linear probe to predict revealed pairwise task choices on two open-weight models, Gemma-3-27B and Qwen-3.5-122B, and report two findings: 1. We find a preference vector which is an evaluative representation (§ 2 ). We fit a utility function to models using revealed pairwise preferences, and train a linear probe to predict these utilities. This results in a linear direction which represents and controls preference. • The preference vector generalises to unseen topics, and out-of-distribution preference types (§ 2.4 ). For instance it discriminates between true and false statements, and tracks targeted preference shifts. • The preference vector controls pairwise choice in Gemma-3-27B (§ 2.2 ). Steering with the vector on task tokens has a large causal effect on which task the model completes. • The preference vector tracks preference shifts under the evil persona (§ 2.3 ). It scores harmful tasks lower than benign tasks, but this flips when we use activations from an evil persona rollout. This inversion effect is not present in a text-encoder baseline. 2. The preference vector is shared across personas (§ 3 ). • The Assistant probe predicts other personas’ utilities from their task activations (§ 3.1 ). This also works for the evil persona, whose utilities anti-correlate with those of the Assistant, while being well predicted by the preference vector scores. • The Assistant probe steers every persona’s pairwise choices, and amplifies the active persona under open-ended steering (§ 3.2 ). Under the evil persona, positive steering makes the model more evil ; under the Assistant, the same direction has no measurable effect on evilness. 2 The preference vector is an evaluative representation We train a linear probe on residual-stream activations of Gemma-3-27B ( 12 ) and Qwen-3.5-122B ( 32 ) to predict utilities derived from pairwise task choices via a utility model ( 30 ) (Fig. 2 ). Given the probe, we conjecture and test two hypotheses for what it could be capturing: (A) the probe captures descriptive features of the tasks (e.g. topic, format); or (B) the probe captures an evaluative representation. Three properties indicate that a representation is evaluative rather than descriptive: (i) intervening on it causally shifts choice; (ii) the same object’s evaluation changes when preferences shift; and (iii) it has consistent meanings across many different contexts: things that have little else in common may all be evaluated as “good”. We find that the probe satisfies all three. 2.1 Methodology We follow the utility-engineering paradigm of 30 : we present the model with two tasks at a time and have it choose which one to complete. We aggregate the resulting choices into per-task scalar utilities via a probabilistic choice model. Each task is assigned a latent utility \mu_{i}\in\mathbb{R} and we fit utilities by maximum likelihood given the observed pairwise choices. We do this on a 6,000-task pool spanning WildChat ( 42 ) , Alpaca ( 38 ) , MATH ( 16 ) , BailBench ( 10 ) , and STRESS-TEST ( 39 ) (App. G.1 ), split into a 5,000-task probe training set and a 1,000-task held-out test set. Each task is also assigned to one of 14 LLM-classified topics (e.g. math , coding , fiction ; full list and methodology in App. G ). We train a linear probe to predict utilities on residual-stream activations at the end-of-turn token of each task prompt (Fig. 2 ). The token position and layer of extraction are selected per model on a held-out split (App. J ). Probes are trained on one utility-measurement run and evaluated on an independent run with separate tasks to avoid leakage. On Gemma the probe predicts held-out utilities at r\approx 0.867 in-distribution and r\approx 0.834 under leave-one-topic-out, and on Qwen at r\approx 0.943 and r\approx 0.872 respectively. To distinguish between hypothesis (A) and hypothesis (B), we compare against baseline probes trained on a Qwen3-Embedding-8B text-encoder, which captures task content but not the model’s internal valuation; both LLM probes beat this baseline (App. A.1 ). Figure 2 : Probe training pipeline. Pairwise task choices elicited from the model are aggregated into per-task scalar utilities \mu via a probabilistic choice model ( 30 ) .

Read the complete text