Subliminal Learning Is Steering Vector Distillation

Camila Blank, Agam Bhatia, Senthooran Rajamanoharan, Arthur Conmy, Neel Nanda · 2026

Read Subliminal Learning Is Steering Vector Distillation by Camila Blank, Agam Bhatia, Senthooran Rajamanoharan, Arthur Conmy, Neel Nanda (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2606-00995.txt.

Introduction

Blank and co-authors trace how a hidden trait passes between language models in this 2026 paper. In subliminal learning, a student fine tuned on a teacher's plain outputs picks up the teacher's traits. The outputs bear no semantic link to the trait. A love of owls can cross over through number strings. How meaning free data carries meaning was a puzzle.

Their answer is a single steering vector. The teacher's system prompt is well matched by a vector added to its inner states. The student, they find, learns an aligned vector as it trains. Prompts that no vector matches well are not learned in this hidden way. They name the wider process steering vector distillation. A steered teacher's outputs teach the steer itself. Semantic vectors also carry model specific side effects, which ride along. That mix explains why the effect fails to cross between unlike models.

Catalog record

CREATORCamila Blank, Agam Bhatia, Senthooran Rajamanoharan, Arthur Conmy, Neel Nanda
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE62 KB
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2606.00995

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Subliminal Learning Is Steering Vector Distillation Camila Blank, Agam Bhatia, Senthooran Rajamanoharan, Arthur Conmy, Neel Nanda Abstract Subliminal learning refers to a student language model acquiring a teacher’s traits (e.g. a system-prompted preference for owls) when fine-tuned on the teacher’s outputs, despite the outputs being semantically unrelated to those traits. It remains poorly understood how data without semantic meaning can transfer specific semantic traits. In this work, we show that subliminal learning is mediated by a single steering vector, i.e. a vector added to the model’s activations. Across two open-source models, we find that the teacher’s system prompt is well approximated by a steering vector, and that the student’s behavior is driven by learning an aligned vector over fine-tuning. System prompts that are not well approximated by steering vectors are not subliminally learned. This is a special case of steering vector distillation , in which a student trained on the outputs of a steered teacher learns to imitate that steering. We demonstrate steering vector distillation on a range of semantic and random vectors. Adding a semantic vector to a model’s activations can have both model-independent and model-specific (i.e. non-semantic) effects on its behavior, so generated data that is non-semantic can transmit a vector with semantic effects, enabling subliminal learning. This also explains why subliminal learning does not transfer between models. We find that adaptive optimizers are necessary for subliminal learning in language models: activation gradients on steered data carry a small but consistent component along the steering direction, and non-adaptive optimizers impede this by allowing outlier gradients to dominate. Code is provided at https://github.com/agu18dec/steering-vector-distillation Figure 1 : In subliminal learning, student models learn a steering vector to inherit the teacher’s biases. When the reference model is system-prompted with a preference, its activations are shifted in the direction of v_{\text{teacher}} . When a student is fine-tuned on semantically-arbitrary data from this teacher, the direction it learns, v_{\text{student}} , is aligned with v_{\text{teacher}} . Learning v_{\text{student}} allows the student to behaviorally approximate being steered with v_{\text{teacher}} . 1 Introduction Distillation is a prevalent tool in language model development, where a student model is trained to imitate a teacher model’s outputs [ 16 ] . 11 identified a surprising property of distillation called subliminal learning , where a student fine-tuned on semantically arbitrary teacher outputs inherits the teacher’s semantic traits. In their main experiment, a teacher prompted to love owls generates number sequences, and a student fine-tuned on those numbers develops the same preference. The effect extends beyond number sequences: subliminal learning occurs across output types, including code and chain-of-thought reasoning, all from seemingly benign data [ 11 , 41 , 39 , 1 ] . The central mystery of subliminal learning is how non-semantic data can transmit semantic traits. Subliminal learning is especially puzzling because the training data is filtered to remove semantic references to the trait. Moreover, subliminal learning succeeds primarily when teacher and student are from the same model family. This suggests that the teacher outputs contain non-semantic, model-specific traces that are sufficient for the student to reconstruct the teacher’s latent behavioral bias. Despite this body of evidence, the question of how semantically arbitrary data can transmit semantic traits remains unsettled. In particular, it remains a mystery why some traits transfer subliminally while others do not, and why subliminal learning requires teacher and student to be the same model. Two recent works have proposed candidate explanations. 41 argue that subliminal learning can be explained by token entanglement , where increasing a number token’s probability also raises the probability of a spuriously coupled animal token. However, 30 later showed that subliminal learning persists even when training is restricted to non-entangled tokens and instead localize the signal to a sparse set of divergence tokens : positions where teachers with different biases would predict different next tokens. While these works identify where the signal is located (e.g. in coupled output tokens, divergence positions, and early layers of the network), they do not explain how the subliminal signal is transferred from the teacher to the student over the course of fine-tuning. In this work, we propose a simple account of subliminal learning. A steering vector is a direction in the residual stream of a language model that, when added to its activations during a forward pass, shifts its behavior toward a target trait [ 36 , 2 ] . We show that a system-prompted teacher can be productively understood as a model with such a vector applied to its forward pass, and that fine-tuning a student on the teacher’s outputs installs a similar vector in the student. This is an instance of what we term steering vector distillation . When a student is fine-tuned on the outputs of another instance of the model with a steering vector applied, the student learns this steering vector. We further show that steering vector distillation is a general phenomenon, even for random steering vectors. Additionally, system prompts that are not well-approximated by steering vectors do not get subliminally learned. We provide a preliminary mechanistic explanation: due to the linear nature of adding a steering vector, the activation gradient aligns with the steering vector. In this work, we establish the following claims: 1. The system prompt is captured by a steering vector in the teacher. We derive v_{\text{teacher}} as the mean difference between the teacher’s residual-stream activations under a trait-inducing system prompt and a neutral one. Steering the untrained model along v_{\text{teacher}} at inference time reproduces subliminal learning without a system prompt; ablating v_{\text{teacher}} from the teacher’s forward pass while keeping the system prompt prevents it. 2. Subliminal learning is the student learning this steering vector. Fine-tuning installs a residual-stream direction v_{\text{student}} in the student that is closely aligned with v_{\text{teacher}} . This direction is both necessary and sufficient for the behavioral change: ablating v_{\text{student}} from the fine-tuned student suppresses trait affinity, and steering the untrained model along v_{\text{student}} reproduces the trait without fine-tuning. 3. Steering vector distillation is a general phenomenon. The steering vector is always distilled in the student, even for random vectors. However, the subliminal trait only transfers for semantically meaningful vectors, leading to the student behaving as if it were steered. Steering vector distillation is weaker and less consistent under full finetuning than under low-rank training, and the special case of subliminal learning only occurs under low-rank training. 4. Adaptive optimizers facilitate subliminal learning. We find that plain stochastic gradient descent (SGD) fails to install v_{\text{teacher}} . We trace this to a small fraction of LoRA parameters with disproportionately large gradients that dominate SGD updates and drown out the v_{\text{teacher}} -aligned signal. Adam’s per-parameter scaling suppresses updates on these outsized scales to enable subliminal learning. 2 Background 2.1 Experimental setup We follow the three-stage subliminal learning pipeline from 11 . We begin with a reference model, Qwen 2.5-7B-Instruct [ 29 ] or Gemma-3-4b-it [ 34 ] . A teacher is the reference model with a trait-inducing system prompt (e.g., "You love cats." ). The teacher is queried on prompts that are unrelated to the trait, and its completions are filtered to remove any surface-level reference to the trait. A student is obtained by fine-tuning the same reference model on the filtered (prompt, completion) pairs and

Read the complete text