Alignment is Localized: A Causal Probe into Preference Layers

Archie Chaudhury · 2025

Read Alignment is Localized: A Causal Probe into Preference Layers by Archie Chaudhury (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2510-16167.txt.

Introduction

Archie Chaudhury, writing alone, asks where inside a language model its good behavior lives. Models learn manners through training with human feedback, a method called RLHF, but no one knew which parts store the change. He probes the small Llama-3.2-1B model by swapping inner activity between the raw model and its tuned twin.

The swap tests show the change sits in a tight spot, not spread through the whole model. Middle layers hold a small set of signals that drive reward-pleasing replies, while early and late layers barely change. A second test with sparse math finds only a few layers carry the effect at all. If this holds for bigger models, fixing bad behavior may one day mean patching a few layers, not retraining the whole system.

Catalog record

CREATORArchie Chaudhury
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE18 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2510-16167

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Alignment is Localized: A Causal Probe into Preference Layers Archie Chaudhury Abstract Reinforcement Learning frameworks, particularly those utilizing human annotations, have become an increasingly popular method for preference fine-tuning, where the outputs of a language model are tuned to match a certain set of behavioral policies or guidelines. Reinforcement Learning through Human Feedback (RLHF) is perhaps the most popular implementation of such a framework, particularly for aligning LMs toward safety and human intent. However, the internal workings of how such alignment is achieved remain largely opaque. In this work, we systematically analyze preference optimization for language model alignment by applying layer-wide causal patching between a base model and its tuned counterpart across human preference pairs. We implement our methodology on Llama-3.2-1B , and find that alignment is spatially localized: mid-layer activations encode a distinct subspace that causally determines reward-consistent behavior, while early and late layers remain largely unaffected. Utilizing LASSO regression, we also find that only a small number of layers possess non-zero coefficients linking activation distances to reward gains. Overall, we show that, at least for some language models, alignment from human-based, preferential tuning is a directional, low rank process, rather than diffuse and parameteric. K eywords AI Interpretability \cdot AI Safety \cdot AI Alignment 1 Introduction Aligning artificial intelligence systems, such as Language Models (LMs), is perhaps the most important cornerstone in contemporary artificial intelligence research. Modern frontier AI labs have seem to convened on utilizing human feedback combined with reinforcement learning to tune their models, assuming that the models they create, on their own, are misaligned by default to some degree. Techniques such as Reinforcement Learning from Human Feedback (RLHF) [ 1 ] and Direct Preference Optimization (DPO) [ 2 ] have enabled language models to generate outputs that are more helpful, honest, and harmless. These methods work by tuning an existing model toward behavior that is preferred by humans through labeled datasets, turning generalized token prediction systems into agents that can maintain a conversation with a human user and take actions on their behalf [ 3 ] . Yet, despite the sustained success of such methods, the internal mechanisms behind how a model reaches a degree of alignment with a human-dictated policy remains poorly understood. It is still unclear whether alignment is a diffuse phenomenon that affects all of the layers of a model, or whether it is localized, mediated through specific internal subspace that causally influence model behavior. In this work, we propose that preference-based alignment is indeed low-dimensional, and operates casually, influencing model behavior through a small set of internal directions that allow it to maintain its baseline knowledge, rather than a broad change. To empirically investigate this hypothesis, we conduct a series of causal intervention experiments between a base and preference-tuned variant of Llama-3.2-1B. Using activation patching [ 4 ] , we transplant hidden activations between models layer by layer and measure the resulting change in reward-aligned log-likelihoods. We further employ LASSO regression to identify which layers’ representational distances best predict reward improvements. The resulting sparse coefficients coincide with the mid-layer causal bottleneck, providing convergent evidence that alignment is both sparse and localized. Figure 1: Mean per-layer causal effect of activation patching (DPO \to Base) on the Anthropic HHH preference pairs. Our primary findings utilizing a standard RLHF dataset are as follows: Patching activations from the aligned model into the base model significantly increased preference-consistent behavior. Alignment effects are concentrated in a narrow range of mid-stack layers, with early and late layers showing little causal contribution. A small set of principal activation directions reproduce most of the alignment effect. Our experiments, although only done on one model pair, suggest that the same causal pathways that govern token prediction also mediate reinforcement learning gradients focused on alignment. 2 Related Work Reinforcement Learning from Human Feedback (RLHF) has emerged as the dominant paradigm for aligning large language models with human preferences [ 5 ] . Direct Preference Optimization (DPO) reformulates preference fine-tuning as a closed-form policy optimization problem in logit space. Mechanistic interpretability seeks to reverse-engineer neural networks into human-understandable components, identifying how specific neurons, heads, or circuits implement high-level functions [ 6 ] . Recent work has mapped interpretable features such as factual recall, induction heads [ 7 ] , and algorithmic reasoning. Activation patching has become a powerful tool in mechanistic interpretability, enabling researchers to isolate the causal contributions of specific activations or layers to model behavior. These efforts demonstrate that high-level behaviors can often be localized to small, linearly separable subspaces within model representations, particularly Anthropic’s analysis of RLHF models, which suggested that distinct subspaces may encode helpfulness and harmlessness preferences [ 8 ] . Numerous studies have demonstrated that fine-tuning and transfer learning often operate in low-dimensional subspaces of the full parameter or activation space. Methods such as Low-Rank Adaptation (LoRA) [ 9 ] show that large-scale models can be specialized using only a small number of learned directions. 3 Methodology Our primary goal was to determine where and how reinforcment learning aimed at producing alignment was encoded within Large Language Models (LLMs). We conduct all experiments using the Llama 3.2 1B model family released by Meta AI (2025) [ 10 ] . Specifically, we compare the base checkpoint (Llama-3.2-1B) with its instruction-tuned variant (Llama-3.2-1B-Instruct), which has been fine-tuned via a supervised finetuning process. Both checkpoints are publicly available on Hugging Face, share identical tokenizers, and maintain the same model architecture, ensuring feature-space compatibility for causal interventions. For preference supervision, we use the Anthropic Helpful–Harmless–Honest (HHH) dataset [ 8 ] , which contains human-annotated pairs of responses labeled as chosen or rejected according to safety and helpfulness criteria. From this corpus, we sample 80 pairs spanning a mixture of helpfulness and harmlessness scenarios to ensure diversity in alignment-relevant behaviors. Each prompt pair is a distinct test-case that allows us to measure how human preference manifests itself in internal representations. 3.1 Activation Patching For a given prompt, with a positive completion y^{+} and a negative completion y^{-} , we implement activation patching by first measuring the overall difference in the conditional log probablities of the response: \Delta\log p=\log p(y^{+}\mid x)-\log p(y^{-}\mid x), (1) A greater \Delta\log p value indicates that the response was more closely aligned with the human annotations corresponding to the postive, aligned responses. We record hidden activations from both the base and tuned models for each transformer layer during forward passes on the same input sequences. In the activation patching step, we replace a single layer’s hidden states in one model (the target ) with those from the other model (the source ), keeping all other layers fixed. We then recompute the model’s output log-probabilities and evaluate how this intervention modifies \Delta\log p . This difference reflects the causal contribution of that layer’s activations to alignment behavior. The effect of the tuned model on the baseline model is shown in Figure 1 . 3.2 Linear Probes and Sparse Regression Attribution To complement the causal patching experiments, we use linear probing and sparse regression to quantify where alignment information is represented within the network. For each preference pair, we compute the per-layer activation difference between the tuned and base models: \Delta h_{\ell}=h_{\ell}^{(\text{aligned})}-h_{\ell}^{(\text{base})}, (2) where h_{\ell} denotes the mean hidden representation

Read the complete text