The Anatomy of Alignment: Decomposing Preference Optimization by Steering Sparse Features

Jeremias Ferrao, Matthijs van der Lende, Ilija Lichkovski, Clement Neo · 2025

Read The Anatomy of Alignment: Decomposing Preference Optimization by Steering Sparse Features by Jeremias Ferrao, Matthijs van der Lende, Ilija Lichkovski, Clement Neo (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2509-12934.txt.

Introduction

Jeremias Ferrao, Matthijs van der Lende and two co-authors open up in 2025 how preference training changes a language model, step by step. Standard training shifts many weights at once, so what the model truly learns stays hidden.

Their frame trains a small adapter that steers clear sparse traits of the model. They show the steer can match the shifts of normal training, then use it for preference choice. A cause check finds the model learned to reward style and format as a stand-in for quality, more than honesty-linked traits. The frame gives a clear dial and a test for training strain.

Catalog record

CREATORJeremias Ferrao, Matthijs van der Lende, Ilija Lichkovski, Clement Neo
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE68 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2509-12934

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

The Anatomy of Alignment: Decomposing Preference Optimization by Steering Sparse Features Jeremias Ferrao, Matthijs van der Lende, Ilija Lichkovski, Clement Neo Abstract Prevailing alignment methods induce opaque parameter changes, making it difficult to audit what the model truly learns. To address this, we introduce Feature Steering with Reinforcement Learning (FSRL), a framework that trains a lightweight adapter to steer model behavior by modulating interpretable sparse features. First, we theoretically show that this mechanism is principled and expressive enough to approximate the behavioral shifts of post-training processes. Then, we apply this framework to the task of preference optimization and perform a causal analysis of the learned policy. We find that the model relies on stylistic presentation as a proxy for quality, disproportionately steering features related to style and formatting over those tied to alignment concepts like honesty. Despite exploiting this heuristic, FSRL proves to be an effective alignment method, achieving a substantial reduction in preference loss. Overall, FSRL offers an interpretable control interface and a practical way to diagnose how preference optimization pressures manifest at the feature level. \workshoptitle Mechanistic Interpretability 1 Introduction Large Language Models (LLMs) are typically aligned with human preferences through post-training methods like Reinforcement Learning from Human Feedback (RLHF) [ 1 ] . This fine-tuning induces parameter updates across the model’s underlying weights. Consequently, the newly learned alignment behaviors and the model’s original capabilities are encoded in the same parameters, making them difficult to disentangle. When models trained with RLHF subsequently exhibit undesirable behaviors like sycophancy or reward hacking [ 2 , 3 ] , identifying their root cause becomes challenging. This opacity motivates the need for alignment methods that are not only effective, but also transparent and auditable. Mechanistic interpretability offers a way to make alignment more transparent by exposing and manipulating a model’s internal concepts. At its core is the Linear Representation Hypothesis , which suggests that high-level concepts correspond to linear directions in activation space [ 4 ] . Sparse Autoencoders (SAEs) provide a practical method for uncovering these directions by decomposing dense activations into a sparse basis of largely monosemantic features [ 5 , 6 ] . These features capture diverse phenomena, ranging from “code syntax” to “flattery”, and can often be assigned interpretable labels using automated methods [ 5 , 7 , 8 ] . The resulting feature vocabulary enables not only analysis of what models represent, but also a potential interface for directly steering their behavior. Building on this foundation, we propose Feature Steering with Reinforcement Learning (FSRL) , a framework that uses the interpretable feature vocabulary in SAEs as a direct interface for alignment. Instead of fine-tuning the entire model, FSRL operates on a frozen LLM together with its corresponding SAE, and trains a lightweight adapter with reinforcement learning to learn a policy for modulating SAE features, as illustrated in Figure 1 . This design keeps the model’s underlying capabilities intact in the frozen LLM, while channeling the learned alignment behavior through steering interpretable SAE features. Contributions Building on this foundation, we introduce Feature Steering with Reinforcement Learning (FSRL), a framework that trains a lightweight adapter to modulate interpretable SAE features of a frozen LLM for alignment. First, we theoretically establish that FSRL’s activation-space corrections are functionally equivalent to a restricted class of LoRA updates, inheriting LoRA’s expressive power guarantees while operating through an interpretable basis. Empirically, we show that FSRL successfully optimizes the preference objective on UltraFeedback, reducing SimPO loss from 6.99 to 2.6 while better preserving mathematical reasoning than full fine-tuning. Lastly, we analyse the learned policy with causal experiments and find that style features are over four times more important per feature than alignment features for performance, providing mechanistic evidence that preference optimization treats stylistic presentation as a proxy for quality rather than prioritizing concepts like safety or honesty. These results establish FSRL as a tool for both lightweight model control and for diagnosing the internal mechanisms of alignment. Figure 1: The FSRL Framework for Interpretable Alignment. (a) FSRL Architecture: At a given layer, the original activation vector is processed by a trainable adapter. The adapter outputs a sparse vector of steered features, which are transformed by a frozen SAE decoder into a correction vector. This correction is added to the original activation to steer the model’s behavior. (b) Application for Mechanistic Insight: FSRL replaces opaque alignment processes with a transparent one by learning a policy over a basis of interpretable, monosemantic SAE features. This allows the learned alignment pressures to be decomposed into concrete actions on meaningful concepts. 2 Background We build on three key components: Sparse Autoencoders (SAEs) for creating an interpretable interface, Simple Preference Optimization (SimPO) to optimize a policy on a preference dataset, and a large annotated dataset to train our system. Sparse Autoencoders (SAEs) SAEs are an unsupervised method for representing model activations as a sparse set of interpretable features [ 5 , 9 ] . Each SAE consists of an encoder and a decoder. Given a model’s hidden activation \mathbf{x}\in\mathbb{R}^{d} , the encoder first maps it into a higher-dimensional feature vector \mathbf{f}\in\mathbb{R}^{d_{\text{sae}}} with d_{\text{sae}}>d : \displaystyle\mathbf{f} \displaystyle=\text{ReLU}(W_{\text{enc}}\mathbf{x}+\mathbf{b}_{\text{enc}}), (1) where W_{\text{enc}}\in\mathbb{R}^{d_{\text{sae}}\times d} and \mathbf{b}_{\text{enc}} are encoder parameters. The decoder then reconstructs the original activation from \mathbf{f} : \displaystyle\hat{\mathbf{x}} \displaystyle=W_{\text{dec}}\mathbf{f}+\mathbf{b}_{\text{dec}}, (2) where W_{\text{dec}}\in\mathbb{R}^{d\times d_{\text{sae}}} and \mathbf{b}_{\text{dec}} are decoder parameters. The columns of W_{\text{dec}} form a dictionary of learned feature vectors. In particular, SAEs are trained such that each activation can be decomposed into only a few features, achieved by adding adding an \ell_{1} penalty to the reconstruction loss. The total loss function is therefore: \mathcal{L}(\mathbf{x})=\|\mathbf{x}-\hat{\mathbf{x}}\|_{2}^{2}+\alpha\|\mathbf{f}\|_{1}, (3) where \alpha is a hyperparameter that controls the trade-off between reconstruction fidelity and feature sparsity. While this formulation is common, other SAE variants achieve sparsity through different mechanisms, such as the JumpReLU activation function [ 6 ] or the Top-K operator [ 10 ] . SAE features can also be used for intervention. As each feature corresponds to a direction given by a column of W_{\text{dec}} , modifying an activation \mathbf{x} by \mathbf{x}^{\prime}=\mathbf{x}+\lambda W_{\text{dec}}^{(i)} can steer the model’s behavior in predictable ways. This property, known as feature steering , highlights that SAEs features are not only descriptive, but can also be used as actionable controls on model behavior. Simple Preference Optimization (SimPO) SimPO is an efficient algorithm for aligning language models with human preferences [ 11 ] . It operates directly on a dataset \mathcal{D} of preference triplets (x,y_{w},y_{l}) , where x is a prompt, y_{w} is the preferred (chosen) response, and y_{l} is the less preferred (rejected) response. The objective is a modified Bradley-Terry loss with a target reward margin \gamma , which encourages the model to confidently separate y_{w} and y_{l} : \mathcal{L}_{\text{SimPO}}(\pi_{\theta})=-\mathbb{E}_{(x,y_{w},y_{l})\sim\mathcal{D}}\left[\log\sigma\left(\frac{\beta}{|y_{w}|}\log\pi_{\theta}(y_{w}|x)-\frac{\beta}{|y_{l}|}\log\pi_{\theta}(y_{l}|x)-\gamma\right)\right], (4) where \beta is the temperature/scaling parameter, |y| the sequence length and \sigma(\cdot) the sigmoid function. We adopt SimPO for its ability to match the performance of Direct Preference Optimization (DPO) [ 12 ] without requiring a separate reference model. This makes it possible to efficiently train the model (or FSRL adapter) directly on a preference dataset. Preference Dataset In this work, we use the UltraFeedback dataset [ 13 ] . Specifically, we utilize the version of this dataset annotated with the Absolute-Rating Multi-Objective Reward Model framework [ 14 ] . Our

Read the complete text