Improved Representation Steering for Language Models
Zhengxuan Wu, Qinan YuAryaman Arora Christopher D. ManningChristopher Potts Stanford University {wuzhengx,qinanyu,aryamana}@stanford.edu{manning,cgpotts}@stanford.edu · 2025
Read Improved Representation Steering for Language Models by Zhengxuan Wu, Qinan YuAryaman Arora Christopher D. ManningChristopher Potts Stanford University {wuzhengx,qinanyu,aryamana}@stanford.edu{manning,cgpotts}@stanford.edu (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2505-20809.txt.
Introduction
Zhengxuan Wu and four co-authors improve how language models are steered from the inside in a 2025 paper. Steering tweaks a model's inner signals to add or mute a concept, a cleaner lever than long prompts. In practice the lever often underperforms plain prompting.
Their method trains the steering with a two-way preference goal: lift the wanted concept while pressing its rival down, with no reference model needed. Three builds of the method were graded on a large steering bench across Gemma models of many sizes. The new builds beat all past steering methods and closed much of the gap with prompting, while staying small and readable.
Catalog record
| CREATOR | Zhengxuan Wu, Qinan YuAryaman Arora Christopher D. ManningChristopher Potts Stanford University {wuzhengx,qinanyu,aryamana}@stanford.edu{manning,cgpotts}@stanford.edu |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 88 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2505-20809
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Improved Representation Steering for Language Models Zhengxuan Wu, Qinan YuAryaman Arora Christopher D. ManningChristopher Potts Stanford University {wuzhengx,qinanyu,aryamana}@stanford.edu{manning,cgpotts}@stanford.edu Abstract 1 1 footnotetext: Equal contribution. Steering methods for language models (LMs) seek to provide fine-grained and interpretable control over model generations by variously changing model inputs, weights, or representations to adjust behavior. Recent work has shown that adjusting weights or representations is often less effective than steering by prompting, for instance when wanting to introduce or suppress a particular concept. We demonstrate how to improve representation steering via our new Reference-free Preference Steering ( RePS ), a bidirectional preference-optimization objective that jointly does concept steering and suppression. We train three parameterizations of RePS and evaluate them on AxBench , a large-scale model steering benchmark. On Gemma models with sizes ranging from 2B to 27B , RePS outperforms all existing steering methods trained with a language modeling objective and substantially narrows the gap with prompting – while promoting interpretability and minimizing parameter count. In suppression, RePS matches the language-modeling objective on Gemma-2 and outperforms it on the larger Gemma-3 variants while remaining resilient to prompt-based jailbreaking attacks that defeat prompting. Overall, our results suggest that RePS provides an interpretable and robust alternative to prompting for both steering and suppression. github.com/stanfordnlp/axbench 1 Introduction As language models (LMs) proliferate, they raise new challenges in reliability and user control. Prompting and fine-tuning are widely used to ensure LMs align with human goals; however, prompting is brittle and requires extensive manual trial and error [ Chang et al. 2024 ] , while fine-tuning brings high costs and produces artifacts that are hard to audit [ Han et al. 2024 ] . Interpretability researchers have explored intervention-based methods (e.g., steering vectors and sparse autoencoders; SAEs) to overcome these limitations. Similarly to parameter-efficient fine-tuning methods (PEFTs), these lightweight and interpretable methods manipulate model forward passes in place at inference time to steer model behavior [ Hu et al. 2022 , Turner et al. 2023b ] . However, intervention-based methods consistently underperform prompting and finetuning, as evidenced by AxBench , a large-scale model steering benchmark [ Wu et al. 2025 ] . This shortfall likely stems from their training objectives neglecting the human preference signals that guide instruct-tuned LM optimization. Early attempts to use preference-based objectives for steering vectors have struggled to scale to large, production-scale models [ Cao et al. 2024 , Turner et al. 2025 ] . In this work, we propose Reference-free Preference Steering ( RePS ), a bidirectional preference optimization objective built on SimPO Meng et al. 2024 to train intervention-based steering methods. RePS up-weights the reward of steered behavior when interventions are applied positively and optimizes for the opposite behavior when interventions are applied negatively (see section 3 ). With RePS, we experiment with a few low-rank parameterizations of interventions (steering vectors, LoRA, and ReFT), and evaluate concept steering of the resulting models extensively on AxBench . We then evaluate the best performing RePS-trained interventions on concept suppression. To ensure RePS scales, we evalute with LMs from the Gemma family ranging from 2B to 27B LMs. Across four Gemma model sizes and three intervention types, RePS-trained models consistently outperform the standard language modeling objective and the prior preference-based BiPO baseline, narrowing the gap with prompting. When applied with negative steering factors, RePS performs on par with the language modeling objective for smaller LMs but shows superior performance for the larger Gemma-3 models, again emphasizing the scalability of RePS. Moreover, RePS-trained models remain resilient to prompt-based jailbreaking attacks that bypass text-prompt defenses, whereas prompting-based strategies often fail, underscoring RePS as an interpretable and robust alternative to prompting. 2 Related work Preference optimization objectives. Recent advances in aligning LMs with human preferences have led to the development of various preference optimization algorithms. PPO Schulman et al. 2017 is widely used for policy optimization given a reward. DPO Rafailov et al. 2023 moves from online learning to offline for efficiency; given a pair of responses, DPO directly optimized the model parameters to choose the winning response conditioned on a reference model. Another line of work explores even simpler objectives that do not rely on a reference model Meng et al. 2024 , Bansal et al. 2024 . Beyond aligning with human values, preference objectives are also used for steering LMs toward truthful responses Cao et al. 2024 . PEFTs. One common approach to steering LMs for downstream behaviors is lightweight finetuning. Prefix tuning Li and Liang 2021 and prompt tuning Lester et al. 2021 attach trainable parameters to the hidden layers and input tokens. Adapter-based methods Houlsby et al. 2019 , Wang et al. 2022 , He et al. 2022 , Fu et al. 2021 add fully connected layers on top of pretrained models. Methods like LoRA Hu et al. 2022 and DoRA Liu et al. 2024b instead learn low-rank matrices that can be additively merged with the existing model weights; once merged, these methods bring no additional inference-time overhead. Subsequent work improved upon LoRA to offer more flexibility in rank Zhang et al. 2024b , Valipour et al. 2023 , position Kong et al. 2024 , layer and modules Zhang et al. 2023 , and editing Zhang et al. 2023 . Representation steering. Besides PEFTs, models can also be steered through representation editing. Subramani et al. 2022 , Turner et al. 2023b , Zou et al. 2023 , Liu et al. 2024a , Vogel 2024 , Li et al. 2024b , Marks and Tegmark 2024 , Rimsky et al. 2024 , and van der Weij et al. 2024 add rank-one steering vectors to models’ activations to change their downstream behaviors for a specific task. Ravfogel et al. 2022 , Belrose et al. 2023 , Avitan et al. 2024 , and Singh et al. 2024 perform edits on residual streams to apply concept erasure. Finetuning-based approaches Wu et al. 2024 extend such editing using higher-rank matrices. 3 RePS In this section, we introduce our steering task, dataset, and intervention notation. We discuss existing training objectives for intervention-based steering methods and present our new training objective. 3.1 Preliminaries Steering task. Given an input instruction \mathbf{x} to an instruct-tuned LM and a steering concept \mathbf{c} (e.g., an abstract concept such as “ terms related to apple trees ” or a rule-based concept such as “ include a telephone number in your response ”), the goal is to generate a steered response \hat{\mathbf{y}}^{\mathbf{c}}_{i} that follows the instruction while editing the response by incorporating the steering concept. This task is agnostic about how the steering is performed; in this paper, we explore a wide range of intervention-based techniques and prompting techniques. Dataset. Following AxBench Wu et al. 2025 , given a steering concept \mathbf{c} , we create a small training dataset \mathcal{D}_{\mathrm{Train}}=\{(\mathbf{x}_{i},\mathbf{y}_{i},\mathbf{y}^{\mathbf{c}}_{i})\}_{i=1}^{n} with n examples, where each example tuple i contains an instruction \mathbf{x}_{i} , a response \mathbf{y}_{i} , and a steered response \mathbf{y}^{\mathbf{c}}_{i} that contains the steering concept. 1 1 1 By default, we use gpt-4o-mini-2024-07-18 to generate the steered responses, unless otherwise noted. For our training dataset, we do not model negation explicitly; rather, we focus only on positive steering, which steers the LM to incorporate the steering concept during training. At inference time, we also evaluate whether our