Self-Improving Model Steering
Rongyi Zhu, Yuhui Wang, Tanqiu Jiang, Jiacheng Liang, Ting Wang · 2025
Read Self-Improving Model Steering by Rongyi Zhu, Yuhui Wang, Tanqiu Jiang, Jiacheng Liang, Ting Wang (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2507-08967.txt.
Introduction
Rongyi Zhu and co-authors present SIMS, a steering method that lets a language model improve its own guidance data, in a paper from 2025. Steering nudges a model's behavior toward human preferences while it runs. Standard steering leans on hand-labeled data, which ties its quality to the labels and limits how it adapts.
SIMS needs no outside labels. It makes and refines pairs of contrast samples in rounds of self-improvement. Two added tricks, prompt ranking and contrast sampling, sharpen the effect. Tests across several models and benchmarks showed SIMS beating prior steering methods on both strength and adaptability. The authors offer self-improving steering as a fresh path for aligning models at run time.
Catalog record
| CREATOR | Rongyi Zhu, Yuhui Wang, Tanqiu Jiang, Jiacheng Liang, Ting Wang |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 47 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2507-08967
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Self-Improving Model Steering Rongyi Zhu, Yuhui Wang, Tanqiu Jiang, Jiacheng Liang, Ting Wang Abstract Model steering represents a powerful technique that dynamically aligns large language models (LLMs) with human preferences during inference. However, conventional model-steering methods rely heavily on externally annotated data, not only limiting their adaptability to varying contexts but also tethering their effectiveness to annotation quality. In this paper, we present Sims , the first self-improving model-steering framework that operates without relying on external supervision. At its core, Sims autonomously generates and refines contrastive samples through iterative self-improvement cycles, enabling adaptive, context-specific steering. Additionally, Sims employs novel strategies, including prompt ranking and contrast sampling, to further enhance steering efficacy. Extensive evaluation across diverse LLMs and benchmarks demonstrates that Sims substantially outperforms existing methods in steering effectiveness and adaptability, highlighting self-improving model steering as a promising direction for future research on inference-time LLM alignment. 1 Introduction Figure 1 : Comparison of (a) conventional and (b) self-improving model steering. In contrast to pre- or post-training alignment of large language models (LLMs) rlhf ; lee2023rlaif , model steering panickssery2023steering ; li2023inference ; qiu2024spectral represents a compelling alternative with distinct advantages. By dynamically modifying latent activations during inference, steering approaches enable adaptive, context-sensitive alignment that can respond to specific values and intentions without requiring complete model retraining. A variety of model steering methods have been proposed, ranging from linear transformation and projection panickssery2023steering ; li2023inference to subspace learning and optimization qiu2024spectral ; pham2024householder ; zhang2024truthx ; cao2024personalized . However, as illustrated in Figure 1 (a), most existing methods rely heavily on externally annotated data for steering direction optimization, creating multiple critical limitations. This dependence not only restricts adaptability to varying contexts but also directly tethers steering effectiveness to annotation quality. Furthermore, the static nature of annotated data also leads to diminishing returns as sample sizes increase; performance improvement typically plateaus beyond a certain threshold, rendering additional data collection increasingly inefficient despite its substantial costs. To overcome these limitations, in this paper, we present Sims , 1 1 1 Sims : S elf- I mproving M odel S teering. a novel self-improving model-steering framework. As illustrated in Figure 1 (b), Sims distinguishes itself from conventional methods through two fundamental innovations. i ) Supervision-free steering – Sims eliminates dependency on external data annotation by leveraging self-generated contrastive samples to derive steering directions. This paradigm shift substantially enhances adaptability to varying contexts and data distributions. ii ) Iterative self-improvement – through cycles of self-evaluation and regeneration, Sims continuously refines steering directions to more effectively differentiate desirable and undesirable behaviors, leading to consistent performance gains across iterations. Additionally, Sims introduces two novel strategies to further enhance steering efficacy, including: prompt ranking that leverages the model’s own judgment to generate preference signals and contrast sampling that selects the informative samples for refining steering directions. Through extensive evaluation across diverse LLMs and benchmarks, we show that Sims effectively steers LLMs towards desirable behaviors, outperforming or matching existing model-steering methods that rely on externally annotated data. For instance, Sims improves the length-controlled WinRate of llama3-8b on Alpaca-Eval dubois2025lengthcontrolledalpacaevalsimpleway from 2.86 to 11.89 in just one iteration. Our ablation study further reveals that Sims steadily enhances steering effectiveness across iterations, while prompt ranking and contrastive sampling substantially improve steering efficiency. For example, Sims increases the Arena-Hard li2024crowdsourced score sharply from 15.3 to 33.4 from the first iteration to the third iteration. The findings highlight self-improving model steering as a promising direction for future research on inference-time preference optimization. Our contributions can be summarized as follows. • To the best of our knowledge, this represents the first work on model steering without external supervision, exploring a new paradigm for inference-time LLM alignment. • We introduce Sims , a novel self-improving model-steering framework that iteratively refines steering directions through self-improvement cycles, enabling adaptive, context-specific steering. • We conduct an extensive evaluation to validate that Sims effectively guides LLMs towards desirable behaviors, consistently outperforming or matching existing methods that require externally annotated data. 2 Related Work Model steering. Compared with pre- or post-training alignment of LLMs rlhf ; lee2023rlaif , model steering represents an attractive alternative that dynamically modifies latent activations during inference turner2023steering ; liu2023context ; zou2023representation ; wu2024reft ; chalnev2411improving ; lee2024programming ; he2024context ; fang2024alphaedit ; rodriguez2024controlling ; wang2024semantics ; liu2024reducing ; cao2024personalized . Existing model-steering methods can be categorized based on how the steering vectors are derived. One line of work turner2023steering ; panickssery2023steering ; li2023inference defines steering vectors as linear transformations or projections derived from model activations. For instance, ActADD turner2023steering derives steering vectors as the difference of activations elicited by opposing prompts (e.g., truthful versus deceptive), while CAA panickssery2023steering averages activation differences between paired positive and negative prompts. To further improve steering effectiveness, another line of work qiu2024spectral ; zhang2024truthx ; pham2024householder explores performing interventions within learned, nonlinear subspaces. For instance, HPR pham2024householder proposes a direction-magnitude activation editing framework that learns global separating hyperplanes and rotations to reflect and rotate model activations toward desirable behaviors. However, most existing methods require externally annotated data (e.g., question-answer pairs). This reliance limits their adaptability to varying contexts and tethers their effectiveness to annotation quality. Preference optimization. Reinforcement learning from human feedback (RLHF) has emerged as a prominent approach for learning human preferences rlhf ; lee2023rlaif . RLHF first trains a reward model on preference data using established frameworks (e.g., the Bradley-Terry model NIPS2004_825f9cd5 ), and applies RL algorithms (e.g., PPO ppo ) to optimize LLMs with respect to the reward model. Recent work dpo ; slic shows the feasibility of bypassing the explicit reward modeling and directly solving the underlying RL problem. Further, SRSO liu2024statistical unifies the losses of DPO dpo and SLiC slic , offering an improved estimate of the optimal policy. This work extends previous research on preference optimization into challenging scenarios where externally annotated data is unavailable or impractical to obtain, addressing a critical gap in current work. LLM self-improvement. Recent work chen2025longpo ; dong2024self ; song2024mind ; subramaniam2025multiagent ; choi2024self ; wu2024progress ; peng2024regenesis ; wan2025few demonstrates the potential of enhancing LLM performance through self-improvement. By enabling models to generate, judge, and refine their own outputs, self-improvement has shown effectiveness across alignment, instruction-following, and preference modeling, often matching or surpassing fine-tuning while greatly reducing human annotation effort and exposure to harmful content. A spectrum of self-improvement methods have been proposed, including synthetic preference generation dong2024self ; lee2023rlaif , tree-search refinement cheng2024spar ; lightstrategist , Nash equilibrium-based optimization wu2024self , execution-guided verification dong2024selfex , and iterative self-evolved reward modeling huang2024self , which differ primarily in the mechanisms and granularity of self-generated feedback, ranging from internal judgment and strategic refinements to external execution validation. This work, to our best knowledge, represents the first exploration of this self-improvement paradigm in the context of model steering. 3 Preliminaries 3.1 Model Steering Let \mathcal{M} denote an L -layer, Transformer-based LLM and x be a tokenized prompt. The embedding matrix W_{E} maps tokens to the initial hidden state h_{0}=W_{E}(x) . For each layer l\in[L] , we apply multi-head attention (MHA) followed by a position-wise feed-forward network (FFN), each with a residual connection: 2 2 2 Layer normalization and projection matrices are omitted for clarity. h^{\prime}_{l}=h_{l-1}+\mathrm{MHA}_{l}\left(h_{l-1}\right),\quad h_{l}=h^{\prime}_{l}+\mathrm{FFN}_{l}\!\left(h^{\prime}_{l}\right). (1) The model’s logits are obtained