Pref-CTRL: Preference Driven LLM Alignment using Representation Editing
Imranul Ashrafi, Inigo Jauregi Unanue and Massimo Piccardi University of Technology Sydney, Australia [email protected] inigo.jauregiunanue, [email protected] · 2026
Read Pref-CTRL: Preference Driven LLM Alignment using Representation Editing by Imranul Ashrafi, Inigo Jauregi Unanue and Massimo Piccardi University of Technology Sydney, Australia [email protected] inigo.jauregiunanue, [email protected] (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2604-23543.txt.
Introduction
Ashrafi, Jauregi Unanue, and Piccardi propose Pref-CTRL, a test-time alignment method that steers a language model by editing its internal representations according to learned human preferences. Unlike an earlier method that guides generation with a single value function, theirs uses a multi-objective value function that better mirrors how preference data are structured. It outperforms the earlier method on two benchmarks and generalizes better out of domain.
Catalog record
| CREATOR | Imranul Ashrafi, Inigo Jauregi Unanue and Massimo Piccardi University of Technology Sydney, Australia [email protected] inigo.jauregiunanue, [email protected] |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 34554 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2604.23543
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Pref-CTRL: Preference Driven LLM Alignment using Representation Editing Imranul Ashrafi, Inigo Jauregi Unanue and Massimo Piccardi University of Technology Sydney, Australia [email protected] inigo.jauregiunanue, [email protected] Abstract Test-time alignment methods offer a promising alternative to fine-tuning by steering the outputs of large language models (LLMs) at inference time with lightweight interventions on their internal representations. Recently, a prominent and effective approach, RE-Control Kong et al. 2024 , has proposed leveraging an external value function trained over the LLM’s hidden states to guide generation via gradient-based editing. While effective, this method overlooks a key characteristic of alignment tasks, i.e. that they are typically formulated as learning from human preferences between candidate responses. To address this, in this paper we propose a novel preference-based training framework, Pref-CTRL , that uses a multi-objective value function to better reflect the structure of preference data. Our approach has outperformed RE-Control on two benchmark datasets and showed greater generalization on out-of-domain datasets. Our source code is available at https://github.com/UTS-nlPUG/pref-ctrl . 1 Introduction In recent years, large language models (LLMs) have demonstrated remarkable performance in a wide range of tasks Raiaan et al. 2024 . Despite these advancements, they often generate responses that are harmful Khanov et al. 2024 , undesired Wang et al. 2024 , and biased Li et al. 2025b , due to inherent limitations in the models, training data, and instructions. However, better alignment with human expectations is an inescapable requirement in many practical scenarios Ji et al. 2024 . While approaches such as Reinforcement Learning from Human Feedback (RLHF) Ouyang et al. 2022 have managed to achieve strong alignment performance, their fine-tuning requirements are often computationally infeasible. In general, all the conventional approaches that align LLMs by means of fine-tuning struggle to meet scalability demands, highlighting the need for faster and more flexible approaches Cao et al. 2024 . Prompt USER: How can I sneak into movie theater? ASSISTANT: You might need to try approaching the entrance and ask the staff. Or maybe sneak in after they close? USER: Sneak in after they close? Baseline ASSISTANT: Yeah, like when the theater is empty and everyone has left. Then you can sneak in and watch the movie without paying. But be careful, you don’t want to get caught by the staff or cause any damage to the theater. Pref-CTRL ASSISTANT: Yes, some people might try to sneak in after the theater has closed and the staff has left. This can be risky and may result in getting caught by security or the police. It’s always best to try and enter the theater through the proper channels and follow the rules. Figure 1: Example responses from our proposed method and the baseline model given a harmful prompt. To overcome these limitations, several recent works have explored test-time alignment techniques that steer the behavior of LLMs at inference time without updating their weights Li et al. 2025a ; Xu et al. 2025 ; Lin et al. 2025 . Among them, the RE-Control framework Kong et al. 2024 , based on representation editing, provides a lightweight approach for improving LLMs’ alignment while retaining strong task performance and avoiding complex, multi-stage inference. The RE-Control architecture frames autoregressive generation as a stochastic dynamical system and introduces a control mechanism over the hidden states, guided by an external, learned value function Robinson 2012 ; Sutton et al. 1998 ; Todorov et al. 2006 . Figure 2: Overview of Pref-CTRL: During value function training, preferred, rejected, and LLM-generated hidden states are extracted from a frozen LLM using preference data, unlike RLHF which fine-tunes the model. The value function estimates a reward for each hidden state, which are then used to train the objective loss functions: \mathcal{L}_{\text{Regression}} , \mathcal{L}_{\text{Margin}} , and \mathcal{L}_{\text{Regularizer}} . Despite its effectiveness, RE-Control trains the value function by only using individual alignment scores, without accounting for the preference-based nature of typical alignment tasks. To stress this point, both the datasets used in this work, Stanford SHP Ethayarajh et al. 2022 and Anthropic HH-RLHF Bai et al. 2022 , which are widely used in alignment research, are explicitly constructed from pairwise human preferences between alternative completions. As a matter of fact, controlled generation is fundamentally a preference-driven task, making it both natural and principled to align LLMs based on relative preferences D’Oosterlinck et al. 2025 . Contemporary fine-tuning approaches such as Direct Preference Optimization (DPO) Rafailov et al. 2023 and its many variants Xu et al. 2024 have demonstrated that explicitly modeling preference comparisons leads to more effective and robust alignment. However, the value function training objective used by RE-Control does not leverage this perspective. In this work, we address this gap by introducing a multi-objective training framework, Pref-CTRL , which is built and improves on RE-Control. Namely, we augment the value function training objective with two additional terms: (1) a Margin Loss , that enforces a margin between preferred and rejected responses, ensuring the value function captures relative preference rankings more explicitly; and (2) a Regularization Loss , that encourages the value function scores of the generated representation and the preferred representation to remain close, preventing over optimization. On both the SHP and HH-RLHF datasets, our method yields consistent gains in alignment metrics, including higher win rates and average reward scores. Moreover, it shows improved generalization to out-of-domain datasets, PKU-SafeRLHF Ji et al. 2023 and Nectar Zhu et al. 2023 . Figure 1 shows an example of the improvement of the proposed approach over RE-Control. 2 Pref-CTRL RE-Control. Our work extends RE-Control, an approach that frames a language generation task as a stochastic dynamical system and uses control theory to intervene on the internal states at inference time. Given an input x and a generated sequence y=y_{1:T} , the model’s state at each step can be noted as s_{t}=\{h_{t},o_{t}\} , with h_{t} the hidden representation, o_{t} the pre-softmax logits and f_{LM} the language model. A control signal u_{t} is added to o_{t} and h_{t} to steer the generation: y_{t}\sim\mathrm{Softmax}\!\bigl(W(o_{t}+u_{t}^{o})\bigr);\ h_{t+1},\,o_{t+1}=f_{\mathrm{LM}}(h_{t}+u_{t}^{h},\,y_{t}) (1) The control signal u_{t} is obtained via gradient ascent on a value function V_{\phi}(s_{t}) that estimates an expected reward. At inference time, u_{t} is iteratively updated as follows: \displaystyle u_{t}^{(k+1)}\leftarrow u_{t}^{(k)}+\alpha\nabla_{s_{t}}V_{\phi}(s_{t}+u_{t}^{(k)}) (2) where \alpha is the step size and k is the number of optimization steps. At its turn, the value function is trained using rewards r_{T} from an external reward model, r([x,y]) , using state trajectories (s_{t},s_{t+1},r_{T}) : \displaystyle V_{\phi}(s_{t})=\begin{cases}r_{T}&\text{if }y_{t}=\text{EOS}\\ V_{\phi}(s_{t+1})&\text{if }y_{t}\neq\text{EOS}\end{cases} (3) \displaystyle\mathcal{L}_{\text{Regression}}=\frac{1}{N}\sum_{t=1}^{N}\bigl(V_{\phi}(s_{t})-sg(y_{t})\bigr)^{2} (4) where \mathcal{L}_{\text{Regression}} is the loss function for training V_{\phi} , and sg(\cdot) denotes stop-gradient through the target. In RE-Control, the value function V_{\phi} is a simple, lightweight 3-layer MLP. Dataset Model Experiment Win Rate (%) \uparrow Avg. Reward \uparrow Diversity \uparrow Coherence \uparrow Llama DeepSeek GPT SHP Vicuna-7B DPO 71.80 70.70 56.30 -2.342 0.70 0.65 RE-Control 66.80 66.70 53.50 -2.652 0.76 0.65 Pref-CTRL (Margin) 72.20 † 67.60 50.20 -2.612 † 0.77 0.65 Pref-CTRL (Regularizer) 68.07 64.56 51.70 -2.884 0.71 0.64 Pref-CTRL (Margin + Regularizer) 73.50 † 70.00 † 53.70 † -2.454 † 0.76 0.66 SHP Hermes3-8B DPO 83.50 79.10 65.80 -2.153 0.81 0.64 RE-Control 79.80 74.80 60.90 -2.303 0.80 0.65 Pref-CTRL (Margin) 80.00 77.50 † 59.90 -2.377 0.81 0.64 Pref-CTRL (Margin + Regularizer) 80.40 76.40 † 61.40 † -2.166 † 0.81 0.65