On-the-fly Preference Alignment via Principle-Guided Decoding

Mingye Zhu, Yi Liu, Lei Zhang, Junbo Guo, Zhendong Mao · 2025

Read On-the-fly Preference Alignment via Principle-Guided Decoding by Mingye Zhu, Yi Liu, Lei Zhang, Junbo Guo, Zhendong Mao (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2502-14204.txt.

Introduction

Mingye Zhu and four co-authors align language model outputs with human preferences at decoding time, with no retraining, in a paper from 2025. Popular alignment needs heavy compute and vast preference data, which limits its reach. Diverse values make that data burden worse.

Their method first builds a stand-in solution to an otherwise infeasible tuning problem. A principle-guided reward is then defined from the gap between a constrained and a free policy. Maximizing that reward during generation steers each output toward the principles. Tests on general and personal alignment match or beat strong baselines at a fraction of the cost.

Catalog record

CREATORMingye Zhu, Yi Liu, Lei Zhang, Junbo Guo, Zhendong Mao
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE79 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2502-14204

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

On-the-fly Preference Alignment via Principle-Guided Decoding Mingye Zhu, Yi Liu, Lei Zhang, Junbo Guo, Zhendong Mao Abstract With the rapidly expanding landscape of large language models, aligning model generations with human values and preferences is becoming increasingly important. Popular alignment methods, such as Reinforcement Learning from Human Feedback, have shown significant success in guiding models with greater control. However, these methods require considerable computational resources, which is inefficient, and substantial collection of training data to accommodate the diverse and pluralistic nature of human preferences, which is impractical. These limitations significantly constrain the scope and efficacy of both task-specific and general preference alignment methods. In this work, we introduce O n-the-fly P reference A lignment via Principle-Guided D ecoding (OPAD) to directly align model outputs with human preferences during inference, eliminating the need for fine-tuning. Our approach involves first curating a surrogate solution to an otherwise infeasible optimization problem and then designing a principle-guided reward function based on this surrogate. The final aligned policy is derived by maximizing this customized reward, which exploits the discrepancy between the constrained policy and its unconstrained counterpart. OPAD directly modifies the model’s predictions during inference, ensuring principle adherence without incurring the computational overhead of retraining or fine-tuning. Experiments show that OPAD achieves competitive or superior performance in both general and personalized alignment tasks, demonstrating its efficiency and effectiveness compared to state-of-the-art baselines. 1 1 1 Code can be found at: https://github.com/stevie1023/OPAD.git. 1 Introduction Figure 1: Given a query and principle, OPAD offered a more poetic and eloquent response (befitting a charismatic poet), whereas prompting with the principle presents a direct answer, failing to follow the principle to act as a poet. As tremendous strides have been made in the development of large language models (LLMs), it remains challenging to align these models with specific principles, such as ethical guidelines or factual consistency, during generation. Popular alignment methods focus primarily on training-time optimization, such as Reinforcement Learning from Human Feedback (RLHF) ( Ouyang et al., 2022 ; Stiennon et al., 2020 ) and Direct Preference Optimization (DPO) ( Rafailov et al., 2023 ) . While these techniques significantly improve the alignment of model outputs, they still face certain limitations ( Lin et al., 2023 ) . RLHF, for instance, is sensitive to hyperparameters and is complex to train ( Casper et al., 2023 ) . DPO, on the other hand, introduces a new parameterization for the RLHF objective that simplifies and stabilizes the training process, but its performance is highly dependent on the quality of the preference pairs used ( Pal et al., 2024 ) . Despite their effectiveness, these alignment methods can be inefficient —requiring substantial computational resources—and impractical , given the pluralistic nature of human preferences. User preferences vary widely across different topics ( Cheng et al., 2023 ) , making it infeasible to curate data or train multiple models to handle customized or personalized applications in a single training phase. This limitation motivates the development of inference-time algorithms that can achieve efficient and on-the-fly alignment. Figure 2: OPAD overview . Given user query \mathbf{x} and principle c , OPAD computes a principle-guided reward r_{\theta}(\mathbf{x},\mathbf{y}_{<t},c) leveraging the divergence between the constrained probability distribution and its unconstrained counterpart. This reward quantifies the alignment between the current prediction and the principle c , and the final aligned policy p_{\theta} is derived by maximizing this reward. Additionally, the superficial alignment hypothesis ( Zhou et al., 2023 ) suggests that a model’s knowledge and capabilities are largely acquired during pretraining, while alignment tuning mostly affects its format and style during interactions with users. This hypothesis is further supported by Lin et al. (2023) , who demonstrated that the base and aligned LLMs perform almost identically in most positions in terms of ranking tokens during decoding , with both quantitative and qualitative analyses. This observation motivates us to rethink the potential of tuning-free alignment approaches. In-context learning, as an inference-time algorithm, has demonstrated effectiveness in enforcing specific guidelines but still fall short on specification-heavy tasks ( Peng et al., 2023 ) and predicting highly divergent or complex contexts ( Kossen et al., 2024 ) . The Best-of- N approach ( Nakano et al., 2021 ) uses a well-trained reward model to select the best response from multiple outputs generated by the model, significantly improving the quality of the final output, albeit at the cost of increased inference time. A more recent and advanced method is Linear Alignment ( Gao et al., 2024 ) , which approximates the Q -function using a first-order Taylor expansion combined with self-contrastive decoding. This policy directly estimates the gradient of the value function by extracting principle prompts that perturb the output policy. However, because this method relies on a first-order gradient approximation, the principle prompts need to be ”subtle” enough to avoid inducing large perturbations, which could lead to poor gradient estimates and suboptimal performance. Existing tuning-free alignment approaches have shown promise, but they still have limitations. Previous works reveal that prompting with guidance can improve task-specific performance, but LLMs often rely on recognizing semantic patterns rather than genuinely understanding the principles embedded in the prompts ( Webson & Pavlick, 2021 ) . This motivates us to explore a deeper, conceptual-level alignment component that arises from principle-based prompting. In this work, we introduce O n-the-fly P reference A lignment via Principle-Guided D ecoding (OPAD), with the overview in Figure 2 . OPAD identifies the incremental alignment improvements that occur when the model responds to principles, treating this as a residual alignment signal that can be optimized as a reward. Specifically, we conceptualize this residual alignment as the divergence between the model’s behavior when constrained by principles and when left unconstrained. Rather than directly tackling the intractable problem of minimizing the Kullback-Leibler (KL) divergence between a constrained policy and the ground truth preference data, we introduce a surrogate objective that maximizes the KL divergence between the constrained policy and its unconstrained counterpart. This approach allows us to translate the residual alignment into a reward function that quantifies the model’s adherence to the target principles. The final aligned policy is derived by maximizing this pre-defined reward, yielding a tuning-free solution that adjusts subsequent token predictions to promote adherence to the target principle at each time step t . OPAD introduces a complementary adjustment mechanism that refines the presentation and expression of the model’s output, aligning it more closely with the target principles while preserving the model’s core knowledge. This allows for more controlled and principled responses without altering the model’s foundational capabilities during inference. Experiments demonstrate that OPAD achieves competitive or superior alignment on both general alignment tasks (e.g., dialogue and summarization) and personalized alignment tasks (e.g., specific principle-driven tasks) while showing decent performances in other automatic evaluation metrics such as perplexity, diversity, and ROUGE scores. Moreover, we show that OPAD-induced policy inherits a larger distribution shift from the base policy compared to traditionally RLHF-aligned models, indicating that OPAD is more adept at modulating the model’s behavior to better reflect the target principles. 2 Related work 2.1 traditional preference alignment Among the wide range of algorithms proposed for preference alignment ( Stiennon et al., 2020 ; Yuan et al., 2023 ; Rafailov et al., 2023 ;

Read the complete text