DiffPO : Diffusion-styled Preference Optimization for Efficient Inference-Time Alignment of Large Language Models

Ruizhe Chen, Wenhao Chai, Zhifei Yang, Xiaotian Zhang, Joey Tianyi Zhou, Tony Quek, Soujanya Poria, Zuozhu Liu · 2025

Read DiffPO : Diffusion-styled Preference Optimization for Efficient Inference-Time Alignment of Large Language Models by Ruizhe Chen, Wenhao Chai, Zhifei Yang, Xiaotian Zhang, Joey Tianyi Zhou, Tony Quek, Soujanya Poria, Zuozhu Liu (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2503-04240.txt.

Introduction

Eight researchers present a method for aligning language models while they generate text in a paper from 2025. Alignment means steering outputs toward what humans prefer. Doing it during generation avoids retraining, but older methods are slow or tied to one model. Their method, DiffPO, borrows the style of diffusion models and works at the sentence level.

Working on whole sentences avoids the delay of token-by-token scoring. The module plugs into many base models without changes to them. Tests on three standard benchmarks showed better alignment at lower delay. Gains held even for a very large model. The authors offer the method as a flexible add-on for deployed systems.

Catalog record

CREATORRuizhe Chen, Wenhao Chai, Zhifei Yang, Xiaotian Zhang, Joey Tianyi Zhou, Tony Quek, Soujanya Poria, Zuozhu Liu
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE44 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2503-04240

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

DiffPO : Diffusion-styled Preference Optimization for Efficient Inference-Time Alignment of Large Language Models Ruizhe Chen, Wenhao Chai, Zhifei Yang, Xiaotian Zhang, Joey Tianyi Zhou, Tony Quek, Soujanya Poria, Zuozhu Liu Abstract Inference-time alignment provides an efficient alternative for aligning LLMs with humans. However, these approaches still face challenges, such as limited scalability due to policy-specific value functions and latency during the inference phase. In this paper, we propose a novel approach, Diffusion-styled Preference Optimization ( DiffPO ), which provides an efficient and policy-agnostic solution for aligning LLMs with humans. By directly performing alignment at sentence level, DiffPO avoids the time latency associated with token-level generation. Designed as a plug-and-play module, DiffPO can be seamlessly integrated with various base models to enhance their alignment. Extensive experiments on AlpacaEval 2, MT-bench, and HH-RLHF demonstrate that DiffPO achieves superior alignment performance across various settings, achieving a favorable trade-off between alignment quality and inference-time latency. Furthermore, DiffPO demonstrates model-agnostic scalability, significantly improving the performance of large models such as Llama-3-70B. Our model and code are available here . 1 Introduction The alignment of large language models (LLMs) with human preferences has recently emerged as a focal area of research Wang et al. 2023 ; Shen et al. 2023 . Prominent techniques such as Reinforcement Learning from Human Feedback (RLHF) Ouyang et al. 2022 and Direct Preference Optimization (DPO) Rafailov et al. 2024 have demonstrated substantial efficacy. However, these methods require the optimization of individual policies, posing challenges such as high consumption of training resources. Inference-time alignment Mudgal et al. 2023 ; Han et al. 2024 provides an efficient alternative through direct adjustment of the model’s output distribution, thus avoiding the need for resource-intensive retraining. Despite its advantages, this approach still requires policy-specific value functions, limiting its scalability across different models. Additionally, the inference-time latency remains high, presenting further challenges to its practical deployment. Figure 1: Comparison with Inference-Time Methods. Points closer to the top-right indicate a superior trade-off between performance and inference time. Figure 2: Illustration of the DiffPO Framework. (a) The objective of LLM alignment is to adjust the output of LLMs to reflect human values and intentions. In this process, preferences are considered at the sentence level , focusing on aspects such as the style and format of the complete output. (b) We propose Diffusion-style Preference Optimization ( DiffPO ), which reconceptualizes the alignment process as a sentence-level denoising process, where the goal is to transform an unaligned sentence \mathbf{y}^{(0)} into an aligned sentence \mathbf{y}^{(T)} step by step. (c) Designed as a plug-and-play module, DiffPO can be directly integrated with the base model output and yield better alignment. In this paper, we investigate an efficient and policy-agnostic preference optimization method. We begin by reconsidering the objective of aligning with humans Yao et al. 2023 ; Shen et al. 2023 . As illustrated in Fig. 2 (a), the alignment process operates at the sentence level, focusing on adjusting key components of the generated content, such as style or format, to better reflect human intentions or values. Inspired by the global controllability of the diffusion process Li et al. 2022 ; Lyu et al. 2023 , we propose Diffusion-styled Preference Optimization ( DiffPO ). DiffPO draws an analogy from the diffusion-based denoising process to model the iterative adjustment required for aligning human preferences, as shown in Fig. 2 (b). By employing parallel decoding Santilli et al. 2023 ; Leviathan et al. 2023 , DiffPO directly predicts sentence-level transitions, thus avoiding the time latency associated with token-level generation. During the training phase, we optimize the DiffPO with an objective that maps generations with varying alignment levels to an aligned target, making it a policy-agnostic, plug-and-play module. The optimized DiffPO can then be seamlessly integrated with the output of the base model, enhancing its alignment level, as demonstrated in Fig. 2 (c). We evaluate the performance of DiffPO on several benchmark datasets, including AlpacaEval 2 Dubois et al. 2024 , MT-bench Zheng et al. 2023 , and HH-RLHF Bai et al. 2022 . Empirical results demonstrate that DiffPO achieves superior alignment performance across various base models and settings. Compared to inference-time alignment techniques, DiffPO strikes an optimal trade-off between alignment performance and inference-time latency, as shown in Fig. 1 . Additional experiments highlight the model-agnostic scalability of DiffPO across different base models. Specifically, DiffPO -9B significantly enhances the performance of models such as Llama-3-70B and GPT-4o, showcasing its capability to improve weak-to-strong supervision. The advantages of DiffPO can be summarized as: • Model-agnostic. DiffPO is optimized to learn sentence-level refinement, independent of the specific base LLMs. This allows it to be applied across a variety of base LLMs. Furthermore, DiffPO does not require access to model parameters, which enhances its compatibility with API-based models and existing preference-aligned models. • Training and Inference Efficiency. As a post-inference alignment strategy, DiffPO adopts a one-for-all approach: it involves training one single DiffPO and applying it for all base models, thus significantly reducing the resource intensiveness associated with policy optimization. Moreover, by framing alignment as sentence-level prediction, DiffPO bypasses the time latency associated with token-level generation, thereby improving inference-time efficiency. 2 Method 2.1 Preliminaries: Large Language Models Next-Token Prediction. The text generation of autoregressive large language models (LLMs) with prompt \mathbf{x} and response \mathbf{y} can be modelled as a next-token prediction process. Given the input \mathbf{x} , The language model \pi(\cdot|\mathbf{x}) autoregressively maps from current tokens (\mathbf{x},\mathbf{y}_{1:n-1}) to a distribution over the next token \mathbf{y}_{n} . The maximum token, N , sets the length limit for LLM outputs, which conclude with an end-of-sentence (EoS) token \mathbf{y}_{N}= EoS that ends the generation. The generated output \mathbf{y} consists of predicted tokens (\mathbf{y}_{1},\mathbf{y}_{2},...,\mathbf{y}_{N}) . Alignment of LLMs. During the alignment of LLMs, the objective is to optimize a language model \pi_{\theta} that maximizes the user’s preference Christiano et al. 2017 ; Ouyang et al. 2022 ; Rafailov et al. 2024 : \displaystyle\max_{\pi_{\theta}}\mathbb{E}_{\begin{subarray}{c}x\sim D,\mathbf{y}\sim\pi_{\theta}(\mathbf{y}|\mathbf{x})\\ \mathbf{y}^{\prime}\sim\pi_{\mathrm{ref}}(\mathbf{y}|\mathbf{x})\end{subarray}} \displaystyle[p(\mathbf{y}\succ\mathbf{y}^{\prime}|\mathbf{x}) \displaystyle-\beta D_{KL}(\pi_{\theta}\|\pi_{\mathrm{ref}})], (1) where p(\mathbf{y}\succ\mathbf{y}^{\prime}|\mathbf{x}) represents the preference, i.e., the probability that \mathbf{y} is preferred over \mathbf{y}^{\prime} given the context \mathbf{x} , which can be generally represented by the reward function r . The parameter \beta controls the deviation from the reference policy \pi_{\mathrm{ref}} , which generally corresponds to the SFT model. Parallel Decoding of LLMs. In comparison to next-token prediction, where token-level generation is performed sequentially to obtain a sentence, parallel decoding has demonstrated the capacity by enabling sentence-level generation and improving content quality Santilli et al. 2023 ; Leviathan et al. 2023 . Concretely, supposing f(\mathbf{y}_{n},\mathbf{y}_{<n},\mathbf{x}):=\mathbf{y}_{n}-\arg\max_{\mathbf{y}}\pi(\mathbf{y}|\mathbf{y}_{<n},\mathbf{x}), parallel decoding re-frames the LLM inference process as solving a system of nonlinear equations w.r.t. all tokens in a sentence \mathbf{y}_{n} for n=1,\ldots,N . It can be solved in a parallel and iterative way: \left\{\begin{aligned} \mathbf{y}_{1}^{(t+1)}&=\arg\max_{\mathbf{y}}\pi(\mathbf{y}\mid\mathbf{x})\\ \mathbf{y}_{2}^{(t+1)}&=\arg\max_{\mathbf{y}}\pi(\mathbf{y}\mid\mathbf{y}_{1}^{(t)},\mathbf{x})\\ &\vdots\\ \mathbf{y}_{N}^{(t+1)}&=\arg\max_{\mathbf{y}}\pi(\mathbf{y}\mid\mathbf{y}_{<N}^{(t)},\mathbf{x})\end{aligned}\right. (2) In this way, for one forward pass of the LLM at time t , we can obtain the next sentence \mathbf{y}^{(t+1)} based on the previous one \mathbf{y}^{(t)} . 2.2 Diffusion-styled Preference Optimization Motivation. The goal of LLM alignment is to align the outputs of LLMs with human values or intentions Yao et al. 2023 . In this process, preferences are defined at the sentence-level , focusing on the style or format of complete generated answers, as illustrated in Fig. 2

Read the complete text