Beyond Pairwise Preferences: Listwise Reward-Aware Alignment for Diffusion Models
Austin Wang, Jiaqi Han, Stefano Ermon, Yisong Yue · 2026
Read Beyond Pairwise Preferences: Listwise Reward-Aware Alignment for Diffusion Models by Austin Wang, Jiaqi Han, Stefano Ermon, Yisong Yue (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2605-26491.txt.
Introduction
Austin Wang and three colleagues move diffusion alignment past pairwise comparisons with Diffusion LAIR, a listwise reward method. Pairwise reduction wastes data when one prompt yields many images and scores beat a winner-loser label. LAIR turns scores across candidate groups into centered weights. It optimizes a weighted regression on implicit reward, the denoising-loss gain over a reference, with a quadratic penalty on its size. The objective uses all candidates at once with a bounded closed-form optimum. Tests on SD1.5 and SDXL beat baselines across generation, composition, editing.
Catalog record
| CREATOR | Austin Wang, Jiaqi Han, Stefano Ermon, Yisong Yue |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 45322 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2605.26491
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Beyond Pairwise Preferences: Listwise Reward-Aware Alignment for Diffusion Models Austin Wang, Jiaqi Han, Stefano Ermon, Yisong Yue Abstract Preference optimization has emerged as an efficient alternative to online reinforcement learning from human feedback (RLHF) for aligning text-to-image diffusion models. However, existing methods largely reduce supervision to binary pairwise comparisons. This pairwise reduction is limiting when training data naturally contains multiple candidate images for the same prompt, and when continuous reward scores can provide richer information than a single winner–loser label. To address these limitations, we propose Diffusion LAIR, a reward-aware listwise preference optimization method for diffusion models. For each prompt, LAIR converts reward scores across a group of candidate images into centered advantage weights, then optimizes an advantage-weighted regression objective on the implicit reward, defined as the denoising-loss improvement of the current model over a fixed reference model, with a quadratic penalty that regularizes the magnitude of the implicit reward. The resulting objective uses all candidates simultaneously rather than selecting pairs, and remains conservative by explicitly controlling the magnitude of the implicit reward. The LAIR objective admits a bounded closed-form optimum in implicit-reward space, clarifying how the regularization strength controls the magnitude of the preference update. Experiments show that Diffusion LAIR outperforms strong preference optimization baselines on SD1.5 and SDXL across text-to-image generation, compositional generation, and image editing benchmarks. 1 Introduction Figure 1 : Sample images generated from SDXL trained with Diffusion LAIR. Text-to-image diffusion models, such as Stable Diffusion ( 26 ; 31 ) and Imagen ( 32 ) , have become a standard backbone for high-quality image generation. Large-scale pretraining, however, does not by itself guarantee alignment with human preferences. Post-training is therefore needed to improve aspects like aesthetics, prompt following, and overall sample quality. As a result, diffusion preference optimization has emerged as a practical approach to this problem by adapting preference-learning objectives to denoising diffusion models ( 37 ; 46 ; 19 ; 13 ; 11 ; 18 ) . Although online reinforcement learning algorithms for diffusion alignment ( 9 ; 3 ; 8 ; 21 ; 45 ; 6 ) have been extensively explored, their on-policy, sequential nature tightly couples trajectory generation, reward evaluation, and policy updates, making training expensive, difficult to parallelize, and sensitive to distribution shift. Offline preference optimization has therefore emerged as a compelling alternative, learning directly from pre-collected human preference data, generated and processed independently of the training loop. Despite these advantages, most diffusion preference objectives retain the pairwise structure of direct preference optimization (DPO), where each update compares a preferred image against a dispreferred image for the same prompt. This pairwise formulation is a poor match for many practical preference datasets: a prompt often has more than two candidate images, and modern reward models can assign continuous scores to each candidate. Reducing this information to selected winner-loser pairs discards the relative quality of the remaining candidates and ignores the magnitude of reward gaps. Recent approaches ( 17 ; 20 ) use vision-language reward models to construct richer preference signals, but still rely on pair selection heuristics to develop pairwise objectives. To address this limitation, we propose Diffusion LAIR (Listwise Advantage-weighted Implicit Reward), which treats diffusion preference optimization as a listwise, reward-aware learning problem. For each prompt, Diffusion LAIR converts reward scores across a group of candidate images into centered advantage weights, which promote high-reward samples and suppress low-reward samples relative to the group. Diffusion LAIR then optimizes an advantage-weighted implicit reward, defined by the denoising-loss improvement of the current model over a fixed reference model, with a quadratic penalty that keeps the update conservative. This construction preserves the reference-based structure of diffusion preference optimization while distributing learning signal across all candidates rather than selected comparison pairs. The resulting objective admits a closed-form optimum in implicit-reward space, where each sample receives a finite target proportional to its centered advantage weight, clarifying how regularization controls the size of the preference update. Our contributions • We propose Diffusion LAIR, a preference optimization objective that learns directly from groups of reward-scored images, rather than reducing supervision to binary preference pairs. • We show that the Diffusion LAIR objective admits a bounded closed-form optimum in implicit-reward space, providing insight into how regularization controls preference updates and induced distribution shift. • We demonstrate strong empirical performance across text-to-image generation, compositional generation, and instruction-based image editing on SD1.5 and SDXL. 2 Related Work 2.1 Reinforcement Learning from Human Feedback A dominant paradigm in training foundation models is to first pre-train on massive web-scale datasets, then post-train using RL techniques to further enhance the model on domain-specific tasks. Reinforcement learning from human feedback (RLHF) has been shown to elicit powerful capabilities from large language models and align them with human preferences ( 7 ; 35 ; 25 ; 47 ; 2 ) . However, these techniques have been relatively less explored for diffusion models. Some works directly backpropagate reward gradients through the diffusion sampling process ( 8 ; 27 ; 30 ) . While effective at improving reward scores, these methods suffer from high computational expense and require the reward model itself to be differentiable. A different line of research adapts RL techniques to diffusion models ( 3 ; 9 ; 21 ; 41 ; 45 ; 40 ; 6 ; 11 ; 43 ) . Despite their impressive performance, these RL methods typically require sampling full denoising trajectories during training and often are prone to reward hacking and training instability. 2.2 Offline Preference Alignment for Diffusion Models Offline preference optimization has become an attractive alternative to traditional RLHF techniques. Crucially, these methods only require preference data, rather than full denoising trajectories, making them significantly more efficient than online RL algorithms. ( 37 ; 42 ) are pioneer works that extend DPO ( 29 ) to diffusion models, learning directly from human preference data. More recent state-of-the-art works include DSPO ( 46 ) , which integrates DPO with denoising score matching; Diffusion KTO ( 19 ) , which formulates alignment as utility maximization from per-sample binary feedback; and InPO ( 22 ) , which improves efficiency by using reparameterized DDIM inversion to identify and optimize latent variables that are most relevant to preference alignment. Nevertheless, most offline diffusion alignment methods ultimately reduce supervision to pairwise comparisons with binary preference labels, missing out on finer-grained signals like listwise rankings and reward scores. Recent works ( 15 ; 17 ) incorporate reward signals to construct rankings or calibrate preferences, but their final objectives still reduce learning to pairwise comparisons between samples. This pairwise reduction discards listwise structure: for candidates sharing the same prompt, the full ordering and reward spacing provide richer supervision than isolated binary preferences and more faithfully capture variation in sample quality within the group. Most closely related to our work, Diffusion LPO ( 1 ) preserves list-level structure through a Plackett-Luce objective over ranked image lists, but primarily uses ordinal preferences. In contrast, our method directly leverages continuous reward scores, capturing both relative ordering and cardinal quality gaps within each candidate group. 3 Background and Preliminaries 3.1 Diffusion Models Denoising diffusion models ( 12 ; 34 ; 14 ) are generative models that enable sampling from a data distribution q({\bm{x}}_{0}) . Given a noise schedule \alpha_{t} and