Diffusion Blend: Inference-Time Multi-Preference Alignment for Diffusion Models

Min Cheng, Fatemeh Doudi, Dileep Kalathil, Mohammad Ghavamzadeh, Panganamala R. Kumar · 2025

Read Diffusion Blend: Inference-Time Multi-Preference Alignment for Diffusion Models by Min Cheng, Fatemeh Doudi, Dileep Kalathil, Mohammad Ghavamzadeh, Panganamala R. Kumar (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2505-18547.txt.

Introduction

Image generators are often fine-tuned to please one reward, like beauty or text match. Real users juggle several wishes at once, and each person weights them differently. Min Cheng and co-authors tackle that problem at generation time, with no extra training per user. Their method, Diffusion Blend, mixes the generation paths of models tuned for single goals.

By blending those paths, the system can serve any weighted mix of rewards a user names. A second variant lets users dial how far outputs may drift from the base model. Across experiments in 2025, the blends beat rival methods and nearly matched models trained for one exact mix each.

Catalog record

CREATORMin Cheng, Fatemeh Doudi, Dileep Kalathil, Mohammad Ghavamzadeh, Panganamala R. Kumar
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE66 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2505-18547

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Diffusion Blend: Inference-Time Multi-Preference Alignment for Diffusion Models Min Cheng, Fatemeh Doudi, Dileep Kalathil, Mohammad Ghavamzadeh, Panganamala R. Kumar Abstract Reinforcement learning (RL) algorithms have been used recently to align diffusion models with downstream objectives such as aesthetic quality and text-image consistency by fine-tuning them to maximize a single reward function under a fixed KL regularization. However, this approach is inherently restrictive in practice, where alignment must balance multiple, often conflicting objectives. Moreover, user preferences vary across prompts, individuals, and deployment contexts, with varying tolerances for deviation from a pre-trained base model. We address the problem of inference-time multi-preference alignment: given a set of basis reward functions and a reference KL regularization strength, can we design a fine-tuning procedure so that, at inference time, it can generate images aligned with any user-specified linear combination of rewards and regularization, without requiring additional fine-tuning? We propose Diffusion Blend, a novel approach to solve inference-time multi-preference alignment by blending backward diffusion processes associated with fine-tuned models, and we instantiate this approach with two algorithms: DB-MPA for multi-reward alignment and DB-KLA for KL regularization control. Extensive experiments show that Diffusion Blend algorithms consistently outperform relevant baselines and closely match or exceed the performance of individually fine-tuned models, enabling efficient, user-driven alignment at inference-time. The code is available at github.com/bluewoods127/DB-2025 1 Introduction Diffusion models, such as Imagen ( Saharia et al. 2022a ) , DALL·E ( Ramesh et al. 2022 ) , and Stable Diffusion ( Rombach et al. 2022 ) , have demonstrated remarkable capabilities in high-fidelity image synthesis from natural language prompts. However, these models are typically trained on large-scale datasets and are not explicitly optimized for downstream objectives such as semantic alignment, aesthetic quality, or user preference. To address this gap, recent works have proposed reinforcement learning (RL) for aligning diffusion models with task-specific reward functions ( Uehara et al. 2024a ; Fan et al. 2023 ; Black et al. 2024 ) , where the core idea is to fine-tune a pre-trained model to maximize a reward, while constraining the update to remain close to the original model via a Kullback–Leibler (KL) regularization. The KL regularization term prevents reward overoptimization (reward hacking), and preserves desirable properties of the pre-trained model ( Ouyang et al. 2022 ; Rafailov et al. 2023 ) such as sample diversity and visual fidelity ( Fan et al. 2023 ; Uehara et al. 2024b ) . Figure 1 : (a). Overview of our Diffusion Blend-Multi-Preference Alignment (DB-MPA) Algorithm. Given two basis reward functions, such as text-image alignment score and aesthetic score, for any preference weight w=(w_{1},w_{2}) specified by the user at the inference time, DB-MPA generates images aligned with reward r(w)=w_{1}r_{1}+w_{2}r_{2} . (b) During the fine-tuning stage, DB-MPA gets an RL fine-tuned model corresponding to each reward function. (c) During the inference time, DB-MPA blends (mixes) the backward diffusion corresponding to each fine-tuned model according to the user-specified preference w . While RL fine-tuning has improved alignment in diffusion models, it typically assumes a fixed reward function and regularization weight. This assumption is restrictive in practice, where alignment must balance multiple, often conflicting objectives, such as aesthetics and prompt fidelity, and user preferences vary across prompts, individuals, and deployment contexts. Static fine-tuning with fixed reward combinations cannot accommodate this variability without retraining separate models for each configuration ( Wang et al. 2024b ; Rame et al. 2023 ; Lee et al. 2024 ) . Moreover, once trained, the trade-offs are fixed, precluding post-hoc adjustment. Similar issues arise with KL regularization: insufficient regularization causes reward hacking, while excessive regularization impedes alignment ( Uehara et al. 2024b ; Liu et al. 2024 ) . In practice, both reward and regularization weights are tuned via grid search, incurring significant computational cost and limiting flexibility. These limitations motivate the need for a more flexible approach: inference-time multi-preference alignment , where the user specifies their preference vector, i.e., weights over a set of basis reward functions such as alignment, aesthetics, or human preference, along with a desired regularization strength that controls deviation from the pre-trained model. Crucially, this alignment must occur without any additional fine-tuning or extensive computation at inference time, which is essential for real-time and resource-constrained settings. Unlike trial-and-error prompt tuning, the ideal solution should offer a principled and computationally efficient solution that can achieve Pareto-optimal trade-offs across multiple preferences. This motivates us to address the following questions: Given a set of basis reward functions (r_{i})^{m}_{i=1} and a basis KL regularization weight \alpha , can we design a fine-tuning procedure such that when the user specifies their reward or regularization preferences through parameters w and \lambda at inference time, the model generates images aligned with the linear reward combination r(w)=\sum^{m}_{i=1}w_{i}r_{i} and regularization weight \alpha(\lambda)=\alpha/\lambda , without requiring additional fine-tuning? In this work, we answer this question affirmatively and provide constructive solutions to it. Our main contributions are the following: • We theoretically show that the backward diffusion process corresponding to the diffusion model aligned with reward function r(w) and regularization weight \alpha can be expressed as the backward diffusion process corresponding to the pre-trained model with an additional control term that depends on r(w) . We propose an approximation result for this control term, which enables us to express it using the control terms corresponding to fine-tuned diffusion models for the basis reward functions (r_{i})^{m}_{i=1} . We also obtain a similar approximation result corresponding to the regularization weight \alpha . • Leveraging the theoretical results we developed, we propose Diffusion Blend- Multi-Preference Alignment (DB-MPA) algorithm, a novel approach that will blend the backward diffusion processes corresponding to the basis reward functions appropriately to synthesize a new backward diffusion process during inference-time that will generate images aligned with the reward r(w) , where w is specified by the user during the inference-time. Using the same approach, we also propose Diffusion Blend-KL Alignment (DB-KLA) algorithm that will generate images aligned with a reward function r and regularization weight \alpha/\lambda , where \lambda is specified by the user during the inference time. • We provide extensive experimental evaluations using the Stable Diffusion ( Rombach et al. 2022 ) baseline model, multiple basis reward functions, regularization weights, standard prompt sets, and demonstrate that our diffusion blend algorithms outperform multiple relevant baseline algorithms, and often achieve a performance close to the empirical upper bound obtained by an individually fine-tuned model for specific w and \lambda . Figure 2 : Comparison of the DB-MPA algorithm with relevant baselines: Stable Diffusion v1.5 ( Rombach et al. 2022 ) , CoDe ( Singh et al. 2025 ) , reward gradient-based guidance (RGG) ( Chung et al. 2023 ) , rewarded soup (RS) ( Rame et al. 2023 ) , and Multi-Objective RL (MORL) ( Roijers et al. 2013 ) . Note that MORL is included only to illustrate the maximum achievable performance by an oracle algorithm. See Section 2 for more details about the baselines. For any preference weight w specified by a user at inference-time, the goal of these algorithms is to generate images aligned with reward r(w)=wr_{1}+(1-w)r_{2} , where r_{1} is the text-to-image alignment score and r_{2} is the aesthetics score. (a) Visual comparison of the images generated for the prompt ‘a blue colored apple’ for w\in\{0.2,0.5,0.8\}

Read the complete text