D-Fusion: Direct Preference Optimization for Aligning Diffusion Models with Visually Consistent Samples

Zijing Hu, Fengda Zhang, Kun Kuang · 2025

Read D-Fusion: Direct Preference Optimization for Aligning Diffusion Models with Visually Consistent Samples by Zijing Hu, Fengda Zhang, Kun Kuang (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2505-22002.txt.

Introduction

Zijing Hu, Fengda Zhang, and Kun Kuang improve how image generators learn from human taste, in a 2025 study. Diffusion models often draw pictures that miss the text prompt. A method called direct preference optimization can tune them, but it stumbles when good and bad samples look too unlike. The model then cannot tell which traits made one image better. Their fix, D-Fusion, builds training pairs that stay visually close.

Masks guide a fusion step that blends self-attention so the good image keeps the bad one's look while fixing the prompt fit. The method also keeps the noise path that training needs. Tests across several learning methods showed firmer prompt alignment.

Catalog record

CREATORZijing Hu, Fengda Zhang, Kun Kuang
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE60 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2505-22002

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

D-Fusion: Direct Preference Optimization for Aligning Diffusion Models with Visually Consistent Samples Zijing Hu, Fengda Zhang, Kun Kuang Abstract The practical applications of diffusion models have been limited by the misalignment between generated images and corresponding text prompts. Recent studies have introduced direct preference optimization (DPO) to enhance the alignment of these models. However, the effectiveness of DPO is constrained by the issue of visual inconsistency, where the significant visual disparity between well-aligned and poorly-aligned images prevents diffusion models from identifying which factors contribute positively to alignment during fine-tuning. To address this issue, this paper introduces D-Fusion, a method to construct DPO-trainable visually consistent samples. On one hand, by performing mask-guided self-attention fusion, the resulting images are not only well-aligned, but also visually consistent with given poorly-aligned images. On the other hand, D-Fusion can retain the denoising trajectories of the resulting images, which are essential for DPO training. Extensive experiments demonstrate the effectiveness of D-Fusion in improving prompt-image alignment when applied to different reinforcement learning algorithms. † † affiliationnotice: Equal contribution 1 Introduction Figure 1: (Misalignment) Diffusion models ( e.g. , Stable Diffusion (SD) ( Rombach et al., 2022 ) ) often encounter the issue that the generated images do not accurately match the given prompts. Existing RL-based fine-tuning methods ( e.g. , DPO ( Wallace et al., 2023 ) ) have limited effectiveness in improving the alignment. For each set of images above, we use the same seed for sampling. Figure 2: (Visual Inconsistency) When people train diffusion models with direct preference optimization (DPO), the visual disparity between well-aligned and poorly-aligned images are enormous. This visual inconsistency limits the success of DPO in enhancing diffusion models. Meanwhile, the visually consistent samples obtained through manual editing lack denoising trajectories and are not suitable for RL training. To this end, we introduce D-Fusion, which constructs RL-trainable visually consistent samples. Diffusion models have made remarkable success in various domains, such as medicine ( Xu et al., 2022 ) , robotics ( Chi et al., 2024 ) , and 3D synthesis ( Poole et al., 2022 ) . Recently, the application of diffusion models in the field of text-to-image generation has gained widespread attention ( Ho et al., 2020 ; Dhariwal & Nichol, 2021 ) . Under the guidance of the given text descriptions, usually called prompts , these models transform random noises to corresponding images via a sequential denoising process. However, as shown in Figure 1 , diffusion models often encounter the issue of prompt-image misalignment ( Jiang et al., 2024 ; Mrini et al., 2024 ) . Prompt-image misalignment refers to the problem that the generated images do not accurately match the given text prompts, which limits the real-world applications of diffusion models. To address this issue, recent studies have explored incorporating reinforcement learning (RL) algorithms to fine-tune pre-trained diffusion models ( Black et al., 2024 ; Clark et al., 2024 ; Fan et al., 2023 ; Wallace et al., 2023 ; Xu et al., 2023 ; Yang et al., 2024a ; Yang et al., 2024b ; Hu et al., 2025 ) . In the paradigm of RL, the step-by-step denoising process of diffusion models is reinterpreted as a sequential decision-making problem . In this formulation, the intermediate noisy image at each timestep is regarded as a state , while each denoising operation corresponds to an action . Among these RL algorithms, direct preference optimization (DPO) stands out for its advantage of eliminating the need for an explicit reward model , making it a widely adopted approach ( Wallace et al., 2023 ; Yang et al., 2024a ) . As illustrated in Figure 2 (a), researchers first sample images from the diffusion model with given prompts, and then evaluate the alignment between images and prompts via human preference or model prediction. These sampled images, along with their preference orders and denoising trajectories, can be further used in DPO to enhance the alignment of diffusion model. However, DPO has so far achieved limited success in improving prompt-image alignment, primarily due to the issue of visual inconsistency in the training data. Visual inconsistency refers to the disparity between images in terms of structure, style or appearance, which is commonly observed in the images denoised from different noises. As shown in Figure 2 (a), high-preference images ( i.e. , well-aligned images) differ from low-preference images ( i.e. , poorly-aligned images) not only in the alignment-related factors, but also in unrelated factors ( e.g. , background). Interfered by unrelated factors, it is difficult for the model to identify which factors contribute positively to alignment. Meanwhile, with great differences, it is difficult to tell which image aligns better sometimes ( e.g. , an image with only monkey and an image with only sandwich). As a result, learning effective denoising policies from those samples becomes challenging. We believe that performing DPO with visually consistent image pairs can help diffusion models learn effective policies. Recent studies have corroborated similar perspectives on RL training of large language models (LLMs) and multimodal large language models (MLLMs) ( Kong et al., 2025 ; Yu et al., 2024 ) . As illustrated in Figure 2 (b), these studies make fine-grained editing or annotations to the hallucinations ( Huang et al., 2024 ; Bai et al., 2024 ) present in the text output of the language model, thereby obtaining factual training data ( i.e. , high-preference data) with consistent linguistic style. Since the editing or annotations to text output are at the token-level , and the decision-making sequence in language model is constructed token-by-token ( Rafailov et al., 2024 ) , RL training can still proceed. By performing DPO with the pairs consisting of hallucinated text and corresponding factual text, the language model receives dense reward signals and achieves fine-grained alignment. However, these methods on language models fail when applied to the text-to-image diffusion models. As illustrated in Figure 2 (b), the decision-making sequence of the diffusion model is constructed timestep-by-timestep . Existing editing methods, such as manual editing, Imagen Editor ( Wang et al., 2023 ) and Imagic ( Kawar et al., 2023 ) , are capable of both aligning images and maintaining visual consistency. Yet, these methods perform editing at the pixel-level , causing the loss of the timestep-by-timestep decision-making sequences. Once edited to better align with the prompts, these images lack corresponding denoising trajectories , making them unsuitable for RL fine-tuning. This motivates us to ask: How can we generate RL/DPO-trainable visually consistent image pairs to fine-tune diffusion models? In this paper, we address this challenge by introducing D-Fusion: self-attention based D enoising trajectory Fusion , a method to construct RL-trainable visually consistent images. Our method offers key innovations in two phases. (1) In the sampling phase, we propose to apply self-attention fusion between a high-preference sample (called reference image) and a low-preference sample (called base image) under the guidance of an auto-extracted mask to obtain a new sample (called target image), as illustrated in Figure 2 (c). The mask, which is derived from the denoising process of the reference image, can reveal the position of alignment-related area. By applying self-attention fusion in the alignment-related area, the target image becomes as well-aligned as the reference image. Simultaneously, with shared random

Read the complete text