Dual Caption Preference Optimization for Diffusion Models
Amir Saeidi [email protected], Yiran Lawrence Luo * [email protected], Agneet Chatterjee [email protected], Shamanthak Hegde [email protected], Bimsara Pathiraja [email protected], Yezhou Yang [email protected], Chitta Baral [email protected] · 2025
Read Dual Caption Preference Optimization for Diffusion Models by Amir Saeidi [email protected], Yiran Lawrence Luo * [email protected], Agneet Chatterjee [email protected], Shamanthak Hegde [email protected], Bimsara Pathiraja [email protected], Yezhou Yang [email protected], Chitta Baral [email protected] (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2502-06023.txt.
Introduction
Amir Saeidi and co-authors improve how image generators learn from human preferences. In preference training, people pick the better of two images for a prompt, and the model learns the taste. The catch is that the prompt often fits both images about equally, so the lesson is muddy. Their method, DCPO, gives each image its own caption, so the winner and loser are described differently and the contrast sharpens.
They built a doubled-caption version of a standard preference dataset, trying three ways to write the second caption: fresh captioning, small edits, and a mix. Trained this way, the base model beat four rivals on image quality and prompt matching across five scoring systems.
Catalog record
| CREATOR | Amir Saeidi [email protected], Yiran Lawrence Luo * [email protected], Agneet Chatterjee [email protected], Shamanthak Hegde [email protected], Bimsara Pathiraja [email protected], Yezhou Yang [email protected], Chitta Baral [email protected] |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 79 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2502-06023
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Dual Caption Preference Optimization for Diffusion Models Amir Saeidi [email protected], Yiran Lawrence Luo * [email protected], Agneet Chatterjee [email protected], Shamanthak Hegde [email protected], Bimsara Pathiraja [email protected], Yezhou Yang [email protected], Chitta Baral [email protected] Abstract Recent advancements in human preference optimization, originally developed for Large Language Models (LLMs), have shown significant potential in improving text-to-image diffusion models. These methods aim to learn the distribution of preferred samples while distinguishing them from less preferred ones. However, within the existing preference datasets, the original caption often does not clearly favor the preferred image over the alternative, which weakens the supervision signal available during training. To address this issue, we introduce Dual Caption Preference Optimization (DCPO) , a data augmentation and optimization framework that reinforces the learning signal by assigning two distinct captions to each preference pair. This encourages the model to better differentiate between preferred and less-preferred outcomes during training. We also construct Pick-Double Caption , a modified version of Pick-a-Pic v2 with separate captions for each image, and propose three different strategies for generating distinct captions: captioning, perturbation, and hybrid methods. Our experiments show that DCPO significantly improves image quality and relevance to prompts, outperforming Stable Diffusion (SD) 2.1, SFT Chosen \text{SFT}_{\text{Chosen}} , Diffusion-DPO, and MaPO across multiple metrics, including Pickscore, HPSv2.1, GenEval, CLIPscore, and ImageReward, fine-tuned on SD 2.1 as the backbone. Our code and dataset are available in Github . Reviewed on OpenReview: https://openreview.net/forum?id=ruZksIJBBd Figure 1: Sample images generated by different methods on the HPSv2, Geneval, and Pickscore benchmarks. After fine-tuning SD 2.1 with \text{SFT}_{\text{Chosen}} , Diffusion-DPO, MaPO, and DCPO on Pick-a-Picv2 and Pick-Double Caption datasets, DCPO produces images with notably higher preference and visual appeal (See more examples in Appendix H ). 1 Introduction Figure 2: The DCPO pipeline in 3 variants: DCPO-c, DCPO-p, and DCPO-h, all of which require a duo of a captioned preferred image (x^{w}_{0},z^{w}) and a captioned less-preferred image (x^{l}_{0},z^{l}) . DCPO-c (Top Left): We use a captioning model to generate distinctive captions respectively for images x^{w}_{0} and x^{l}_{0} given the shared prompt c . DCPO-p (Bottom Left): We take prompt c as the caption for image x^{w}_{0} , then we use a Large Language Model (LLM) to generate a semantically perturbed prompt z_{p}^{l} given prompt c as the caption for image x^{l}_{0} . DCPO-h (Right): A hybrid method where the generated caption z^{l} is now perturbed into z_{p}^{l} for image x^{l}_{0} . Our Pick-Double Caption Dataset discussed in Section 4.1 is constructed using DCPO-c. Image synthesis models ( Rombach et al., 2022 ; Esser et al., 2024 ) have achieved remarkable advancements in generating photo-realistic and high-quality images. Text-conditioned diffusion ( Song et al., 2020 ) models have led this progress due to their strong generalization abilities and proficiency in modeling high-dimensional data distributions. As a result, they have found wide range of applications in image editing ( Brooks et al., 2023 ) , video generation ( Wu et al., 2023a ) and robotics ( Carvalho et al., 2023 ) . Consequently, efforts have focused on aligning them with human preferences, targeting specific attributes like safety ( Liu et al., 2024b ) , style ( Everaert et al., 2023 ) , and personalization ( Ruiz et al., 2023 ) , thereby improving their usability and adaptability. Similar to the alignment process of Large Language Models (LLMs), aligning diffusion models involves two main steps: 1. Pre-training and 2. Supervised Fine-Tuning (SFT). Recent fine-tuning based methods have been introduced to optimize diffusion models according to human preferences by leveraging Reinforcement Learning with Human Feedback (RLHF) ( Ouyang et al., 2022 ) , the aim of which is to maximize an explicit reward. However, challenges such as fine-tuning a separate reward model and reward hacking have led to the adoption of Direct Preference Optimization (DPO) ( Rafailov et al., 2024 ) techniques like Diffusion-DPO ( Wallace et al., 2024 ) . Intuitively, Diffusion-DPO involves maximizing the difference between a preferred image and a less preferred image for a given prompt. Although DPO-based methods are effective in comparison to SFT-based approaches, applying direct optimization in multi-modal settings presents certain challenges. Current preference optimization datasets consist of a preferred ( x^{w} ) and a less preferred ( x^{l} ) image for a given prompt ( c ). Ideally, x^{w} should show a higher correlation with c compared to x^{l} . However, we find that in current datasets, the preferred and less-preferred images exhibit a substantial overlap in their semantic distributions for the given prompt c , which we refer to as semantic overlap in the data. Additionally, irrelevant information in prompt c restricts the U-Net’s ability to predict noises from x^{l} in the diffusion reverse process, which we refer to as irrelevant prompts . This entails that there is a lack of sufficient distinguishing features between the two pairs ( x^{w} , c ), ( x^{l} , c ), thereby increasing the complexity of the optimization process. To address the aforementioned bottleneck, we propose DCPO: Dual Caption Preference Optimization , a novel preference optimization technique designed to align diffusion models by utilizing two distinct captions corresponding to the preferred and less preferred image. DCPO broadly consists of two steps: a text generation framework that develops better-aligned captions and a novel objective function that utilizes these captions as part of the training process. The text generation framework aims to mitigate the semantic overlap between the distributions of preferred and less-preferred images, given the same prompts, in existing datasets. We hypothesize that prompt c does not serve as the optimal signal for optimization because they do not convey the reasons why an image is preferred or dis-preferred; based on the above, we devise the following techniques to generate better aligned captions. The first method involves using a captioning model Q_{\phi}(z^{i}|x^{i},c) ; which generates a new prompt {z^{i}} based on an image {x^{i}} and the original prompt {c} , where i\in(w,l) . The second method introduces perturbation techniques f , such that c=z^{w},z^{l}=f(c); i.e., generating z^{l} , to represent the less preferred image, considering the original prompt c as the prompt aligned with the preferred image. We investigate multiple semantic variants of f , where each variant differs in the degree of perturbation applied to the original caption c . Finally, we also explore a hybrid combination of the above methods, where we combine the strong prior of the captioning model and the efficient nature of the perturbation method. All the above methods are designed to generate captions that effectively discriminate between the preferred and less preferred images. We introduce a novel objective function that allows DCPO to incorporate z^{w} and z^{l} into its optimization process. Specifically, during optimization, the policy model p_{\theta} increases the likelihood of the preferred image x^{w} conditioned on the prompt z^{w} , while simultaneously decreasing the likelihood of the less preferred image x^{l} conditioned on the prompt z^{l} . The results in Tables 1 and 2 demonstrate that DCPO consistently outperforms other methods, with notable improvements of +0.21 in Pickscore, +0.31 in HPSv2.1, +1.8 in normalized ImageReward, +0.15 in CLIPscore, and +2% in GenEval. Additionally, DCPO achieved 58% in general preference and 66% in visual appeal compared to Diffusion-DPO on the PartiPrompts dataset, as evaluated by GPT-4o (see Figure 7