Discriminator-Free Direct Preference Optimization for Video Diffusion
Haoran Cheng, Qide Dong, Liang Peng, Zhizhou Sha, Weiguo Feng, Jinghui Xie, Zhao Song, Shilei Wen, Xiaofei He, Boxi Wu · 2025
Read Discriminator-Free Direct Preference Optimization for Video Diffusion by Haoran Cheng, Qide Dong, Liang Peng, Zhizhou Sha, Weiguo Feng, Jinghui Xie, Zhao Song, Shilei Wen, Xiaofei He, Boxi Wu (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2504-08542.txt.
Introduction
Haoran Cheng and nine co-authors adapt preference training for video generation without a learned judge, in a paper from 2025. Preference training needs pairs of better and worse outputs. For video, making thousands of fresh clips per round is costly, and human or machine judges miss flicker and broken motion.
Their trick uses real videos as the good cases and edited versions as the bad cases. Edits include reversed, shuffled, or noise-damaged clips, which supply clear quality signals at low cost. Theory supports the method even when real and made videos differ in spread. Tests on a public video model showed clear efficiency gains.
Catalog record
| CREATOR | Haoran Cheng, Qide Dong, Liang Peng, Zhizhou Sha, Weiguo Feng, Jinghui Xie, Zhao Song, Shilei Wen, Xiaofei He, Boxi Wu |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 50 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2504-08542
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Discriminator-Free Direct Preference Optimization for Video Diffusion Haoran Cheng, Qide Dong, Liang Peng, Zhizhou Sha, Weiguo Feng, Jinghui Xie, Zhao Song, Shilei Wen, Xiaofei He, Boxi Wu Abstract Direct Preference Optimization (DPO), which aligns models with human preferences through win/lose data pairs, has achieved remarkable success in language and image generation. However, applying DPO to video diffusion models faces critical challenges: (1) Data inefficiency—generating thousands of videos per DPO iteration incurs prohibitive costs; (2) Evaluation uncertainty—human annotations suffer from subjective bias, and automated discriminator fail to detect subtle temporal artifacts like flickering or motion incoherence. To address these, we propose a discriminator-free video DPO framework that: (1) Uses original real videos as win cases and their edited versions (e.g., reversed, shuffled, or noise-corrupted clips) as lose cases ; (2) Trains video diffusion models to distinguish and avoid artifacts introduced by editing. This approach eliminates the need for costly synthetic video comparisons, provides unambiguous quality signals, and enables unlimited training data expansion through simple editing operations. We theoretically prove the framework’s effectiveness even when real videos and model-generated videos follow different distributions. Experiments on CogVideoX demonstrate the efficiency of the proposed method. 1 Introduction Direct Preference Optimization (DPO) [ 32 ] , which leverages win/lose paired data to align model outputs with human preferences, has demonstrated remarkable success in LLMs [ 15 , 2 ] and text-to-image generation [ 37 , 39 , 22 , 45 ] . Recent advances in video diffusion models have spurred interest in adapting DPO to video generation [ 26 , 24 , 43 ] . However, existing approaches face significant challenges in practicality and scalability. As illustrated in Fig. 1 -(a), the DPO pipeline for video diffusion operates by first synthesizing outputs through model inference, then employing a preference discriminator to evaluate and rank these outputs based on human-aligned quality metrics. This process arises with two primary obstacles: First, high computational costs —generating thousands of videos per DPO iteration is prohibitively expensive (e.g., 550 seconds per 720P video for CogVideoX [ 40 ] on NVIDIA H100); Second, preference discrimination struggles with unreliable evaluation. Human annotators often struggle with inconsistent standards for subjective video quality assessment, while automated methods face difficulties in consistently distinguishing subtle artifacts across video pairs. This discrimination challenge is further exacerbated by the narrow quality margins typically observed in generated videos, making reliable preference judgments particularly complex. Figure 1 : Comparison between DPO and our proposed framework. Traditional DPO relies on computationally expensive generated video pairs, which suffer from ambiguous quality margins and scalability issues. Our method replaces generated pairs with real&edited video pairs, where edited videos serve as lose cases, and original real videos act as win cases. This approach eliminates generative overhead, provides explicit preference signals, and enables infinite scalability. To address these challenges, we propose a novel Discriminator-Free DPO (DF-DPO) format that replaces generated video pairs with real&edited video pairs, as illustrated in Fig. 1 -(b). Edited videos (e.g., reversed playback, frame-shuffled, or noise-corrupted real videos) serve as lose cases, while original real videos act as win cases. This approach offers three advantages: (1) Cost efficiency—real&edited pairs eliminate generative overhead; (2) Explicit preference signals—editing directly introduces artifacts that models must avoid; (3) Infinite scalability—editing operations enable rapid dataset expansion. While standard DPO assumes alignment between training data and model-generated distributions, existing DPO implementations for diffusion models like DiffusionDPO [ 37 ] empirically violate this principle by employing external datasets (e.g., Pick-a-Pic [ 19 ] ) where training and generation distributions diverge, which may cause reward misalignment and excessive regularization. We perform analysis in Chapter 4 , establishing theoretical safeguards against these issues, validating our real&edited pair paradigm as both practical and principled. We implement our method on CogVideoX and compare it against supervised fine-tuning (SFT). Experiments demonstrate superior alignment with human preferences, validating our framework’s efficacy. In summary, our contributions are: 1. We propose a video DPO framework using real/edited video pairs, eliminating costly generated data and ambiguous preference labels. 2. We establish that DPO remains effective with cross-distribution training data, theoretically bridging real and generated video domains. 3. We demonstrate the superiority of our approach through systematic comparisons with supervised fine-tuning (SFT) baselines on CogVideoX, achieving significant improvements in human preference alignment. 2 Related works 2.1 Video Diffusion Models The rise of diffusion models [ 11 , 17 , 33 ] has significantly advanced text-to-video tasks. Some approaches [ 18 , 4 , 35 ] inflate pre-trained T2I models by adding spatial-temporal 3D convolutions. Several works [ 3 , 6 ] demonstrate a data-centric perspective technique to enhance the performance of T2V models. Recent advances in generative modeling have spurred significant progress in video generation, driven by both commercial and open‐source research efforts. Commercial systems such as Sora [ 28 ] , Gen-3 [ 34 ] , Veo2 [ 10 ] , Kling [ 21 ] , and Hailuo [ 27 ] demonstrate impressive text-to-video capabilities along with extensions to image-to-video synthesis and specialized visual effects. These systems, however, typically rely on intricate pipelines with extensive pre- and post-processing. In contrast, open-source approaches like HunyuanVideo [ 20 ] , CogVideoX [ 40 ] , Open-Sora [ 44 ] , Open-Sora-Plan [ 23 ] and StepVideoT2V [ 26 ] are built on transparent architectures—ranging from variations of full-attention Transformers to adaptations of DiT frameworks [ 29 ] —which not only foster community engagement but also facilitate reproducible research. 2.2 RLHF in Generative Models Aligning generative models with human preferences has been a central theme in the evolution of large language models (LLMs) through techniques such as Reinforcement Learning from Human Feedback (RLHF) [ 1 , 8 , 14 , 13 , 36 ] . Although similar strategies have been applied to text-to-image diffusion models—leveraging supervised fine-tuning with preference data [ 30 , 12 , 38 ] and reward model-based optimization [ 9 , 31 , 16 ] —the direct adaptation of these methods to video diffusion is less explored. Recently, Direct Preference Optimization (DPO) [ 32 ] has emerged as an alternative to RLHF, bypassing the need for a separate reward model training phase by directly fine-tuning the generative model with preference data. While DPO and its variants have been successfully applied in LLMs [ 15 , 2 ] , text-to-image diffusion models [ 37 , 39 , 22 , 45 ] , and video diffusion models [ 26 , 24 , 43 ] . 3 Preliminaries 3.1 Diffusion Models For diffusion models, visual contents are generated by transforming an initial noise to the desired sample through multiple sequential steps [ 17 ] . It is a Markov chain process where the model continually denoises the initial noise vector \mathbf{x}_{T} and finally generates a sample \mathbf{x}_{0} . The generation step from \mathbf{x}_{t} to \mathbf{x}_{t-1} is given by: \mathbf{x}_{t}\sim q(\mathbf{x}_{t}|\mathbf{x}_{t-1})=\mathcal{N}(\mathbf{x}_{t};\sqrt{\alpha_{t}}\mathbf{x}_{t-1},\beta_{t}\mathbf{I}), (1) where \beta_{t} is the variance schedule, determining the amount of noise added at each timestep t . \alpha_{t} is a parameter obtained by \alpha_{t}=1-\beta_{t} which represents the proportion of the original data retained. The denoising model \epsilon_{\theta} , which learns to predict the noise added to \mathbf{x}_{0} for timestep t , is trained by minimizing the loss between the ground-truth \epsilon and prediction. The loss function is defined as