DenseDPO: Fine-Grained Temporal Preference Optimization for Video Diffusion Models

Ziyi Wu, Anil Kag, Ivan Skorokhodov, Willi Menapace, Ashkan Mirzaei, Igor Gilitschenski, Sergey Tulyakov, Aliaksandr Siarohin · 2025

Read DenseDPO: Fine-Grained Temporal Preference Optimization for Video Diffusion Models by Ziyi Wu, Anil Kag, Ivan Skorokhodov, Willi Menapace, Ashkan Mirzaei, Igor Gilitschenski, Sergey Tulyakov, Aliaksandr Siarohin (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2506-03517.txt.

Introduction

Ziyi Wu and co-authors give in 2025 a fine grain tune step for clip mint models that learn from pick pairs. Stock pick tune asks folk to rank whole clips made from split noise. Judges then lean to low move clips, which bear fewer flaws, and fine flaws wash out.

Their pairs grow by mint from marred takes of one true clip, so move form stays matched while small bits split. Picks are marked on short spans, not whole clips. A third of the marks then beats the stock tune on move, at par on text and grade. Chat judges can mark the spans too.

Catalog record

CREATORZiyi Wu, Anil Kag, Ivan Skorokhodov, Willi Menapace, Ashkan Mirzaei, Igor Gilitschenski, Sergey Tulyakov, Aliaksandr Siarohin
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE75 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2506-03517

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

DenseDPO: Fine-Grained Temporal Preference Optimization for Video Diffusion Models Ziyi Wu, Anil Kag, Ivan Skorokhodov, Willi Menapace, Ashkan Mirzaei, Igor Gilitschenski, Sergey Tulyakov, Aliaksandr Siarohin Abstract Direct Preference Optimization (DPO) has recently been applied as a post - training technique for text-to-video diffusion models. To obtain training data, annotators are asked to provide preferences between two videos generated from independent noise. However, this approach prohibits fine-grained comparisons, and we point out that it biases the annotators towards low-motion clips as they often contain fewer visual artifacts. In this work, we introduce DenseDPO, a method that addresses these shortcomings by making three contributions. First, we create each video pair for DPO by denoising corrupted copies of a ground truth video. This results in aligned pairs with similar motion structures while differing in local details, effectively neutralizing the motion bias. Second, we leverage the resulting temporal alignment to label preferences on short segments rather than entire clips, yielding a denser and more precise learning signal. With only one - third of the labeled data, DenseDPO greatly improves motion generation over vanilla DPO, while matching it in text alignment, visual quality, and temporal consistency. Finally, we show that DenseDPO unlocks automatic preference annotation using off-the-shelf Vision Language Models (VLMs): GPT accurately predicts segment-level preferences similar to task-specifically fine-tuned video reward models, and DenseDPO trained on these labels achieves performance close to using human labels. Additional results are available at https://snap-research.github.io/DenseDPO/ . 1 Introduction Recent advances in diffusion models [ 24 ] have enabled high-quality text-guided video generation [ 56 , 69 , 35 , 2 , 7 , 25 , 64 ] . Despite tremendous progress, existing video generators still fall short on temporal coherence, visual fidelity, and prompt alignment [ 86 ] , impeding their industry-level applications. Inspired by the success of learning from human feedback in language models [ 50 , 3 ] and image diffusion [ 68 , 5 , 18 ] , recent works have explored preference alignment in video diffusion [ 58 , 84 , 40 ] . Among them, methods based on Direct Preference Optimization (DPO) [ 59 ] stand out as they bypass the need for an explicit reward model [ 46 , 78 , 45 , 63 ] . However, existing DPO methods for video diffusion are largely adapted from their image-based counterparts, without addressing the unique challenges inherent to video generation. Typically, these methods first generate videos from independent noise maps, followed by human preference labeling to construct comparison pairs. Yet, human preferences in video are influenced by multiple, sometimes inversely correlated, factors, such as the visual quality (i.e., pixel-level fidelity) and the dynamic degree (i.e., strength of global motion). Indeed, current video generation models excel at producing high-quality slow-motion videos, while struggling to synthesize more challenging dynamic scenes [ 8 ] . As a result, when annotators are asked to express preferences, they often favor artifact-free slow-motion clips. Applying DPO training on such preference data further reinforces video generators’ bias toward slow-motion content, ultimately suppressing the model’s ability to generate dynamic and motion-rich videos. A natural approach to enhance competing factors, drawing inspiration from Pareto optimization, is to fix some attributes within each video pair while varying others. Motivated by the guided image synthesis approach in SDEdit [ 49 ] , we generate a pair by introducing different partial noise to a ground-truth video and perform denoising. The resulting videos in each pair share high-level semantics and motion trajectories while differing in local visual details [ 8 , 70 ] , allowing us to reduce spurious correlations. However, guided sampling inherently reduces the diversity across generated video pairs, leading to degraded DPO performance [ 51 , 61 ] . A straightforward solution might be to annotate more data pairs. Instead, we propose extracting richer and more accurate supervision from each video pair by collecting segment-level preference labels. Prior works show that multi-dimensional scores are superior to a single label in preference alignment [ 89 , 42 , 78 ] . Unlike images, videos have a unique temporal dimension [ 76 ] . In practice, we observe that human preferences over video pairs often vary across time, as artifacts may appear at different timestamps in each video, leading to inconsistent preferences. This issue is more severe with modern video generators as they produce longer videos. Therefore, we split videos into short segments (e.g., 1s slices), and collect per-segment preference labels. Thanks to temporally aligned videos from guided sampling, there is a clear one-to-one correspondence between segment pairs, simplifying the annotation process. Segment feedback also reduces the amount of ties when both videos contain artifacts, and provides more accurate supervision. In addition, it allows us to apply existing vision-language models (VLMs) [ 1 , 4 ] which can produce reliable judgment on short segments. Our main contributions are threefold: (i) A DenseDPO framework tailored towards video generation, with improved data construction and preference granularity over vanilla DPO; (ii) DenseDPO retains the motion strength of the base model while matching other metrics of vanilla DPO, with significantly higher data efficiency; (iii) We show that existing VLMs fail to label preference over long videos (e.g., 5s), but they perform well in segment-level preference, achieving results close to human labels. Figure 1 : Text-to-video results with our DenseDPO aligned model. Our method improves both visual quality and temporal consistency of the model, enabling generation of challenging motion. 2 Related Work Preference learning for image diffusion. Inspired by the success of human feedback learning in language modeling [ 50 , 3 ] , similar approaches have been adapted to image generation. One major line of work focuses on training reward models from human preference labels [ 77 , 74 , 73 , 89 , 33 ] , these models can be used as loss functions to optimize generators by direct gradient backpropagation [ 13 , 75 , 9 , 31 , 57 , 77 ] or policy gradients [ 5 , 18 ] . Another line of work uses predicted rewards on training data to re-weight the diffusion loss [ 38 ] , or only trains on high-scoring samples [ 16 , 14 , 39 ] . However, all these methods require an explicitly trained reward model, and may suffer from the reward hacking issue [ 77 , 13 ] . In contrast, Diffusion-DPO [ 68 ] and D3PO [ 80 ] directly optimize the model on pre-collected human preference pairs, bypassing the need for online reward feedback. Building on this paradigm, subsequent works have explored constructing more informative comparison data [ 85 , 32 , 29 ] , improved DPO objective [ 41 , 71 ] , and credit assignment over denoising timesteps [ 43 , 90 ] . Preference learning for video diffusion. Early approaches to preference learning in video diffusion directly borrow techniques from image diffusion, such as direct reward optimization [ 58 , 84 , 40 , 54 , 66 ] and training loss re-weighting [ 20 ] , and even relying on image reward models to provide supervision. Recent papers thus focus on developing better video reward models [ 46 , 78 , 45 ] . One strategy aggregates multiple video quality assessment metrics

Read the complete text