Improving Dynamic Object Interactions in Text-to-Video Generation with AI Feedback
Hiroki Furuta Heiga Zen Dale Schuurmans Aleksandra Faust, Yutaka Matsuo Percy Liang Sherry Yang [email protected] [email protected] · 2024
Read Improving Dynamic Object Interactions in Text-to-Video Generation with AI Feedback by Hiroki Furuta Heiga Zen Dale Schuurmans Aleksandra Faust, Yutaka Matsuo Percy Liang Sherry Yang [email protected] [email protected] (2024) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2412-02617.txt.
Introduction
Hiroki Furuta and co-authors use, in a 2024 paper, AI feedback to teach text-to-video models how objects really move. Today's generators produce striking clips, yet objects in them slide, merge, and fall in ways that break physics, especially when several interact. The authors borrow the alignment playbook from language models: score the output, then fine-tune toward better scores. Their first result is theoretical: the offline fine-tuning methods on offer all serve one shared objective, so the choice of reward and data matters more than the algorithm.
Human scoring cannot scale, so they let vision-language models judge the motion. Simple yes-or-no AI feedback on object dynamics beat standard quality metrics and drove the largest gains in interaction scenes, confirmed by human raters. The authors offer the recipe as a practical path to physically saner video.
Catalog record
| CREATOR | Hiroki Furuta Heiga Zen Dale Schuurmans Aleksandra Faust, Yutaka Matsuo Percy Liang Sherry Yang [email protected] [email protected] |
|---|---|
| YEAR | 2024 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 68 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2412-02617
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Improving Dynamic Object Interactions in Text-to-Video Generation with AI Feedback Hiroki Furuta Heiga Zen Dale Schuurmans Aleksandra Faust, Yutaka Matsuo Percy Liang Sherry Yang [email protected] [email protected] Abstract Large text-to-video models hold immense potential for a wide range of downstream applications. However, they struggle to accurately depict dynamic object interactions, often resulting in unrealistic movements and frequent violations of real-world physics. One solution inspired by large language models is to align generated outputs with desired outcomes using external feedback. In this work, we investigate the use of feedback to enhance the quality of object dynamics in text-to-video models. We aim to answer a critical question: what types of feedback, paired with which specific self-improvement algorithms, can most effectively overcome movement misalignment and realistic object interactions? We first point out that offline RL-finetuning algorithms for text-to-video models can be equivalent as derived from a unified probabilistic objective. This perspective highlights that there is no algorithmically dominant method in principle; rather, we should care about the property of reward and data. While human feedback is less scalable, vision-language models could notice the video scenes as humans do. We then propose leveraging vision-language models to provide perceptual feedback specifically tailored to object dynamics in videos. Compared to popular video quality metrics measuring alignment or dynamics, the experiments demonstrate that our approach with binary AI feedback drives the most significant improvements in the quality of interaction scenes in video, as confirmed by AI, human, and quality metric evaluations. Notably, we observe substantial gains when using signals from vision language models, particularly in scenarios involving complex interactions between multiple objects and realistic depictions of objects falling. 1 Introduction Large video generation models pre-trained on internet-scale videos have broad applications such as generating creative video content ( Ho et al., 2022a ; Hong et al., 2022 ; Singer et al., 2022 ; Blattmann et al., 2023b ) , creating novel games ( Bruce et al., 2024 ) , animations ( Wang et al., 2019 ) , movies ( Zhu et al., 2023 ) , and personalized educational content ( Wang et al., 2024 ) , as well as simulating the real-world ( Yang et al., 2023b ; Brooks et al., 2024 ) and controlling robots ( Du et al., 2024 ; Ko et al., 2023 ) . Despite these promises, even SoTA text-to-video models today still suffer from hallucination, generating unrealistic objects or movements that violate physics ( OpenAI, 2024 ; Bansal et al., 2024 ; Yang et al., 2024b ) , generating static scenes, and ignoring specified movement altogether ( Figure 1 ), which hinders the practical use of text-to-video models. Expanding both the dataset and the model size has proven effective in reducing undesirable behaviors in large language models (LLMs) ( Hoffmann et al., 2022 ) . However, when it comes to video generation, this scaling process is more complex. Creating detailed language labels for training text-to-video models is a labor-intensive task, and the architecture for video generation models has not yet reached a point where it can effectively scale in the same way LLMs have ( Yang et al., 2024b ) . In contrast, one of the most impactful advances in improving LLMs has been the integration of external feedback ( Christiano et al., 2017 ; Ouyang et al., 2022 ) . This raises important questions about what kinds of feedback can be gathered for text-to-video models and how such a feedback can be integrated into training to reduce hallucination and enhance the quality of events in video. Figure 1: Generated videos from open models: VideoCrafter ( Chen et al., 2024 ) and VADER ( Prabhudesai et al., 2024 ) . Both pre-trained and RL-finetuned models fail to represent object movement in the prompts. Improving the dynamics of object interaction in the generated video is still an open problem. In this paper, we investigate the recipe for improving dynamic interactions with objects in text-to-video diffusion models ( Ho et al., 2022b ) through external feedback. We cover various self-improvement methods based on reinforcement learning (RL) and automated feedback for text-to-video generation. We begin by noting that two representative offline algorithms, reward-weighted regression (RWR) ( Peters and Schaal, 2007 ) and direct preference optimization (DPO) ( Rafailov et al., 2023 ) , introduced independently in prior works, have stemmed from the same objective through the lens of probabilistic inference, with some design choices such as KL-regularization and policy projection. This equivalence highlights that there is no dominant algorithm in principle; rather, we should care about coupling with reward and data depending on the specific use case. For feedback choice, we test popular metric-based feedback on alignment ( Radford et al., 2021 ) , human preference ( Wu et al., 2023a ; Kirstain et al., 2023 ) , and dynamics ( Huang et al., 2024 ) , and propose leveraging large-scale vision-language models (VLMs) to provide perceptual feedback carefully focusing on object interactions, because VLMs could interpret the video scenes as humans do. RL-finetuning effectively maximizes various types of reward feedback compared to SFT with self-generated data. Compared to popular video quality metrics in prior works ( Prabhudesai et al., 2024 ; Li et al., 2024b ; Yuan et al., 2023 ) , measuring alignment and dynamics, our approach with AI feedback drives the most significant improvements in the quality of interaction scenes; as confirmed by VLMs, human, and quality metric evaluations including smoothness and consistency from VBench ( Huang et al., 2024 ) . Qualitatively, pre-trained text-to-video models often struggle with multi-step interactions, new object appearances, and spatial 3D relationships, while RL fine-tuning helps address these limitations. We lastly provide recommended algorithm choices for practitioners. 2 Preliminaries Denoising Diffusion Probabilistic Models. We adopt a denoising diffusion probabilistic model (DDPM) ( Sohl-Dickstein et al., 2015 ; Ho et al., 2020 ) for generating a H -frame video \textbf{x}_{0}=[x_{0}^{1},...,x_{0}^{H}] . DDPM considers approximating the reverse process for data generation with a parameterized model p_{\theta}(\textbf{x}_{0})=\int p_{\theta}(\textbf{x}_{0:T})d\textbf{x}_{1:T} , where p_{\theta}(\textbf{x}_{0:T})=p(\textbf{x}_{T})\prod_{t=1}^{T}p_{\theta}(\textbf{x}_{t-1}\mid\textbf{x}_{t}) , and p(\textbf{x}_{T})=\mathcal{N}(0,I) , p_{\theta}(\textbf{x}_{t-1}\mid\textbf{x}_{t})=\mathcal{N}(\mu_{\theta}(\textbf{x}_{t},t),\sigma_{t}^{2}I) . DDPM also has a forward process, where the Gaussian noise is iteratively added to the data with different noise level \beta_{t} : q(\textbf{x}_{1:T})=\prod_{t=1}^{T}q(\textbf{x}_{t}\mid\textbf{x}_{t-1}) and q(\textbf{x}_{t}\mid\textbf{x}_{t-1})=\mathcal{N}(\sqrt{1-\beta_{t}}\textbf{x}_{t-1},\beta_{t}I) . Considering the upper bound of negative log-likelihood \mathbb{E}_{q}[-\log p_{\theta}(\textbf{x}_{0})] with practical approximations, we obtain the objective for DDPM \mathcal{J}_{\texttt{DDPM}} as follows, \begin{split}\mathbb{E}_{q}\left[\sum_{t=1}^{T}D_{\text{KL}}(q(\textbf{x}_{t-1}\mid\textbf{x}_{t},\textbf{x}_{0})~||~p_{\theta}(\textbf{x}_{t-1}\mid\textbf{x}_{t}))\right]\approx\mathbb{E}_{\textbf{x}_{0},\epsilon,t}\left[\|\epsilon-\epsilon_{\theta}(\sqrt{\alpha_{t}}\textbf{x}_{t}+\sqrt{\beta_{t}}\epsilon,t)\|^{2}\right]:=\mathcal{J}_{\texttt{DDPM}},\end{split} (1) where \alpha_{t}=1-\beta_{t} . After training parameterized denoising model \epsilon_{\theta} , the video might be generated from the initial noise \textbf{x}_{T}\sim\mathcal{N}(\textbf{0},I) through the iterative denoising from t=T to 1 : \textbf{x}_{t-1}:=\frac{1}{\sqrt{\alpha_{t}}}\left(\textbf{x}_{t}-\frac{\beta_{t}}{\sqrt{\bar{\beta}_{t}}}\epsilon_{\theta}(\textbf{x}_{t},t)\right)+\sigma_{t}u_{t} , where u_{t}\sim\mathcal{N}(0,I) and \bar{\beta}_{t}=\prod_{s=1}^{t}\beta_{s} . We also adopt text and first-frame conditioning \textbf{c}=[c_{\text{text}},c_{x_{0}^{1}}] with classifier-free guidance ( Ho and Salimans, 2022 ) , and will consider the conditional formulation p(\textbf{x}_{0}\mid\textbf{c}) in the following sections. Figure 2: ( Left ) A pipeline for RL-finetuning with feedback. We first generate videos from the pre-trained models, and then use VLMs capable of video understanding (or metric-based reward) to obtain feedback labels. Those data are leveraged for offline and iterative RL-finetuning. We describe the pseudo algorithm in Algorithm 1 . ( Right ) Example videos from Something-Something-V2. We define five principles of challenging object movements: object removal (OR), multiple objects (MO), deformable object (DO), directional movement (DM), and falling down (FD). Probabilistic Inference View of RL-Finetuning Objectives.