VPO: Aligning Text-to-Video Generation Models with Prompt Optimization
Jiale Cheng, Ruiliang Lyu, Xiaotao Gu, Xiao Liu, Jiazheng Xu, Yida Lu, Jiayan Teng, Zhuoyi Yang, Yuxiao Dong, Jie Tang, Hongning Wang, Minlie Huang · 2025
Read VPO: Aligning Text-to-Video Generation Models with Prompt Optimization by Jiale Cheng, Ruiliang Lyu, Xiaotao Gu, Xiao Liu, Jiazheng Xu, Yida Lu, Jiayan Teng, Zhuoyi Yang, Yuxiao Dong, Jie Tang, Hongning Wang, Minlie Huang (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2503-20491.txt.
Introduction
Jiale Cheng and a large team optimize the prompts fed to text-to-video generators, judging the rewrite by the video it produces. Generators are trained on polished, detailed descriptions, but users type terse, vague requests. Off-the-shelf prompt polishers can distort intent, drop details, or add risks. Writing in twenty twenty-five, the team's framework follows three principles: harmlessness, accuracy, and helpfulness. First they build and refine a supervised data set embodying those principles. Then preference learning adds two feedback channels, one on the rewritten text and one on the generated video itself.
Experiments showed clear gains in safety, alignment with intent, and video quality over baseline polishers. Closing the loop at the video level is the distinctive step. The prompt is treated as a control knob for the final artifact, not as prose to prettify.
Catalog record
| CREATOR | Jiale Cheng, Ruiliang Lyu, Xiaotao Gu, Xiao Liu, Jiazheng Xu, Yida Lu, Jiayan Teng, Zhuoyi Yang, Yuxiao Dong, Jie Tang, Hongning Wang, Minlie Huang |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 30 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2503-20491
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
VPO: Aligning Text-to-Video Generation Models with Prompt Optimization Jiale Cheng, Ruiliang Lyu, Xiaotao Gu, Xiao Liu, Jiazheng Xu, Yida Lu, Jiayan Teng, Zhuoyi Yang, Yuxiao Dong, Jie Tang, Hongning Wang, Minlie Huang Abstract Video generation models have achieved remarkable progress in text-to-video tasks. These models are typically trained on text-video pairs with highly detailed and carefully crafted descriptions, while real-world user inputs during inference are often concise, vague, or poorly structured. This gap makes prompt optimization crucial for generating high-quality videos. Current methods often rely on large language models (LLMs) to refine prompts through in-context learning, but suffer from several limitations: they may distort user intent, omit critical details, or introduce safety risks. Moreover, they optimize prompts without considering the impact on the final video quality, which can lead to suboptimal results. To address these issues, we introduce VPO, a principled framework that optimizes prompts based on three core principles: harmlessness, accuracy, and helpfulness. The generated prompts faithfully preserve user intents and, more importantly, enhance the safety and quality of generated videos. To achieve this, VPO employs a two-stage optimization approach. First, we construct and refine a supervised fine-tuning (SFT) dataset based on principles of safety and alignment. Second, we introduce both text-level and video-level feedback to further optimize the SFT model with preference learning. Our extensive experiments demonstrate that VPO significantly improves safety, alignment, and video quality compared to baseline methods. Moreover, VPO shows strong generalization across video generation models. Furthermore, we demonstrate that VPO could outperform and be combined with RLHF methods on video generation models, underscoring the effectiveness of VPO in aligning video generation models. Our code and data are publicly available at https://github.com/thu-coai/VPO . Figure 1 : (Upper) The critical gap between the training and inference stages in video generation models. During training, video descriptions are detailed, descriptive, and well-crafted, while user inputs in the inference stage are often concise, unclear, and poorly structured, creating a mismatch that hampers model performance. (Lower) Comparison of VPO with traditional prompt optimization methods. Traditional methods rely on the in-context learning capabilities of LLMs, which can lead to issues such as safety issues, misalignment, and low-quality videos. In contrast, VPO serves as a harmless, accurate, and helpful prompt optimizer for high-quality video generation. 1 Introduction Recent advancements in text-to-video generation models have significantly improved the ability to produce high-quality video content [ 4 , 18 , 15 , 29 , 28 ] . These models typically rely on extensive, high-quality video-text annotated datasets, where detailed labels capture specific video content. While training descriptions are detailed and carefully crafted, real-world user inputs are often brief, ambiguous, or poorly structured. This critical gap between model training and inference can lead to suboptimal outputs, making it crucial to optimize and refine user inputs to generate desired high-quality video outputs (Figure 1 ). Current approaches to optimizing user inputs for video generation mostly utilize large language models (LLMs) with in-context learning [ 28 , 29 , 15 ] . However, these methods suffer from several limitations: • Safety concerns : Existing methods do not explicitly ensure that optimized prompts remain safe, potentially leading to inappropriate or harmful video content. • Imprecise refinement : Current methods may unintentionally alter the user’s intent or introduce biases, leading to outputs that deviate from the original query. • Neglecting final video quality : Current approaches optimize prompts to be semantically richer, but do not explicitly consider how the refined prompt impacts the actual generated video, limiting their practical effectiveness. To address these challenges, we propose a set of alignment principles for video prompt optimization, inspired by the HHH (Harmless, Honest, Helpful) principle in LLMs [ 1 ] . An aligned prompt optimization model should adhere to the following: • Harmless : The refined prompt should avoid any harmful content, such as depictions of blood or violence. • Accurate : The optimized prompt should be precisely aligned with the user input, except for safety issues. • Helpful : The refined prompt should be detailed and descriptive, and can help the video generation model produce high-quality video content. Building on these principles, we introduce VPO , a principle-driven framework for optimizing user inputs for text-to-video generation, which integrates two critical stages: principle-based supervised fine-tuning (SFT) and multi-feedback preference optimization. In the SFT stage, we first leverage the in-context learning capabilities of LLMs to construct an initial SFT dataset. This data is then improved through the application of principle-based critique and refinement, ensuring the generated prompts are safe, accurately aligned with the original query, and sufficiently detailed. The resulting fine-tuned model poses a foundational ability to generate prompts following the proposed principles. In the subsequent preference optimization phase, we incorporate both text-level feedback and video-level reward signals to further improve the SFT model. By combining these feedback sources, we construct preference data that prioritizes harmlessness, accuracy, and helpfulness in the optimized prompt. Using Direct Preference Optimization (DPO), we train the model to generate prompts that consistently result in safer, more precise, and higher-quality video outputs. This two-stage framework explicitly integrates alignment principles into the prompt optimization process, bridging the critical gap between model training and real-world inference in video generation. We have conducted experiments on several popular video generation models, including CogVideoX [ 28 ] and Open-Sora [ 29 ] . Through extensive experiments, we demonstrate the effectiveness of VPO in generating harmless, accurate, and helpful prompts for high-quality video content, outperforming traditional prompt rewriting methods. Notably, on CogVideoX, VPO improves the win rate of 37.5% over original user queries and 14% over the official prompt optimization method in human evaluation. Moreover, VPO significantly reduces the unsafety rate, enhancing the overall harmlessness of video generation models. We also highlight the necessity of multi-feedback preference optimization, demonstrating its crucial role in refining prompts for improved video generation. Furthermore, we demonstrate the potential of VPO as an RLHF method. VPO not only surpasses Diffusion DPO [ 25 ] , but also brings additional benefits over Diffusion DPO. Additionally, VPO exhibits good generalization capabilities across different models—when trained on CogVideoX, it enhances the performance of Open-Sora 1.2, indicating the potential for training a general prompt optimizer for diverse video generation models. Our contributions can be summarized as follows: • We propose the principle of the alignment of video prompt optimizer: harmless, accurate, and helpful. Based on this, we introduce VPO, a systematic framework that integrates principle-based supervised fine-tuning and multi-feedback preference optimization, to build an aligned video prompt optimizer. • We demonstrate the effectiveness of VPO for improving query refinement accuracy, enhancing the safety of generated videos, and boosting the overall performance of video generation models. Furthermore, VPO exhibits good generalization across different video generation models. • We demonstrate that VPO is a novel and competitive alignment method for text-to-video models, outperforming Diffusion DPO and providing orthogonal benefits. 2 Related Work 2.1 Video Generation With the widespread application of diffusion models in the domain of image generation [ 21 , 20 ] , research efforts have increasingly shifted towards video generation models [ 3 , 8 , 23 ] . Video Diffusion Models [ 11 ] firstly extend image diffusion models to video generation by employing a 3D UNet architecture. Text2Video-Zero [ 14 ]