Retrieval, Refinement, and Ranking for Text-to-Video Generation via Prompt Optimization and Test-Time Scaling

Zillur Rahman, Alex Sheng, Cristian Meo · 2026

Read Retrieval, Refinement, and Ranking for Text-to-Video Generation via Prompt Optimization and Test-Time Scaling by Zillur Rahman, Alex Sheng, Cristian Meo (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2603-01509.txt.

Introduction

Text-to-video models can make striking clips, but the words you type steer the result more than most users expect. Zillur Rahman, Alex Sheng, and Cristian Meo address that prompt problem in this 2026 work. Fixes that edit the clip later can add flaws, while retraining the core model costs too much for most teams.

Their framework, called 3R, improves the prompt itself and needs no training of the video model. It pulls helpful modifiers from a search base, tunes prompts toward what people prefer, and smooths frames over time. Any current video model can sit behind the same front end. Tests show gains in still-image quality and in smooth motion across a clip, which points to prompt design as a cheap lever.

Catalog record

CREATORZillur Rahman, Alex Sheng, Cristian Meo
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE34541
LICENSEhttp://creativecommons.org/licenses/by-sa/4.0/
RIGHTSCC BY SA
SOURCEarXiv

Source: https://arxiv.org/abs/2603.01509

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Retrieval, Refinement, and Ranking for Text-to-Video Generation via Prompt Optimization and Test-Time Scaling Zillur Rahman, Alex Sheng, Cristian Meo Abstract While large-scale datasets have driven significant progress in Text-to-Video (T2V) generative models, these models remain highly sensitive to input prompts, demonstrating that prompt design is critical to generation quality. Current methods for improving video output often fall short: they either depend on complex, post-editing models, risking the introduction of artifacts, or require expensive fine-tuning of the core generator, which severely limits both scalability and accessibility. In this work, we introduce 3R, a novel RAG based prompt optimization framework. 3R utilizes the power of current state-of-the-art T2V diffusion model and vision language model. It can be used with any T2V model without any kind of model training. The framework leverages three key strategies: RAG-based modifiers extraction for enriched contextual grounding, diffusion-based Preference Optimization for aligning outputs with human preferences, and temporal frame interpolation for producing temporally consistent visual contents. Together, these components enable more accurate, efficient, and contextually aligned text-to-video generation. Experimental results demonstrate the efficacy of 3R in enhancing the static fidelity and dynamic coherence of generated videos, underscoring the importance of optimizing user prompts. 1 Introduction Due to exciting advancements in diffusion-based generative models and large-scale training procedures, modern text-to-video (T2V) models have achieved impressive capabilities for using natural language prompts to generate photorealistic video content ( 15 ) ( 17 ) . Despite rapid uptake in natural language processing (NLP) and subsequent innovations in text-to-image (T2I) generation 16 ; 4 , and improving image aesthetics ( 1 ) , their impact on video quality is limited ( 8 ) . Besides, the application of test-time optimization ( 29 ) in text-to-video settings remains in early exploratory stages, with meaningful open challenges ( 7 ) . This presents valuable opportunities to address problems in T2V like prompt adherence, visual quality, physical plausibility, and temporal coherence. To generate videos from texts, users provide a short text prompt to the video generation model. Recent works show that a long detailed text prompt generates better quality videos than the short user provided prompt ( 8 ; 26 ) . This underscores the importance of enhancing the user prompt before feeding it to a T2V model. The short user prompts do not contain detailed contextual information required to generate vivid visual content. Moreover, videos generated from the same prompt differ in quality due to the stochastic nature of the diffusion models. So generating multiple videos from one prompt and selecting the one that better fits the user prompt with better visual quality could be effective. To address these, in this paper, we explore avenues combining retrieval, refinement, and ranking within this emerging paradigm of inference-time compute algorithms to improve video generation quality in T2V settings. We study a black-box problem definition that is designed for direct plug-and-play applicability to off-the-shelf T2V models in real-world settings. Our contributions can be summarized as follows: • We propose Retrieval-Refinement-Ranking (3R), a retrieval based training free prompt optimization framework for T2V generation. • We propose an initial prompt refinement module that creates a detailed context rich description aligning with the user prompt. • We validate the effectiveness of 3R on EvalCrafter benchmark where it achieves SOTA results among open-source models. 2 Related Works Text-to-Video Models Text-to-video generation models ( 13 ; 18 ; 23 ; 28 ) have seen rapid advances in both model capabilities and practical accessibility. T2V models receive input prompts consisting of natural language descriptions, and comprehend described scenes, actions, objects and generate visual contents. T2V models are being used in generating animations ( 3 ) , movies ( 30 ) , commercials, etc. Prompt Optimization Frameworks IPO ( 25 ) introduces an iterative optimization algorithm to align video foundation models with human preferences. It creates a human preference dataset and trains a critique model with that dataset. An iterative optimization loop is used to align a base T2V model with human preference, and thus improving subject consistency, motion smoothness, and aesthetic quality. CCC ( 7 ) introduces a simple vision language model for text to video generation. Each candidate video is queried multiple times to get a list of issues and a content score is computed from the number of common issues. Based on those issues, initial prompts are refined to generate better results. In ( 5 ) , authors introduced RAPO: a RAG based prompt optimization model for text to video generation. A dataset is used to extract relevant modifiers to augment the user prompt. Then a fine-tuned Llama3 model ( 6 ) is used to refactor the augmented prompt into the format of training prompts. Finally, another fine-tuned LLama3 is used to select the better prompt between the refactored prompt and a refined user prompt. Google publishes VISTA ( 11 ) , one of the most computationally expensive models where pairwise video comparison is used to select candidate videos that are evaluated by multi-modal language models for their visual, audio, and context quality. Then, LLMs review the issues and refine the original prompts to generate videos again. The entire model runs for maximum 5 iterations and each iteration can have maximum 30 videos, making it 150 videos per prompt. 3 Method This section discusses the key design choices of 3R. The overall pipeline is illustrated in Fig. 1 and a pseudo algorithm is illustrated in Appendix A.1 . Figure 1: Overview of 3R pipeline. A short user prompt I is used to extract a few relevant subject, scene, actions modifiers from a relation database \mathcal{D} . Then {M}_{LLM} is used to merge those modifiers iteratively to the original user prompt to get detaild prompt P_{m} , and {R}_{LLM} checks P_{m} for any contradictory or missing information from the original prompt I , and generate N refined prompts. The refined prompts are fed to a T2V base model \mathcal{G} to generate initial videos for each prompt. Next, a video selection model selects the best candidate based on a question answering test, and a temporal interpolation network enhances temporal consistency of the final video. Modifiers Using synthetic data augmentation to rearrange knowledge for more data-efficient learning has been proven an effective pathway to mitigate the challenge of adapting a pre-trained model to a small corpus of domain-specific documents ( 27 ) . Its main purpose is to overcome the model’s context limitations, enabling effective context construction for diverse user queries. Given an original user prompt intent I , first, scene modifiers p_{j} are extracted from a relation database \mathcal{D} using a pre-trained sentence transformer. Then using cosine similarity score, we select scenes from the relation graph if it is above a threshold \tau . Each scene comes with its list of subject, action and environment modifiers. We select top-k modifiers for each scene. P_{ret}=\{p_{j}\in\mathcal{D}\mid\text{sim}(\phi(I),\phi(p_{j}))>\tau\} (1) We iteratively merge each modifier with I using a pre-trained LLM M_{LLM} in a few-shot manner. Existing method like RAPO ( 5 ) uses a large comma separated list of all the modifiers as an initial description and prompts the LLM to merge each modifier. However, this process sometimes generates incorrect and misleading results since some modifiers have little to no relevance to the I . To mitigate

Read the complete text