Unified Text-Image-to-Video Generation: A Training-Free Approach to Flexible Visual Conditioning

Bolin Lai Sangmin Lee Xu Cao Xiang Li James M. Rehg, 1 Georgia Tech 2 UIUC 3 Korea University, [email protected] [email protected] {xucao2,xiangl12,jrehg}@illinois.edu, Project Page: https://bolinlai.github.io/projects/FlexTI2V/ · 2025

Read Unified Text-Image-to-Video Generation: A Training-Free Approach to Flexible Visual Conditioning by Bolin Lai Sangmin Lee Xu Cao Xiang Li James M. Rehg, 1 Georgia Tech 2 UIUC 3 Korea University, [email protected] [email protected] {xucao2,xiangl12,jrehg}@illinois.edu, Project Page: https://bolinlai.github.io/projects/FlexTI2V/ (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2505-20629.txt.

Introduction

Bolin Lai, Sangmin Lee, Xu Cao, Xiang Li, and James Rehg introduced FlexTI2V in 2025, a training-free way to guide video generation with both text and images. Most methods fine-tune a text-to-video model for a few fixed kinds of visual guidance, which is costly and rigid. Their approach aims for any number of guide images at any positions in the clip.

The method first inverts each guide image into a noisy latent form. During generation, a random patch swapping step blends visual features from those images into the video frames in local patches. No retraining of the base model is needed. The catalog creator field mixes in affiliations and emails; the head byline governs. Readers get a flexible conditioning trick that reuses strong video models as they are.

Catalog record

CREATORBolin Lai Sangmin Lee Xu Cao Xiang Li James M. Rehg, 1 Georgia Tech 2 UIUC 3 Korea University, [email protected] [email protected] {xucao2,xiangl12,jrehg}@illinois.edu, Project Page: https://bolinlai.github.io/projects/FlexTI2V/
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE52 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2505-20629

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Unified Text-Image-to-Video Generation: A Training-Free Approach to Flexible Visual Conditioning Bolin Lai Sangmin Lee Xu Cao Xiang Li James M. Rehg, 1 Georgia Tech 2 UIUC 3 Korea University, [email protected] [email protected] {xucao2,xiangl12,jrehg}@illinois.edu, Project Page: https://bolinlai.github.io/projects/FlexTI2V/ Abstract Text-image-to-video (TI2V) generation is a critical problem for controllable video generation using both semantic and visual conditions. Most existing methods typically add visual conditions to text-to-video (T2V) foundation models by finetuning, which is costly in resources and only limited to a few pre-defined conditioning settings. To tackle these constraints, we introduce a unified formulation for TI2V generation with flexible visual conditioning. Furthermore, we propose an innovative training-free approach, dubbed FlexTI2V , that can condition T2V foundation models on an arbitrary amount of images at arbitrary positions. Specifically, we firstly invert the condition images to noisy representation in a latent space. Then, in the denoising process of T2V models, our method uses a novel random patch swapping strategy to incorporate visual features into video representations through local image patches. To balance creativity and fidelity, we use a dynamic control mechanism to adjust the strength of visual conditioning to each video frame. Extensive experiments validate that our method surpasses previous training-free image conditioning methods by a notable margin. Our method can also generalize to both UNet-based and transformer-based architectures across various image conditioning settings. Figure 1 : We propose FlexTI2V , a novel training-free approach that can add flexible image conditioning to off-the-shelf text-to-video foundation models. We are able to frame an arbitrary number of images at arbitrary positions in the synthetic video with vivid motion and smooth transitions. In this figure, images with blue edges are condition images, and images with green edges are generated video frames. 1 Introduction In recent years, diffusion models [ 13 ] have shown excellent performance in text-to-video (T2V) generation [ 14 , 11 , 44 , 16 , 24 , 3 , 50 , 56 , 52 ] , serving as video foundation models for various downstream video generation tasks. Given the vagueness and ambiguity of language [ 26 , 25 ] , recent work has investigated synthesizing videos conditioned on RGB images for precise controllability on the appearance of people, objects and background [ 1 , 27 , 5 ] , which is as known as image-to-video (I2V) generation. In these models, RGB images can be used as conditions alone [ 1 , 27 , 53 , 33 ] , or incorporated with textual prompts for accurate guidance [ 57 , 9 , 54 , 20 ] . For clarity, we explicitly refer to the latter as text-image-to-video (TI2V) generation following [ 20 , 36 ] . Figure 2 : Comparison with classic TI2V tasks. Our task requires video generation conditioned on any number of images at any positions in the output video, which unifies existing classic TI2V tasks. The images with blue and pink edges are condition images, and images with green edges are generated video frames. To include extra image conditioning in video generation, the most widely-used approach is finetuning pretrained T2V model with paired texts, images and videos. However, the prohibitive demands of high-quality data and computing resources, and the risks of involving unexpected bias from the data during finetuning impede the development of TI2V models. Moreover, as shown in Fig. 2 , off-the-shelf TI2V models can only apply the image conditioning setting pre-defined during training ( e.g. , image animation [ 9 , 58 , 4 ] and video frame interpolation [ 43 , 39 ] ), lacking sufficient flexibility in the number and positions of condition images. Any changes in the conditioning setting require re-training of the entire model. To avoid the computing overhead and undesired bias of finetuning, training-free image conditioning approaches are promising solutions to incorporate flexible visual control into off-the-shelf T2V models. Prior to our work, recent studies have been conducted on training-free controllable video generation with various conditions [ 22 , 23 , 59 , 17 , 28 , 29 ] . For training-free image conditioning, the previous methods either synthesize only one video frame in one denoising pass, making it inefficient in inference [ 36 ] , or fill in the missing video frames between two bounded images solely based on visual cues, losing semantic control of generated frames in between [ 7 ] . Similar to finetuned TI2V models, they are also only capable of following pre-defined conditioning settings, and still short of flexibility for image conditioning. These inevitable deficiencies make the existing training-free methods suboptimal for TI2V generation problem, thus demanding more efforts and novel solutions. In this work, we formally define a generic TI2V generation problem with high flexibility in the amounts and positions of condition images (see Figs. 1 , 2 ), unifying the existing TI2V tasks such as image animation, rewinding and video frame interpolation. To address aforementioned issues, we introduce FlexTI2V , an innovative flexible training-free approach that can condition an off-the-shelf text-to-video diffusion model on an arbitrary number of images at arbitrary positions in the synthetic video. Specifically, we first reverse the images to noisy representations at each denoising step. To harmonize the difference in denoising paths of T2V models and image conditioning, we propose a novel random patch swapping method to incorporate visual features of input images into each video frame, by swapping a random portion of patches between video frames and condition images. We also use a dynamic adjustment strategy to control the percentage of swapped patches and denoising steps for each frame to prevent frozen videos. The extensive experiments suggest that our approach outperforms the previous training-free image conditioning methods for video diffusion models. Overall, our main contributions are summarized as follows: \bullet We formally define a generic text-image-to-video generation problem, unifying the various settings of existing TI2V tasks. \bullet We introduce FlexTI2V, a novel training-free approach to incorporate flexible image conditioning into pretrained text-to-video diffusion models. The key innovation is a random patch swapping method to incorporate visual guidance into the denoising process in a plug-and-play manner. \bullet The experiments indicate that our approach notably surpasses the existing training-free image conditioning methods. Our method also generalizes seamlessly to both UNet and transformer architectures. 2 Related Work Video Generation Conditioned on Texts and Images Text-to-video (T2V) generation has gained impressive advancement since the emergence of diffusion models [ 14 , 11 , 15 , 3 , 24 , 56 , 50 , 52 ] . Given the ambiguity of language in describing visual details, RGB images are exploited as the condition for video generation. Image conditioning can be used alone for image-to-video (I2V) generation [ 2 , 6 , 53 , 33 , 1 , 27 ] , or integrated with textual prompts for text-image-to-video (TI2V) generation [ 37 , 57 , 21 , 4 , 51 , 20 ] . Here, we mainly discuss previous TI2V methods. Hu et al. [ 19 ] propose TI2V task for the first time, and develop a transformer-based architecture to achieve decent controllability and diversity in generated videos. Guo et al. [ 9 ] develop a practical framework, AnimateDiff, to train a model-agnostic plug-and-play motion module that can turn any personalized text-to-image diffusion model into an image animator. Xing et al.

Read the complete text