EIDT-V: Exploiting Intersections in Diffusion Trajectories for Model-Agnostic, Zero-Shot, Training-Free Text-to-Video Generation
Diljeet Jagpal, Xi Chen, Vinay P. Namboodiri, [1ex] University of Bath; Fudan University[1ex] {dkjj20, xc841, vpn22}@bath.ac.uk, [email protected] · 2025
Read EIDT-V: Exploiting Intersections in Diffusion Trajectories for Model-Agnostic, Zero-Shot, Training-Free Text-to-Video Generation by Diljeet Jagpal, Xi Chen, Vinay P. Namboodiri, [1ex] University of Bath; Fudan University[1ex] {dkjj20, xc841, vpn22}@bath.ac.uk, [email protected] (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2504-06861.txt.
Introduction
Diljeet Jagpal, Xi Chen, and Vinay Namboodiri turn still-image AI into video in this 2025 paper. The catalog record's creator field mixes in affiliations and emails, but the head byline names these three authors. Their method needs no retraining of the image model. It steers the model's own denoising paths to form moving frames.
A grid layout lets different parts of the frame change at different times. One helper model writes prompts for each frame, and another spots what should differ between frames. Switch timing trades smoothness against variety in each cell. Scores and user tests rated the clips high on consistency and looks. The approach works across many image models.
Catalog record
| CREATOR | Diljeet Jagpal, Xi Chen, Vinay P. Namboodiri, [1ex] University of Bath; Fudan University[1ex] {dkjj20, xc841, vpn22}@bath.ac.uk, [email protected] |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 57 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2504-06861
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
EIDT-V: Exploiting Intersections in Diffusion Trajectories for Model-Agnostic, Zero-Shot, Training-Free Text-to-Video Generation Diljeet Jagpal, Xi Chen, Vinay P. Namboodiri, [1ex] University of Bath; Fudan University[1ex] {dkjj20, xc841, vpn22}@bath.ac.uk, [email protected] Abstract Zero-shot, training-free, image-based text-to-video generation is an emerging area that aims to generate videos using existing image-based diffusion models. Current methods in this space require specific architectural changes to image-generation models, which limit their adaptability and scalability. In contrast to such methods, we provide a model-agnostic approach. We use intersections in diffusion trajectories, working only with the latent values. We could not obtain localized frame-wise coherence and diversity using only the intersection of trajectories. Thus, we instead use a grid-based approach. An in-context trained LLM is used to generate coherent frame-wise prompts; another is used to identify differences between frames. Based on these, we obtain a CLIP-based attention mask that controls the timing of switching the prompts for each grid cell. Earlier switching results in higher variance, while later switching results in more coherence. Therefore, Our approach can ensure appropriate control between coherence and variance for the frames. Our approach results in state-of-the-art performance while being more flexible when working with diverse image-generation models. The empirical analysis using quantitative metrics and user studies confirms our model’s superior temporal consistency, visual fidelity and user satisfaction, thus providing a novel way to obtain training-free, image-based text-to-video generation. Further examples and code at https://djagpal02.github.io/EIDT-V/ Eight equally spaced frames from 24-frame GIFs generated by our EIDT-V model. Top row shows SD3 Medium Esser et al. 2024 results for prompt: “A peacock displaying its feathers” . Bottom row shows SDXL Podell et al. 2024 results for prompt: “A child blowing bubbles that float and pop gently” . These examples highlight the model’s ability to generate high-quality videos with semantic and temporal coherence. † † * Corresponding author. † † Supported by EPSRC (EP/T518013/1, EP/Y021614/1); AAM available under a CC BY licence. 1 Introduction Recent advances in image generation have established diffusion models as state-of-the-art (SOTA) tools for producing visually compelling and coherent images. Open source techniques such as Stable Diffusion Rombach et al. 2022 ; Podell et al. 2024 ; Esser et al. 2024 and Flux Flux 2024 show efficient high-quality outputs made possible by the diffusion models Song et al. 2021b ; Karras et al. 2022 . Despite this, extending these capabilities to video remains challenging due to the unique temporal coherence and dynamic motion requirements. SOTA, such as OpenAI’s Sora 2024 and Meta’s MovieGen Polyak et al. 2024 , have made significant progress in high-quality text-to-video generation. However, they rely on extensive training and complex architectures, which limits accessibility. These models typically require high-end GPUs or sit behind paywalls, making them out of reach for most users. Although lower cost models Ho et al. 2022b ; Zhou et al. 2023 ; Ho et al. 2022a ; Gupta et al. 2024 ; He et al. 2023 ; Blattmann et al. 2023b ; Blattmann et al. 2023a ; Gu et al. 2024 ; Yang et al. 2023 ; Liu et al. 2024 ; An et al. 2023 ; Liu et al. 2023a exist, the difference in quality is significant, and the inference and training costs are still beyond the reach of most. Hence, research is also being pursued to explore training-free approaches Khachatryan et al. 2023 ; Hong et al. 2024 ; Huang et al. 2023 . These approaches require architectural adjustments tied to specific diffusion models, limiting flexibility and scalability while producing even lower-quality outputs. We propose a model-agnostic, zero-shot text-to-video generation approach using image-based diffusion models. Our method operates entirely in the latent space, achieving compatibility with various image-based diffusion models without modifications or additional training. This model-agnostic design enables high-quality video generation across different architectures, establishing a flexible and robust foundation for video generation. Contributions: The main contributions of our work are: • Grid-Based Prompt Switching for Conditional Image Generation: We develop a novel grid-based prompt switching technique specifically for conditional image generation. This method divides each image into localized grid regions, providing fine-grained control over variance. • Enhanced Text control via LLM Modules : Our approach incorporates two in-context trained modules within a LLM: one module generates frame-wise prompts from text inputs, while the other detects differences between consecutive frames to guide temporal consistency. • CLIP-based attention masking - The text-guided difference between consecutive frames is used through a CLIP-based attention masking module that generates the prompt switch times, thereby controlling the variance and coherence spatially and between frames. All these contributions result in a complete pipeline thoroughly validated compared with corresponding baselines and ablations. We provide more comprehensive metrics to validate the generated results’ temporal consistency and visual fidelity, along with suitable user studies. 2 Related Works 2.1 Image Diffusion Diffusion models Sohl-Dickstein et al. 2015 ; Ho et al. 2020 ; Dhariwal and Nichol 2021 ; Song et al. 2021b ; Song et al. 2021a have become essential for generating high-quality images, offering improved stability and scalability over previous SOTA GANs Goodfellow et al. 2014 . With continuous advancements Song et al. 2023 ; Liu et al. 2023b , diffusion models are evolving rapidly. The open-source Stable Diffusion (SD) models illustrate this progression: moving from SD1 Rombach et al. 2022 to SDXL Podell et al. 2024 , the architecture became significantly more complex. In SD3 Esser et al. 2024 , the original UNet architecture Ronneberger et al. 2015 was replaced by a multi-modal version of the diffusion image transformer Peebles and Xie 2023 and enhanced with rectified flows Liu et al. 2023b . These rapid architectural shifts often render tools for earlier versions obsolete, underscoring the need to develop adaptable tools. 2.2 State-of-the-Art Video Generation Models SOTA video generation models, such as Runway’s Gen-3 Alpha Runway 2024 , Meta’s MovieGen Polyak et al. 2024 , and others Kling 2024 , 2024 , have demonstrated high-quality video synthesis with solid temporal coherence. However, they are costly to train and for inference, highlighting the need for computationally efficient alternatives. 2.3 Low Cost Video Generation Initial efforts, such as VDM Ho et al. 2022b and MagicVideo Zhou et al. 2023 , adapted image diffusion processes to video, with MagicVideo leveraging latent spaces to lower computational requirements. Multi-stage models, like Imagen Video Ho et al. 2022a , utilize cascaded sub-models to enhance video resolution and frame rate, but the complexity of these stages keeps computational demands high. Similarly, transformer-based approaches, such as WALT Gupta et al. 2024 , employ memory-efficient windowed attention, though maintaining high-resolution coherence across frames still requires substantial resources. Models like LVDM He et al. 2023 and Video LDM Blattmann et al. 2023b use hierarchical and latent diffusion techniques to extend video length, aiming for efficiency but still facing scalability challenges. Even efficiency-focused methods like Latent-Shift An et al. 2023 and Ed-t2v Liu et al. 2023a that enhance motion fidelity with techniques such as temporal shift modules and identity attention remain relatively expensive, making truly low-cost, accessible video generation a continuing challenge. 2.4 Zero-Shot Video Generation Zero-shot video generation uses pre-trained image models for video synthesis tasks without additional training, enhancing accessibility. Text2Video-Zero Khachatryan et al. 2023 was an early example, utilizing approximated optical flow and replacing self-attention with cross-attention