Interactive Video Generation via Domain Adaptation
Ishaan Rawal, Suryansh Kumar · 2025
Read Interactive Video Generation via Domain Adaptation by Ishaan Rawal, Suryansh Kumar (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2505-24253.txt.
Introduction
Ishaan Rawal and Suryansh Kumar improve interactive video generation, in a 2025 paper. Text driven diffusion models now make high quality video, but letting a user steer object paths is hard. Training free methods steer paths by masking attention, yet the picture quality suffers.
The authors trace both faults to domain shift. Masking shifts the inner data spread, since pretrained models never saw masked attention. A mask normalization layer fixes that spread before use. A second fault is a starting gap: random noise does not match the added path signals, so a temporal prior enforces smooth spacetime order at each denoising step. Tests showed better picture quality and path control than past methods.
Catalog record
| CREATOR | Ishaan Rawal, Suryansh Kumar |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 46 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2505-24253
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Interactive Video Generation via Domain Adaptation Ishaan Rawal, Suryansh Kumar Abstract This supplementary material extends the empirical study of our main paper by detailing implementation settings, datasets, and additional experiments that further substantiate our claims. We first document our standardized experimental protocol: all videos are synthesized with the pre-trained Zeroscope backbone. Baselines are reproduced from the official Peekaboo [ 21 ] and Trailblazer [ 32 ] repositories to ensure a fair comparison. Next, we describe the evaluation dataset—the 126 prompt + bounding-box pairs introduced by Peekaboo—and outline the conditional (CLIP-SIM, Coverage, mIoU) and unconditional (FID, KID, JeDi) metrics used. We clarify our OWL-ViT filtering protocol and reference set (Animal Kingdom) for completeness. The core of this supplementary draft presents three sets of results: (i) Additional qualitative results demonstrating that the synergy of mask normalization and temporal intrinsic denoising reliably enforces trajectory control while preserving visual fidelity. (ii) Extended comparisons demonstrate that our method consistently outperforms other approaches, especially in challenging cases where baselines either lose the subject or fail to follow the prescribed motion path. (iii) Comprehensive ablations quantifying the influence of mask normalization and the temporal-prior gradient scale. We also provide video results as part of the supplementary material. These additional empirical studies further establish the aptness of our domain-adaptation take on the IVG problem. 1 Introduction Controlling text-to-video diffusion models has enabled new forms of human-AI interaction for creative and simulation-based tasks. A key challenge in this area is trajectory control, where users specify the subject’s spatio-temporal position movement across frames, commonly referred to as Interactive Video Generation (IVG) [ 21 , 32 ] . At present, optimal trajectory control in IVG remains a challenging problem. A suitable solution to IVG must maintain approximate object’s size and position across frames, ensuring spatial consistency despite appearance changes, and generate smooth transitions, even under complex dynamic motion. Many popular methods for IVG require retraining or fine-tuning [ 23 , 7 , 51 , 52 , 60 ] , often incorporating additional components such as Low-Rank Adaptation (LoRA) [ 18 , 65 , 55 , 42 ] and learned embeddings [ 25 , 26 , 29 , 37 , 47 , 61 , 64 , 49 , 50 ] . These approaches are often prohibitive, as they rely on custom-annotated datasets, incur significant computational costs, and typically require a reference image, limiting their applicability. To counter these limitations, training-free solutions have been proposed recently [ 21 , 22 , 58 , 32 , 54 , 39 ] . These approaches take per-frame bounding boxes as input and benefit from the rapid progress in video generation models, as shown in fig. 1 . They achieve trajectory control through customized noise initialization techniques [ 40 , 39 , 54 ] and attention masking [ 21 , 22 , 53 , 57 , 32 ] . Despite these advancements, a critical weakness persists: these methods often trade off trajectory control for perceptual quality. This tradeoff limits their applicability, motivating the need for new techniques that improve video generation quality while preserving effective trajectory control. In this paper, we introduce a principled approach to enhance training-free IVG using attention masking, while explicitly enforcing spatio-temporal coherence across frames. We begin by identifying two key limitations in existing methods. First, we find that video diffusion models are not trained with masked attention; applying attention masks during inference introduces internal covariate shift which was overlooked in prior works. Second, it is known that the imperfect nature of the diffusion noising process causes information leakage in the noised latent, known as the initialization gap [ 53 ] . As a result, the model implicitly expects trajectory-conditioned latent inputs at inference, yet most, if not all, IVG methods do not account for this discrepancy. Having identified domain shifts unique to masked video diffusion models, we ask: Can we interpret and ultimately improve both control and perceptual quality in masked video diffusion models through domain adaptation at inference time? Figure 1 : The task is to generate videos conditioned on a text prompt and a bounding box trajectory mask. Each row shows one instance: the gray panel on the left visualizes the bounding box trajectory (darker boxes appear later in time; arrows indicate direction for clarity), and the following frames show the generated video. The text prompt is shown below each video, with the subject in bold. Our model generates coherent, trajectory- and caption-aligned videos—without any training or finetuning. Let us formalize this idea. Assume a source domain distribution as \mathcal{D}_{s} and a target domain distribution as \mathcal{D}_{t} . As alluded to above, the challenges with current IVG methods involve two independent domain shift problems: (i) internal covariate shift between unmasked attention outputs ( \mathcal{D}_{s} ) and masked attention outputs ( \mathcal{D}_{t} ), and (ii) a distribution gap between randomly initialized latents ( \mathcal{D}_{s} ) and IVG-conditioned latents ( \mathcal{D}_{t} ). In both cases, the objective is to minimize the discrepancy between \mathcal{D}_{s} and \mathcal{D}_{t} . We adopt a two-pronged approach to address these problems independently. First, to resolve the internal covariate shift induced by masking, we propose mask normalization , a novel pre-normalization layer. Inspired by style transfer, a classic example of domain adaptation, mask normalization aligns the feature distributions between the outputs of unmasked attention (as the content image ; \mathcal{D}_{s} ) and masked attention (as the reference image ; \mathcal{D}_{t} ). Our analysis shows that mask normalization effectively reduces the variance shift in activations caused by attention masking. Second, to address the initialization gap, we draw inspiration from Deep Image Prior (DIP) [ 46 ] , a well-known domain adaptation method in image denoising. At each denoising step, we treat the current latent as a noisy image ( \mathcal{D}_{s} ) and the expected IVG-conditioned latent as the clean image ( \mathcal{D}_{t} ). Similar to DIP, we leverage the intrinsic diffusion prior to iteratively denoise the latent. This process is further guided by a correlative temporal prior using classifier-free guidance, which explicitly enforces spatio-temporal coherence across tokens and frames. In summary, this paper presents a new perspective on IVG through the lens of domain adaptation. We introduce the following contributions to improve training-free IVG with masked diffusion models: (1) Mask normalization, a novel pre-normalization layer that mitigates internal covariate shift by aligning feature distributions; (2) Intrinsic denoising, a modification to the diffusion sampling process that leverages the denoising prior; and (3) A classifier-conditioned temporal prior that guides intrinsic denoising to ensure spatio-temporal coherence. Our approach achieves state-of-the-art performance, outperforming all prior training-free IVG methods. 2 Related Works In recent years, diffusion models have surpass GANs [ 15 ] in data generation quality, particularly in image and video synthesis. Unlike GANs, which often suffer from mode collapse and training instability, diffusion models benefit from a well-defined denoising process that enables fine-grained control and high-fidelity visual generation. Given our paper’s focus on interactive video generation (IVG) using diffusion models, we restrict our discussion to directly relevant approaches. Video Diffusion Models. Text-based video generation using latent diffusion models has advanced significantly in recent years [ 4 , 8 , 17 , 66 , 45 ] . While these models achieve high semantic alignment with textual prompts, they lack fine-grained spatiotemporal control at