Consistency-Preserving Diverse Video Generation
Xinshuang Liu, Runfa Blark Li, Truong Nguyen · 2026
Read Consistency-Preserving Diverse Video Generation by Xinshuang Liu, Runfa Blark Li, Truong Nguyen (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2602-15287.txt.
Introduction
Liu and colleagues propose a joint-sampling setup for flow-matching video makers that improves batch variety while keeping time smoothness. Text-to-video making is pricey, so only a few samples are usually made per prompt. In this low-sample regime, raising the value of each batch needs high cross-video variety. Recent methods improve variety for images, but for videos they often degrade within-video time smoothness and require pricey backprop through a video reader.
Their method applies variety-driven updates and then cuts only the parts that would lower a time goal. To avoid image-space slopes, they compute both goals with light hidden-space nets, skipping video reading and reader backprop. Tests on a state-of-the-art text-to-video flow-matching model show variety close to strong joint-sampling rivals while greatly lifting time smoothness and color realism.
Catalog record
| CREATOR | Xinshuang Liu, Runfa Blark Li, Truong Nguyen |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 18 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2602-15287
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Consistency-Preserving Diverse Video Generation Xinshuang Liu, Runfa Blark Li, Truong Nguyen Abstract Text-to-video generation is expensive, so only a few samples are typically produced per prompt. In this low-sample regime, maximizing the value of each batch requires high cross-video diversity . Recent methods improve diversity for image generation, but for videos they often degrade within-video temporal consistency and require costly backpropagation through a video decoder. We propose a joint-sampling framework for flow-matching video generators that improves batch diversity while preserving temporal consistency. Our approach applies diversity-driven updates and then removes only the components that would decrease a temporal-consistency objective. To avoid image-space gradients, we compute both objectives with lightweight latent-space models, avoiding video decoding and decoder backpropagation. Experiments on a state-of-the-art text-to-video flow-matching model show diversity comparable to strong joint-sampling baselines while substantially improving temporal consistency and color naturalness. Code will be released. I Introduction Background and goals. Video generation is a fundamental problem in computer vision, enabling applications in media content creation [ 1 , 2 ] and virtual reality [ 3 , 4 ] . However, generating videos is computationally expensive, limiting the number of samples that can be produced under a fixed computation budget. To maximize the utility of each generation, we aim to jointly sample a batch of diverse videos. Unlike images, videos also demand temporal consistency : frames within each video must remain coherent. Thus, our goal is twofold: (i) ensure high diversity across generated videos, and (ii) maintain temporal consistency within each video. Our approach. We propose a flow-matching sampling framework that targets both goals. Inspired by diverse image generation methods [ 5 , 6 ] , we encourage diversity by using the gradient of a batch diversity objective to push samples apart (a diversity gradient ) during the flow-matching sampling process [ 7 ] . For consistency, we compute a temporal-consistency objective and remove from the diversity gradient any component that would decrease this objective (using a consistency gradient ). This regulation protects temporal consistency while retaining diversity updates that are neutral or beneficial to consistency. Latent-space models. Prior diverse image generation methods [ 5 , 6 ] compute diversity gradients in image space and backpropagate through the decoder. For videos, the higher dimensionality makes such gradient computations memory-intensive and often infeasible to perform in parallel. We instead train lightweight latent models so both diversity and consistency objectives can be computed without decoder passes. Contributions. We introduce (1) a consistency-preserving joint sampling method for flow-matching video generators via gradient regulation, and (2) latent-space embedding and interpolation models that enable lightweight computation in latent space. Experiments demonstrate that our approach substantially improves temporal consistency and color naturalness while achieving diversity comparable to strong joint-sampling baselines. Fig. 1 : Joint video generation with enhanced cross-video diversity and preserved intra-video temporal consistency based on latent-space embedding and interpolation. II Preliminaries and Related Work II-A Image and Video Generative Models We briefly review generative models for visual data and motivate our choice of flow matching for video generation. GANs [ 8 ] map noise to realistic images via adversarial training; VideoGAN [ 9 ] extends GANs to videos, but adversarial training is prone to mode collapse. Normalizing flows [ 10 ] provide exact likelihoods via invertible mappings; VideoFlow [ 11 ] applies this approach to video forecasting, but invertibility constraints can scale poorly to high resolution or long videos. Diffusion models [ 12 ] generate data by iterative denoising; video diffusion models [ 13 ] are effective but require many sequential steps per sample. Flow matching [ 7 ] learns a time-dependent velocity field to transport a base distribution p_{0} to the data distribution via an ODE. A sample x_{1} is obtained by integrating \frac{\mathrm{d}}{\mathrm{d}t}x_{t}=v_{\theta}(x_{t},t),\quad x_{0}\sim p_{0}, (1) from t=0 to t=1 , where v_{\theta}(x_{t},t) is the learned velocity field. Rectified flow [ 14 ] encourages nearly straight trajectories and has been adopted in text-to-video systems such as Wan [ 15 ] . We adopt flow matching for efficient, high-quality sampling under limited compute budgets. II-B Joint Generation to Enhance Diversity Recent works improve diversity in diffusion and flow-matching image generators by adding a diversity velocity to the sampling dynamics [ 5 , 6 ] . These methods define a batch-level diversity objective h(x_{t}^{(1:n)}) over samples x_{t}^{(1)},\dots,x_{t}^{(n)} . For example, D^{(i,i^{\prime})}=\big\|F(x_{t}^{(i)})-F(x_{t}^{(i^{\prime})})\big\|^{2}_{2},\quad K^{(i,i^{\prime})}=\frac{D^{(i,i^{\prime})}}{\text{med}(D)}\,, (2) where \text{med}(D) is the median of all pairwise differences (treated as a constant to stabilize gradients). Following [ 6 ] , we evaluate h on extrapolated end samples \hat{x}_{1}^{(i)}=x_{t}^{(i)}+(1-t)\,v_{\theta}(x_{t}^{(i)},t) . To increase h during sampling, these methods introduce a diversity velocity u : u\big(x_{t}^{(i)},x_{t}^{(-i)},t\big)=\lambda_{t}\,\Phi\!\left(\nabla_{x_{t}^{(i)}}\,h(x_{t}^{(1:n)})\right), (3) where \lambda_{t} controls the strength and \Phi optionally normalizes the gradient (we follow [ 16 ] and normalize u to match the norm of v_{\theta} ). The n samples evolve under coupled ODEs: \dot{x}_{t}^{(i)}=v_{\theta}\big(x_{t}^{(i)},t\big)+u\big(x_{t}^{(i)},x_{t}^{(-i)},t\big),\quad x_{0}^{(i)}\overset{\text{i.i.d.}}{\sim}p_{0}, (4) for i=1,\dots,n . While effective for images, these methods do not explicitly address temporal consistency or the cost of video-space gradients. Our work builds on this idea and tackles both issues. III Methodology III-A Consistency-Preserving Joint Video Generation Figure 1 provides an overview of our method. All guidance objectives are computed in latent space, avoiding decoder forward/backward passes. We follow prior work to compute the diversity objective and add its gradient to the flow-matching dynamics via Eq. ( 4 ). Specifically, we use a determinantal point process (DPP) objective [ 17 ] (effective for image diversity [ 6 ] ) over video-level and frame-level embeddings: \displaystyle e_{v}^{(i)} \displaystyle=\text{Proj}\big(M_{v}(\hat{x}_{1}^{(i)})\,\big|\,A_{v}b_{v}\big), (5) \displaystyle e_{f}^{(i,j)} \displaystyle=\text{Proj}\big(M_{f}(\hat{x}_{1}^{(i,j)})\,\big|\,A_{f}b_{f}\big), (6) where M_{v} and M_{f} are latent-space video-level and frame-level embedding models, A_{v} and A_{f} are learned alignment matrices, and \hat{x}_{1}=x_{t}+(1-t)v_{\theta}(x_{t},t) is the extrapolated terminal latent. Here, i indexes the video sample and j indexes the frame within video i ; b_{v} and b_{f} are prompt embeddings from the reference video and frame encoders (VideoPrism-B [ 18 ] and CLIP [ 19 ] , respectively). The operation \text{Proj}(a|b) removes the component of vector a along b : \text{Proj}(a|b)=a-\frac{a\cdot b}{\|b\|_{2}^{2}}\,b\,. (7) Using these embeddings, we compute difference matrices: \displaystyle D^{(i,i^{\prime})}_{v} \displaystyle=\big\|e_{v}^{(i)}-e_{v}^{(i^{\prime})}\big\|_{2}^{2}, (8) \displaystyle D^{(i,i^{\prime})}_{f} \displaystyle=\frac{1}{T}\sum_{j=1}^{T}\big\|e_{f}^{(i,j)}-e_{f}^{(i^{\prime},j)}\big\|_{2}^{2}, (9) \displaystyle D^{(i,i^{\prime})} \displaystyle=\frac{D^{(i,i^{\prime})}_{v}+D^{(i,i^{\prime})}_{f}}{2}, (10) \displaystyle K^{(i,i^{\prime})} \displaystyle=\frac{D^{(i,i^{\prime})}}{\text{med}(D)}\,. (11) Using K , we compute the DPP diversity objective O_{d} and its gradients with respect to each latent x_{t}^{(i)} : g_{d}^{(i)}=\nabla_{x_{t}^{(i)}}O_{d}\,. (12) Similarly, we compute a consistency objective using a latent frame-interpolation model M_{c} : O_{c}=-\big\|\hat{x}_{1}-M_{c}(\hat{x}_{1})\big\|_{2} (13) and its gradients: g_{c}^{(i)}=\nabla_{x_{t}^{(i)}}O_{c}\,. (14) We regulate the diversity gradient by removing only the component that would decrease O_{c} . For each sample (dropping (i) for clarity), let \alpha=\frac{g_{d}\cdot g_{c}}{\|g_{c}\|_{2}^{2}},\quad g_{\parallel}=\alpha\,g_{c},\quad g_{\perp}=g_{d}-g_{\parallel}\,. (15) We then define the regulated diversity gradient as g_{\mathrm{reg}}=g_{\perp}+\max(\alpha,0)\,g_{c}\,, (16) which drops the negative projection of g_{d} onto g_{c} . Since g_{c}=\nabla O_{c} , an update g decreases O_{c} to first order only if g^{\top}g_{c}<0 . Thus, we keep g_{\perp} and discard only the anti-aligned component of g_{d} (when \alpha<0 ). We use g_{\mathrm{reg}} in place of g_{d} when forming the diversity velocity in Eq. ( 4 ). This is related to gradient regulation used to improve image quality [ 16 ] , but here it preserves temporal consistency in video generation. III-B Training Latent Embedding and Interpolation Models We train small convolutional networks M_{v} , M_{f} , and M_{c} as latent-space video embedding, frame embedding, and frame