VALA: Learning Latent Anchors for Training-Free and Temporally Consistent Video Editing

Zhangkai Wu, Xuhui Fan, Zhongyuan Xie, Kaize Shi, Longbing Cao · 2025

Read VALA: Learning Latent Anchors for Training-Free and Temporally Consistent Video Editing by Zhangkai Wu, Xuhui Fan, Zhongyuan Xie, Kaize Shi, Longbing Cao (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2510-22970.txt.

Introduction

Zhangkai Wu and four co-authors present VALA, a tool for editing video with text prompts while keeping frames steady over time. Training-free editors reuse image diffusion models frame by frame, but hand-picked anchor frames add bias and limit scale. Inconsistent anchors make objects flicker or drift between frames.

VALA learns to pick key frames on its own and squeezes their features into compact semantic anchors. A contrast-based goal helps those anchors keep both content and timing cues. The result aims for edits that follow the prompt yet stay smooth, giving creators a lighter path to consistent video changes.

Catalog record

CREATORZhangkai Wu, Xuhui Fan, Zhongyuan Xie, Kaize Shi, Longbing Cao
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE38 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2510-22970

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

VALA: Learning Latent Anchors for Training-Free and Temporally Consistent Video Editing Zhangkai Wu, Xuhui Fan, Zhongyuan Xie, Kaize Shi, Longbing Cao Abstract Recent advances in training-free video editing have enabled lightweight and precise cross-frame generation by leveraging pre-trained text-to-image diffusion models. However, existing methods often rely on heuristic frame selection to maintain temporal consistency during DDIM inversion, which introduces manual bias and reduces the scalability of end-to-end inference. In this paper, we propose VALA ( V ariational A lignment for L atent A nchors), a variational alignment module that adaptively selects key frames and compresses their latent features into semantic anchors for consistent video editing. To learn meaningful assignments, VALA propose a variational framework with a contrastive learning objective. Therefore, it can transform cross-frame latent representations into compressed latent anchors that preserve both content and temporal coherence. Our method can be fully integrated into training-free text-to-image based video editing models. Extensive experiments on real-world video editing benchmarks show that VALA achieves state-of-the-art performance in inversion fidelity, editing quality, and temporal consistency, while offering improved efficiency over prior methods. 1 Introduction Recent progress in diffusion models (DM) ( Song, Meng, and Ermon 2021 ; Rombach et al. 2022 ; Hertz et al. 2023 ; Zhang, Rao, and Agrawala 2023 ) has significantly advanced text-to-image (T2I) generation, enabling the synthesis of high-quality, semantically coherent frame set from natural language prompts or reference image level inputs. Following the success of open source T2I models ( Rombach et al. 2022 ) , T2I based training-free video editing (VE) models has emerged as a promising direction. These approaches leverage pre-trained diffusion models, such as Stable Diffusion (SD) ( Rombach et al. 2022 ) , to perform text-driven VE without requiring additional training modules. By introducing temporal consistency into frame-wise editing pipelines, ( Qi et al. 2023 ; Geyer et al. 2024 ) can preserve the generation fidelity and cross-frame consistency by exploiting attention maps in the original U-Net architecture. Furthermore, recent training-free VE methods ( Li et al. 2024b ; Yang et al. 2025 ) have expanded through the integration of T2I plugins, enabling more diverse and controllable edits. Compared to those fine-tuning based methods ( Wu et al. 2023a ; Zhang et al. 2024 ; Feng et al. 2024 ) , training-free frameworks are more efficient and broadly applicable, making them suitable for a wide range of video tasks with low computational cost. Figure 1: Comparison between existing heuristic latent frame selection methods and our variational alignment approach. Upper: heuristic methods fix reference frames without learning. Bottom: our method VALA adaptively extracts semantic frame latents via variational alignment to have compact latent anchors. A typical training-free T2I-based VE pipeline consists of two main stages: an inversion stage that maps real video frames with a reference prompt into the noise space to acquire the editing-preserved noise latents, and an editing stage that denoises the latent representations conditioned on a target prompt to generate frames aligned with user intent. To ensure temporal consistency, early methods such as FateZero ( Qi et al. 2023 ) and TokenFlow ( Geyer et al. 2024 ) flatten the attention mechanism to jointly process multiple frames, thereby capturing motion continuity along the time axis in the inversion stage. Although it mostly ensures cross-frame consistency, how to select appropriate latent frames during inversion still remains an open challenge. Recent methods seek to improve temporal consistency and reduce redundancy by selecting representative frames, i.e., \{Q,K_{1},K_{2},...,K_{N}\} as illustrated in Figure 1 as latent anchors. Most approaches rely on either fixed sampling strategies ( Geyer et al. 2024 ; Qi et al. 2023 ) (e.g., the middle frame) or clustering heuristics ( Li et al. 2024a ) (e.g., K-means) to determine key latents. While these techniques offer some computational efficiency, they suffer from two key limitations. First, they lack adaptability to semantic or motion variations within the video, leading to suboptimal performance in complex scenes with abrupt transitions. Second, using manually defined static frame selection rules limits the flexibility and generalization of the learned representations. In this paper, we present a new framework for training-free VE based on deep variational learning. Our approach introduces a principled latent alignment module, named Variational Alignment for Latent Anchors (VALA), which selects the key latent frame within a unified variational inference formulation. Specifically, we reformulate the alignment problem as a probabilistic latent anchor compression one. Given the latent frame of a video segment, VALA learns to assign each element to one of several latent anchor prototypes that act as compact semantic summaries across frames. Instead of reconstructing a fixed latent layout, the selected anchors are optimized under a contrastive objective to preserve both spatial fidelity and temporal coherence, providing more discriminative representations for downstream cross-frame attention modules. Rather than relying on fixed anchor layouts or manually selected frame indices, VALA infers the most informative latent components through a learned variational posterior, enabling the dynamic set of anchors customized to each individual editing instance. This enables the model to capture video specific dynamics while accounting for variations in content, motion, and prompt semantics. The resulting anchor-based layout can be seamlessly incorporated into existing attention mechanisms, replacing dense spatial-temporal latents with a more compact and semantically structured representation. On real-world VE benchmarks, VALA consistently outperforms existing training-free methods in terms of reconstruction fidelity, temporal consistency, and perceptual quality, and also reduces memory consumption and inference time. It achieves strong results across standard metrics, including PSNR, SSIM, CLIP similarity, and temporal warping error. User studies further indicate a clear preference for outputs generated by VALA. The ablation study highlights the selection of priors, anchor compression ratio, and assignment strategy in improving alignment and editing quality. The main contributions can be summarized in: • VALA , a lightweight deep variational learning and inference framework that unifies key frame selection under a probabilistic formulation. • State of the art performance , achieving superior reconstruction fidelity, temporal consistency, and perceptual quality compared to prior training-free VE approaches. • Uncertainty and efficiency , offering a structured latent anchor probabilistic framework integrated with variational inference that improves temporal reasoning and reduces memory and runtime costs. 2 Related Work T2I-Based VE Models Due to the high inference cost of text-to-video (T2V) models ( Ho et al. 2022 ; Singer et al. 2022 ; Wu et al. 2023a ) and the limited availability of high-quality video–text pairs, many recent works build on open source T2I models ( Rombach et al. 2022 ) to enable VE guided by text or visual cues provided by users. Tune-A-Video (TAV) ( Wu et al. 2023b ) finetunes a LoRA module that embeds temporal features to support multiframe inputs, improving controllability and editing quality. Leveraging the scalability of pre-trained LoRA modules, TAV-based approaches have advanced the integration of visual components into the base model. For example, CCEdit ( Feng et al. 2024 ) introduces two separate injection networks to control appearance and structure, CAMEL ( Zhang et al. 2024 ) employs CLIP-based prompt embeddings to improve motion consistency across frames, and DeCo ( Zhong et al. 2025 ) incorporates a NeRF-based module to improve 3D perception of human subjects. T2I-Based training-free VE Models While TAV-based methods add trainable components, training-free approaches offer faster inference

Read the complete text