MiVE: Multiscale Vision-language features for reference-guided video Editing
Tong Wang, Meng Zou, Chengjing Wu, Xiaochao Qu, Luoqi Liu, Xiaolin Hu, Ting Liu · 2026
Read MiVE: Multiscale Vision-language features for reference-guided video Editing by Tong Wang, Meng Zou, Chengjing Wu, Xiaochao Qu, Luoqi Liu, Xiaolin Hu, Ting Liu (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2605-14664.txt.
Introduction
Wang, Zou, Wu, Qu, Liu, Hu and Liu present MiVE, a framework for reference-guided video editing from a source video, a text instruction, plus a reference image. Separate encoders suffer modality gaps, while unified encoders lose fine spatial detail by using only final-layer features. MiVE instead draws on many layers of a vision-language model, since early layers hold spatial detail and deeper layers hold global meaning. It is claimed to rank first in human preference tests.
Catalog record
| CREATOR | Tong Wang, Meng Zou, Chengjing Wu, Xiaochao Qu, Luoqi Liu, Xiaolin Hu, Ting Liu |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 48014 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2605.14664
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
MiVE: Multiscale Vision-language features for reference-guided video Editing Tong Wang, Meng Zou, Chengjing Wu, Xiaochao Qu, Luoqi Liu, Xiaolin Hu, Ting Liu Abstract Reference-guided video editing takes a source video, a text instruction, and a reference image as inputs, requiring the model to faithfully apply the instructed edits while preserving original motion and unedited content. Existing methods fall into two paradigms, each with inherent limitations: decoupled encoders suffer from modality gaps when processing instructions and visual content independently, while unified vision-language encoders lose fine-grained spatial details by relying solely on final-layer representations. We observe that VLM layers encode complementary information hierarchically—early layers capture localized spatial details essential for precise editing, while deeper layers encode global semantics for instruction comprehension. Building on this insight, we present MiVE ( M ult i scale V ision-language features for reference-guided video E diting), a framework that repurposes VLMs as multiscale feature extractors. MiVE extracts hierarchical features from Qwen3-VL and integrates them into a unified self-attention Diffusion Transformer, eliminating the modality mismatch inherent in cross-attention designs. Experiments demonstrate that MiVE achieves state-of-the-art performance by ranking highest in human preference, outperforming both academic methods and commercial systems. Figure 1 : Qualitative comparison on reference-guided video editing. MiVE faithfully propagates edits from the reference image while preserving fine-grained details, outperforming the commercial system Kling O1. See Section 6 for more results. 1 Introduction Reference-guided video editing aims to propagate edits from a reference image—typically the first frame modified to reflect desired changes—throughout an entire video sequence while preserving original motion and unedited content. Formally, given a source video \mathbf{x}_{\text{src}} and a text instruction \mathbf{x}_{\text{text}} , we first obtain an edited reference image \mathbf{x}_{\text{ref}} by applying an external image editing model (e.g., FLUX.1 Kontext ( 9 ) ) to the first frame of \mathbf{x}_{\text{src}} according to \mathbf{x}_{\text{text}} . The objective is to generate an edited video \hat{\mathbf{x}}_{\text{tgt}} that (1) faithfully propagates the visual attributes and appearance changes specified in \mathbf{x}_{\text{ref}} across all frames, and (2) maintains the motion dynamics and semantic integrity of \mathbf{x}_{\text{src}} in regions not intended for modification. As illustrated in Figure 1 and the online Supplementary Videos 1 1 1 Supplementary Videos and additional visual comparisons are available at https://mivepaper.github.io . , the hair color in this case should be altered to match the reference image, while the hair style of the original video must also be preserved. This task presents a fundamental challenge: the model must simultaneously discern what to modify from the reference and what to preserve from the source. These objectives are inherently competing. While the model must faithfully propagate visual attributes from the reference image, it must also maintain the motion dynamics and semantic integrity of the original video, even for regions occluded in the reference. Resolving this tension demands deep cross-modal reasoning over the editing instruction and source video. However, existing approaches often encounter fundamental limitations in multimodal fusion. Predominant models adopt decoupled architectures , where language models (e.g., T5 ( 14 ) ) and visual encoders (e.g., SigLIP ( 25 ) ) process inputs independently ( 5 ; 22 ; 10 ) . This separation restricts cross-modal reasoning, especially in reference-guided video editing, as late-stage cross-attention often fails to bridge the semantic gap between modalities. A natural remedy is to employ Vision-Language Models (VLMs) as unified encoders, as adopted by recent works such as Kling O1 ( 4 ) . However, two critical challenges remain. First, these methods rely solely on final-layer representations, missing fine-grained spatial details essential for video editing (See Supplementary Videos 1 for examples). While hierarchical encoding is well-known in deep neural networks, our analysis confirms VLMs follow this pattern—early layers encode spatial details, final layers capture semantics—motivating multiscale extraction. Second, how to inject these features is non-trivial. Cross-attention is asymmetric: visual tokens query conditional features, but conditional tokens remain agnostic to the visual content, limiting fine-grained correspondence. Effective editing demands both richer representations and bidirectional fusion within a shared space. Motivated by this insight, we propose MiVE ( M ult i scale V ision-language features for reference-guided video E diting). MiVE comprises three core components designed to leverage this hierarchical complementarity. First, a multiscale context extraction module harvests features from both early and final VLM layers, projecting them into the diffusion latent space. Second, reference-aware latent encoding concatenates reference frames with video latents to preserve structural cues. Third, a unified self-attention backbone processes all tokens—visual latents and multi-level VLM features—within a single attention manifold. Unlike conventional cross-attention, our unified token modeling enables the model to learn modality alignment within a shared space, facilitating deep integration of fine-grained spatial details and global semantic context throughout the generation process. Extensive experiments (Section 6.2 ; see Supplementary Videos 1 for visual comparisons) across diverse editing scenarios demonstrate that MiVE achieves state-of-the-art performance, ranking first in human evaluations and surpassing both leading academic methods and commercial systems in automated evaluation using Gemini-3-Flash ( 6 ) across six dimensions. Systematic ablation studies (Sec. 6.3 ) further validate our design choices, confirming that integrating multiscale VLM features via unified self-attention is essential for balancing semantic understanding with spatial fidelity. 2 Related Work 2.1 Conditioning Encoders Most multimodal video models follow a decoupled architecture, where T5-like ( 14 ) language models encode text instructions and CLIP-style visual encoders process reference images ( 18 ; 23 ) . This separation introduces a semantic gap that cross-attention cannot fully bridge. Vision-Language Models (VLMs) offer a unified alternative by jointly processing visual and textual inputs. Recent VLMs such as Qwen3-VL ( 2 ) and MiniCPM-V-4.5 ( 24 ) demonstrate that joint encoding enables richer cross-modal interactions, enabling deeper cross-modal integration for video editing tasks. 2.2 Video Editing Existing video editing techniques can be broadly categorized into mask-guided methods and mask-free methods. Mask-guided editing. Recent work such as VACE ( 8 ) and VideoPainter ( 3 ) employs explicit segmentation masks for spatial control in video editing. VACE proposes a unified framework that integrates various visual conditions and optionally incorporates a reference frame as guidance through latent-space concatenation of noisy latents, mask latents, and conditioning video. VideoPainter adopts a different approach with a dual-branch Diffusion Transformer that requires an edited first frame to anchor identity features, which are subsequently propagated to masked regions via an identity resampling strategy. While effective in controlled settings, these methods encounter difficulties when object motion is rapid or backgrounds are complex—obtaining accurate, temporally consistent masks becomes non-trivial and can limit their practical applicability. Mask-free editing. Recent efforts have moved toward end-to-end editing paradigms to reduce reliance on explicit masks. Lucy Edit ( 16 ) establishes a foundational model through channel concatenation but struggles with fine-grained local manipulation. Wan-Animate ( 5 ) incorporates reference images and control signals (face keypoints and pose skeletons) for character replacement and animation; however, it cannot adequately handle environment relighting or occlusion recovery. Several methods infer editing regions implicitly from text instructions. LoVoRA ( 21 ) learns object boundaries across space and time but still requires mask supervision during training. VideoCoF ( 22 ) introduces Chain-of-Frames reasoning but incurs prohibitive computational costs for longer videos. ReCo ( 26 ) reframes the problem as contextual learning, though this approach also introduces significant overhead. Other methods explore implicit prior