Omni-Video : Democratizing Unified Video Understanding and Generation
Zhiyu Tan Hao Yang Luozheng Qin Jia Gong Mengping Yang Hao Li, [5pt] Fudan University · 2025
Read Omni-Video : Democratizing Unified Video Understanding and Generation by Zhiyu Tan Hao Yang Luozheng Qin Jia Gong Mengping Yang Hao Li, [5pt] Fudan University (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2507-06119.txt.
Introduction
Zhiyu Tan and five co-authors present Omni-Video, a unified framework for video understanding, generation and instruction-based editing, in a 2025 report. Image models have merged these tasks, but video has lagged behind. Their key idea is to teach a multimodal language model to output continuous visual clues, which a diffusion decoder then turns into video.
A light vision head sits on the language model, and an adapter feeds its tokens to the decoder. A multi-stage training scheme links the two parts with limited data and compute. Tests show sound generalization across generation, editing and understanding tasks. The authors frame the system as a step toward open, shared video foundation models.
Catalog record
| CREATOR | Zhiyu Tan Hao Yang Luozheng Qin Jia Gong Mengping Yang Hao Li, [5pt] Fudan University |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 55 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2507-06119
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Omni-Video : Democratizing Unified Video Understanding and Generation Zhiyu Tan Hao Yang Luozheng Qin Jia Gong Mengping Yang Hao Li, [5pt] Fudan University Abstract Notable breakthroughs in unified understanding and generation modeling have led to remarkable advancements in image understanding, reasoning, production and editing, yet current foundational models predominantly focus on processing images, creating a gap in the development of unified models for video understanding and generation. This report presents Omni-Video , an efficient and effective unified framework for video understanding, generation, as well as instruction-based editing. Our key insight is to teach existing multimodal large language models (MLLMs) to produce continuous visual clues that are used as the input of diffusion decoders, which produce high-quality videos conditioned on these visual clues. To fully unlock the potential of our system for unified video modeling, we integrate several technical improvements: 1) a lightweight architectural design that respectively attaches a vision head on the top of MLLMs and a adapter before the input of diffusion decoders, the former produce visual tokens for the latter, which adapts these visual tokens to the conditional space of diffusion decoders; and 2) an efficient multi-stage training scheme that facilitates a fast connection between MLLMs and diffusion decoders with limited data and computational resources. We empirically demonstrate that our model exhibits satisfactory generalization abilities across video generation, editing and understanding tasks. Our code will be publicly available at: https://github.com/SAIS-FUXI/Omni-Video . † † † \dagger Equal Contribution. † † * Corresponding Authors. Figure 1: Comprehensive illustration of the fundamental capabilities of our Omni-Video on (a) photorealistic text-to-image/video generation, (b) image/video editing ( i.e., change the style of source image toward target image and remove objects in the source image), and (c) video understanding (describe directions, background, motion, and camera movement, etc. ), all these tasks could be accomplished within a single unified architecture. 1 Introduction The remarkable advancement of artificial intelligence in recent years, especially multimodal understanding chen2024internvl ; bai2025qwen25_vl empowered by large-language models (LLMs) and multimodal generation dalle3 ; sdxl ; wan2025wan models driven by diffusion models, has enabled thrilling possibilities to go beyond existing experiences across various modalities, including vision redmon2016you ; rombach2022high ; he2016deep ; dosovitskiyimage , language devlin2019bert ; brown2020language ; raffel2020exploring , as well as audio baevski2020wav2vec ; van2016wavenet ; park2019specaugment ; radford2023robust , etc. Despite these progresses, most understanding and generation systems are often designed separately, i.e. , each specialized for a specific task or modality, leading to significant modeling redundancy and laborious computing complexity. Alternatively, unified multimodal models chen2025janus ; tong2024metamorph have emerged as a promising solution and gained increasing attention for their ability to input/output diverse data types seamlessly. Such unified pipeline is trained jointly across multiple modalities to model task-agnostic representations wang2025diffad ; chou2018unifying within a single framework, thereby removing tailored modality-specific components while achieving strong performance on both individual and compositional tasks hu2021unit ; wang2024git . Following this philosophy, recent efforts chen2025blip3 ; deng2025bagel have focused on developing unified architectures that simultaneously handle multimodal understanding and generation tasks. Considering that the transformer-based autoregressive (AR) architecture serves as the predominant architectural paradigm for multimodal understanding , earlier approaches project both texts and images into discrete tokens to facilitate AR modeling across modalities chen2025janus ; wang2024emu3 . For instance, Emu3 wang2024emu3 tokenize images, texts, and videos into a discrete space and train a single transformer model via next-token prediction on these multimodal sequences. Chameleon team2024chameleon designed an early fusion scheme to project all modalities into a share discrete representational space for better interaction and reasoning. Motivated by the great success of diffusion models sdxl ; dalle3 in visual generation, the most recent works chen2025blip3 ; deng2025bagel design hybrid diffusion-AR architectures that incorporate diffusion decoders to translate high-level visual tokens into photorealistic outputs. Representative works such as Transfusion zhou2024transfusion and Show-O xie2024show respectively employ continuous and discrete diffusion modeling with full attention for improved visual quality. However, training the above unified models from scratch requires substantial computational resources and a massive amounts of multimodal data. Additionally, integrating MLLMs and diffusion models within a unified system poses significant challenges in architectural modifications and hyperparameter tuning. To tackle these challenges, MetaMorph tong2024metamorph demonstrated that an improved visual understanding inherently endows MLLMs with visual generation capabilities. Building on this insight, a straightforward visual predictive instruction tuning (VPiT) strategy was introduced to train LLMs to generate continuous visual tokens, which are subsequently decoded into plausible images using a modified diffusion model. Accordingly, MetaMorph achieved competitive performance on both image understanding and generation tasks with minimal architectural changes and computational overhead. Nevertheless, previous unified models have predominantly targeted static images and show limited capability when it comes to processing videos, where accurately modeling temporal dynamics, ensuring motion coherence, and maintaining frame-wise consistency are essential. Furthermore, the elevated data and computational demands for video modeling impose significant scalability constraints on current unified frameworks. Consequently, delivering an appropriate framework for unified video understanding and generation is critically important for immersive media understanding, production, and even interaction. To fill the vacant, this technical report proposes Omni-Video , an efficient and effective framework for unified video understanding and generation that combines a lightweight architectural design with a multi-stage training scheme. Borrowing the exceptional video understanding and generation capabilities of existing MLLMs ( e.g., VILA lin2024vila ) and text-to-video (T2V) diffusion ( e.g., Wan wan2025wan ) models, our method aims at teaching existing video understanding MLLMs to produce video tokens that are ‘understandable’ for existing T2V diffusion models. More specifically, we first tuning MLLMs to produce textual tokens and visual tokens with separate text and vision heads following tong2024metamorph , the vision head generates continuous video tokens that is aligned with the representation of a pretrained vision encoder, e.g., SigLIP2 tschannen2025siglip2 . Then, we adapt the continuous video tokens into the T2V diffusion model’s conditional space with a dedicated lightweight adapter, such that the T2V model could synthesize faithful videos directly from the continuous video tokens. Notably, only a small subset of model parameters, i.e., the text and vision head in stage-1 and the adapter in stage-2, are optimized. To further activate the potential visual generation capability of MLLMs, we unlock the parameters of text and vision head, adapter, as well as the T2V model and train the whole system with meticulously curated high-quality understanding and generation data, thereby improving the synthesis quality and the understanding ability. This selective fine-tuning strategy requires only minimal data and computational resources, allowing for efficient adaptation without compromising the foundational performance of the pretrained model. In addition to generate harmonious videos conditioned on user instructions, we also extend our unified framework towards detail-preserving visual editing scenarios. One straightforward solution is to feed original signals into the visual encoder of MLLMs and retrain the unified model to reconstruct unedited areas and penalize editing areas with paired editing data yu2024promptfix ; zhao2024ultraedit ; zhang2024hive . This approach, however, incurs substantial computational overhead with massive data and often leads to suboptimal performance as MLLMs primary focus on high-level semantic features rather low-level details. Since T2V diffusion models are trained to recover clean videos from corrupted signals, they inherently excel at processing fine-grained visual information. Accordingly, we leverage pretrained VAEs to extract low-level compressed priors from original input and