Many-for-Many: Unify the Training of Multiple Video and Image Generation and Manipulation Tasks

Tao Yang, Ruibin Li, Yangming Shi, Yuqi Zhang, Qide Dong, Haoran Cheng, Weiguo Feng, Shilei Wen, Bingyue Peng, Lei Zhang · 2025

Read Many-for-Many: Unify the Training of Multiple Video and Image Generation and Manipulation Tasks by Tao Yang, Ruibin Li, Yangming Shi, Yuqi Zhang, Qide Dong, Haoran Cheng, Weiguo Feng, Shilei Wen, Bingyue Peng, Lei Zhang (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2506-01758.txt.

Introduction

Ruibin Li, Tao Yang and a team of co-workers wrote a 2025 paper on one model that learns many video and image tasks together. Most generators train for a single job, like text-to-video. Retraining a giant base model for each new task is costly. Their framework pools training data across tasks instead.

A light adapter unifies the varied inputs each task brings. Training mixes images and video in stages, building the model from scratch. Depth maps join as a cue to sense 3D space. Two sizes were trained, each handling over ten tasks. The larger one rivals top open and commercial video engines.

Catalog record

CREATORTao Yang, Ruibin Li, Yangming Shi, Yuqi Zhang, Qide Dong, Haoran Cheng, Weiguo Feng, Shilei Wen, Bingyue Peng, Lei Zhang
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE55 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2506-01758

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Many-for-Many: Unify the Training of Multiple Video and Image Generation and Manipulation Tasks Tao Yang, Ruibin Li, Yangming Shi, Yuqi Zhang, Qide Dong, Haoran Cheng, Weiguo Feng, Shilei Wen, Bingyue Peng, Lei Zhang Abstract Diffusion models have shown impressive performance in many visual generation and manipulation tasks. Many existing methods focus on training a model for a specific task, especially, text-to-video (T2V) generation, while many other works focus on finetuning the pretrained T2V model for image-to-video (I2V), video-to-video (V2V), image and video manipulation tasks, etc . However, training a strong T2V foundation model requires a large amount of high-quality annotations, which is very costly. In addition, many existing models can perform only one or several tasks. In this work, we introduce a unified framework, namely many-for-many , which leverages the available training data from many different visual generation and manipulation tasks to train a single model for those different tasks. Specifically, we design a lightweight adapter to unify the different conditions in different tasks, then employ a joint image-video learning strategy to progressively train the model from scratch. Our joint learning leads to a unified visual generation and manipulation model with improved video generation performance. In addition, we introduce depth maps as a condition to help our model better perceive the 3D space in visual generation. Two versions of our model are trained with different model sizes (8B and 2B), each of which can perform more than 10 different tasks. In particular, our 8B model demonstrates highly competitive performance in video generation tasks compared to open-source and even commercial engines. Our models and source codes are available at https://github.com/leeruibin/MfM.git . 1 Introduction Visual data generation has a wide range of applications in industry and our daily lives, such as video games [ 48 ] , advertising [ 59 ] , media content creation [ 34 ] , etc . Along with the great success of text-to-image (T2I) generation models [ 38 , 40 , 33 , 11 ] , video generation techniques [ 30 , 57 , 34 , 29 , 21 , 7 , 45 ] have recently witnessed significant progress driven by the rapid development of diffusion models (DMs) [ 15 , 40 , 31 , 27 ] . Current research is preliminarily focused on text-to-video (T2V) generation. Early attempts [ 13 , 2 , 1 ] are often built on pre-trained T2I models such as Stable Diffusion (SD) [ 40 ] by encoding motion dynamics into latent codes [ 20 ] or inserting additional temporal layers [ 13 , 2 , 1 ] . Despite significant advancements, these methods tend to produce unnatural motions and are limited by the small number of generated frames. Recently, diffusion transformers (DiT) [ 31 , 11 , 57 ] have been widely adopted in numerous image and video generation methods [ 11 , 23 , 30 , 45 ] due to their excellent scalability. In particular, SORA [ 30 ] demonstrates remarkable performance in the creation of highly realistic videos, sparking widespread interest in the community and inspiring many subsequent T2V works [ 57 , 29 , 21 , 45 , 42 , 22 ] . For example, trained on web-scale datasets, the open-source models CogVideoX [ 57 ] , HunyuanVideo [ 21 ] and Wan2.1 [ 45 ] have attracted significant attention. The commercial models Runway [ 42 ] and Kling [ 22 ] have demonstrated impressive performance in practical use. Other video generation tasks, such as image-to-video (I2V) and video-to-video (V2V), are commonly regarded as downstream problems of T2V. By fine-tuning pre-trained T2V models with relatively small resources [ 57 , 21 , 45 ] , various models tailored to different tasks can be obtained, including I2V [ 47 , 45 ] , video super-resolution [ 51 ] , reference-to-video [ 28 , 19 ] , etc . In this work, we aim to economically train a model from scratch, which can, however, perform a number of visual generation and manipulation tasks effectively, including T2V, I2V, V2V, etc . To this end, we introduce a simple yet effective framework, called Many-for-Many (MfM in short), to unify the training of different tasks. The key difference between the various visual generation/manipulation tasks lies in their varying conditions. Therefore, we propose to standardize the conditions using a lightweight adapter, thereby enabling multi-task joint training. Adhering to the foundation model’s training recipe, we progressively update our MfM model from a low resolution to higher resolutions. Specifically, we employ a joint image-video learning strategy, which equips our model with capabilities for both image generation and manipulation. An advantage of our MfM training framework is that the many data that cannot be used to train T2V models in previous methods now can be used to train our unified model. Therefore, MfM learning not only leads to a unified model but also enhances video generation performance. Additionally, we incorporate depth maps as a condition to improve our model’s understanding of 3D space in the real world. Two versions (2B and 8B) of our MfM model are trained. As shown in Table 1 , our model can perform more than 10 different visual generation and manipulation tasks. Figure 1 illustrates some examples of MfM tasks. Extensive experiments are performed to demonstrate the effectiveness and flexibility of our MfM model. In particular, our 8B model achieves highly competitive performance in the challenging T2V and I2V tasks by using only 10% of the training data used in state-of-the-art open-source T2V foundation models [ 57 , 21 , 45 ] . Table 1 : The size and supported tasks of the current main open-source video foundation models. Model Size Training Data Supported Tasks Unified Training Video Image CogVideoX [ 57 ] 28&5B unkown unkown T2V, I2V ✗ MovieGen [ 34 ] 30B 100M 1B T2V, Peronalized T2V (PT2V) ✗ StepVideo [ 29 ] 30B 2B 3.8B T2V, I2V ✗ HunyuanVideo [ 21 ] 13B \mathcal{O}(100) M \mathcal{O}(1) B T2V, I2V ✗ Wan2.1 [ 45 ] 1.3B&14B 1.5B 10B T2V, I2V ✗ MfM 2B&8B 120M 160M T2V, I2V, video extension, FLF2V, FLC2V, video manipulation, etc. ✓ Figure 1 : Examples of MfM on typical video generation and manipulation tasks. Generated frames are highlighted within black boxes. Note that MfM uses a single model to perform these tasks. 2 Related Work Diffusion Models for Visual Generation . Since the seminal work of denoising diffusion probabilistic model (DDPM) [ 15 ] , remarkable progress has been achieved in training diffusion models (DMs) for image and video generation [ 40 , 11 , 1 , 30 , 21 ] . In particular, Rombach et al . [ 40 ] proposed to train DMs in latent space, achieving impressive image generation results with significantly reduced computational costs. The development of Stable Diffusion (SD) [ 40 ] has sparked a surge of research in text-to-image (T2I) generation [ 33 , 61 , 41 ] . SDXL [ 33 ] expands SD by using a larger model and more sophisticated architecture design. With the advancement in Diffusion Transformer (DiT) [ 31 ] and Flow Matching (FM) [ 27 ] , Esser et al. [ 11 ] proposed MMDiT to scale up the pre-training of T2I models, resulting

Read the complete text