SyncVP: Joint Diffusion for Synchronous Multi-Modal Video Prediction
Enrico Pallotta, Sina Mokhtarzadeh Azar, Shuai Li, Olga Zatsarynna, Juergen Gall · 2025
Read SyncVP: Joint Diffusion for Synchronous Multi-Modal Video Prediction by Enrico Pallotta, Sina Mokhtarzadeh Azar, Shuai Li, Olga Zatsarynna, Juergen Gall (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2503-18933.txt.
Introduction
Enrico Pallotta and four co-authors present a video prediction system from 2025 that forecasts the next frames of a scene in more than one form at once. Plain color video alone often lacks clues about depth and motion that smart systems need to plan ahead. Their system, called SyncVP, predicts color and a second data type together, in step.
SyncVP builds on trained diffusion models for each data type and adds a sharing module that lets the streams trade clues across space and time. Tests used city street scenes and robot arm clips, with depth as the extra stream. The team also tried meaning maps and weather data. Results reached the best scores reported on the standard test sets they used.
Catalog record
| CREATOR | Enrico Pallotta, Sina Mokhtarzadeh Azar, Shuai Li, Olga Zatsarynna, Juergen Gall |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 41 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2503-18933
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
SyncVP: Joint Diffusion for Synchronous Multi-Modal Video Prediction Enrico Pallotta, Sina Mokhtarzadeh Azar, Shuai Li, Olga Zatsarynna, Juergen Gall Abstract Predicting future video frames is essential for decision-making systems, yet RGB frames alone often lack the information needed to fully capture the underlying complexities of the real world. To address this limitation, we propose a multi-modal framework for Synchronous Video Prediction (SyncVP) that incorporates complementary data modalities, enhancing the richness and accuracy of future predictions. SyncVP builds on pre-trained modality-specific diffusion models and introduces an efficient spatio-temporal cross-attention module to enable effective information sharing across modalities. We evaluate SyncVP on standard benchmark datasets, such as Cityscapes and BAIR, using depth as an additional modality. We furthermore demonstrate its generalization to other modalities on SYNTHIA with semantic information and ERA5-Land with climate data. Notably, SyncVP achieves state-of-the-art performance, even in scenarios where only one modality is present, demonstrating its robustness and potential for a wide range of applications. Figure 1 : SyncVP is a diffusion model for synchronized multi-modal video prediction. It generates multi-modal future frames like RGB and depth for a given observation that can consist of both modalities (left) or only one modality (right). 1 Introduction Video prediction, the task of forecasting future video frames based on past video frames, has gained significant attention in recent years [ 44 , 3 , 39 , 42 ] due to its broad range of applications. Autonomous driving, weather forecasting, healthcare and human-machine interaction are just a few examples of scenarios in which the ability to anticipate future events is critical. In these contexts, accurate video prediction enables systems to react and adapt in real-time, enhancing both safety and efficiency or providing valuable information for decision-making processes. While traditional video prediction focuses on generating future frames of RGB videos, many real-world applications involve multiple sensory inputs and require a deep understanding of the underlying real world dynamics. This leads to a natural extension of the task to the multi-modal domain, where additional data such as depth or semantic information are used alongside RGB frames. Several works have thus investigated how to exploit other modalities for conditioning the generation of images or videos, like text [ 5 , 36 , 58 , 28 , 48 , 14 ] , depth [ 49 , 21 , 24 , 13 , 57 ] , pose [ 29 , 57 ] , flow [ 32 ] , sketches [ 43 ] and audio [ 35 , 50 ] . Although significant progresses have been made in guiding video generation through the use of auxiliary modalities, few studies have focused on developing models capable of leveraging multiple modalities while simultaneously generating each of them. In this paper, we thus propose an approach for multi-modal video prediction as shown in Fig. 1 . Given a set of observed frames, the model predicts future frames of two modalities like RGB and depth in a consistent manner. While our approach is based on a latent diffusion model [ 56 ] , we demonstrate that a naive concatenation of the modalities performs poorly since each modality has its own characteristics. Instead, we propose a pair of synchronized diffusion models that share relevant information through an efficient cross-attention mechanism, which works separately on the spatial and temporal dimensions. Another key aspect is the noise sharing between the modalities, which not only ensures synchronous predictions across the modalities but also has a very positive impact on the loss convergence. Finally, we propose a novel cross-modality guidance approach for training, which enables the model to predict multi-modal frames even if only one modality is observed as shown in Fig. 1 . We summarize the contributions of our Synchronous multi-modal Video Prediction framework (SyncVP) in the following key points: • A generalized and scalable multi-modal framework for video prediction that can exploit pre-trained modality specific diffusion models with little finetuning. • We introduce a lightweight and efficient module for cross-modality information exchange through a split spatio-temporal cross-attention mechanism that computes only a single shared attention matrix. • We propose to use a shared forward diffusion process by applying the same noise to each modality. This method leads to significant improvements in loss convergence and conditional generation performance compared to using independent noises. • We propose cross-modality guidance , a joint modality training technique that enables simultaneous multi-modal video prediction also with partial conditioning. By randomly masking one or both modality inputs during training, the model learns to predict future frames even when one modality is missing. 2 Related Work We discuss related works for video prediction and multi-modal generation, as this work bridges these two tasks and establishes a multi-modal framework for synchronous video prediction. 2.1 Video prediction Early approaches for video frames prediction primarily focused on recurrent networks like ConvLSTM and RNNs [ 6 , 40 , 44 ] . Due to the inherent uncertain nature of future frames in a video, several works proposed stochastic methods based on variational models (VAEs, VRNNs) [ 47 , 7 , 11 , 41 , 1 , 2 , 3 , 4 ] . GANs have also proven effective in video prediction [ 9 , 39 , 27 , 22 ] . Other approaches tackled this task as a neural process [ 54 ] mapping input spatio-temporal coordinates to target pixel values, using tailored video Transformers [ 53 ] or, inspired by the human vision system [ 19 ] , defining a model for the frequency domain through the use of a multi-level wavelet transform. In the last three years, Diffusion models have been dominating the field of image generation [ 16 , 33 ] , and they have shown outstanding performances on videos [ 17 , 5 ] . Based on the diffusion paradigm, several works proposed models for video prediction [ 18 , 42 , 55 , 59 , 26 ] , each of them proposing different backbone models, past frames conditioning techniques or ways to model the spatio-temporal dependencies. Although these studies have progressively improved the quality of generated videos, none has yet focused on the integration and exploitation of multi-modal information. 2.2 Multi-modal generation Multi-modal generation is the task of generating semantically aligned outputs across different data modalities. In the image domain, we find several works that aim to learn the joint distribution of multiple modalities. LDM3D [ 38 ] trains a diffusion model for generating an RGB+D image from text by concatenating the RGB and depth images along the channel dimension before feeding the resulting vector to the latent encoder. MT-Diffusion [ 8 ] instead defines a common diffusion space by aggregating the latent representations of each modality, which are separately encoded. DiffX [ 45 ] proposes a multi-path VAE to encode and decode all the modalities in a single shared latent space before training a diffusion model for layout-guidance. HyperHuman [ 25 ] uses a joint diffusion UNet with expert branches of RGB, depth and surface normal for image generation. The generated depth and normal images are then used in a second refining step as conditioning for generating an RGB image with higher resolution. Finally, Xing et al. [ 50 ] propose an inference time optimization technique that uses a pre-trained