VideoPanda : Video Pan oramic D iffusion with Multi-view A ttention
Kevin Xie Amirmojtaba Sabour ∗ Jiahui Huang Despoina Paschalidou Greg Klar Umar Iqbal Sanja Fidler Xiaohui Zeng NVIDIA · 2025
Read VideoPanda : Video Pan oramic D iffusion with Multi-view A ttention by Kevin Xie Amirmojtaba Sabour ∗ Jiahui Huang Despoina Paschalidou Greg Klar Umar Iqbal Sanja Fidler Xiaohui Zeng NVIDIA (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2504-11389.txt.
Introduction
Kevin Xie and co-authors present a system that makes full surround video from text or a single normal video. Their 2025 paper adds cross-view attention layers to a video model, so the views it makes agree with each other. Joined views form a smooth panorama fit for virtual reality headsets.
Training used both text prompts and single-view clips as guides. The system can also extend a video step by step to make longer pieces. To keep training costs sane, the team sampled fewer frames and views, yet the model still handled more at test time. On real and fake test sets, its panoramas looked more real and held together better than past methods.
Catalog record
| CREATOR | Kevin Xie Amirmojtaba Sabour ∗ Jiahui Huang Despoina Paschalidou Greg Klar Umar Iqbal Sanja Fidler Xiaohui Zeng NVIDIA |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 42 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2504-11389
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
VideoPanda : Video Pan oramic D iffusion with Multi-view A ttention Kevin Xie Amirmojtaba Sabour ∗ Jiahui Huang Despoina Paschalidou Greg Klar Umar Iqbal Sanja Fidler Xiaohui Zeng NVIDIA Abstract High resolution panoramic video content is paramount for immersive experiences in Virtual Reality, but is non-trivial to collect as it requires specialized equipment and intricate camera setups. In this work, we introduce VideoPanda, a novel approach for synthesizing 360 ∘ 360^{\circ} videos conditioned on text or single-view video data. VideoPanda leverages multi-view attention layers to augment a video diffusion model, enabling it to generate consistent multi-view videos that can be combined into immersive panoramic content. VideoPanda is trained jointly using two conditions: text-only and single-view video, and supports autoregressive generation of long-videos. To overcome the computational burden of multi-view video generation, we randomly subsample the duration and camera views used during training and show that the model is able to gracefully generalize to generating more frames during inference. Extensive evaluations on both real-world and synthetic video datasets demonstrate that VideoPanda generates more realistic and coherent 360 ∘ 360^{\circ} panoramas across all input conditions compared to existing methods. Visit the project website at https://research.nvidia.com/labs/toronto-ai/VideoPanda/ for results. Figure 1: Generated samples conditioned on a single-view video and text prompt. Both single-view video inputs were generated using existing video generation models ( Brooks et al. 2024 ; Runway 2024 ) . Auto-regressive generation is applied to extend the video length. 1 Introduction A key aspect of achieving true immersion in a virtual environment is allowing users to look around freely, by rotating their head and exploring their surroundings from all possible angles. To enable such experiences, it is essential to have access to high-quality and high-resolution panoramic videos. However, recording such videos is both expensive and time-consuming, as it requires intricate camera setups and specialized equipment. As a result, the available panoramic video content on platforms such as YouTube or Vimeo remains limited compared to single-view videos. In this work, we aim to address this issue, by developing a generative model capable of synthesizing panoramic videos either from text prompts or by expanding single-view videos (either generated from models like Sora ( Brooks et al. 2024 ) or recorded) into panoramic format. We consider this an essential step towards making immersive content more accessible and scalable. Recently, diffusion models have shown remarkable success in generating images ( Ho et al. 2022 ; Blattmann et al. 2023a ) , 3D models ( Shi et al. 2023b ; Poole et al. 2022 ) , and videos ( Brooks et al. 2024 ; Blattmann et al. 2023b ) from text prompts. Despite their promising capabilities, generation of panoramic videos using diffusion models presents significant challenges, mainly due to the scarcity of high-quality panoramic video datasets. Furthermore, while substantial progress has been made towards advancing standard video generation pipelines ( Girdhar et al. 2023 ; Hong et al. 2022 ; Chen et al. 2024 ; Zheng et al. 2024 ) , very few works have attempted to apply these techniques to panoramic video generation. Existing methods are either limited to specific domains such as driving scenarios ( Wen et al. 2024 ; Wu et al. 2024 ; Li et al. 2023 ; Zhao et al. 2024 ; Liu et al. 2024b ) or restricted to generating static scenes ( Wu et al. 2023 ; Zhang et al. 2024 ) . 360DVD ( Wang et al. 2024a ) directly generates equirectangular panorama video (with text condition), which presents a large domain gap to base model pretrained on perspective view. We perform an extensive comparison to 360DVD in the text-conditional setting and demonstrate our improved visual quality. In this paper, we introduce VideoPanda , a novel approach capable of generating high-quality panoramic videos from text prompts and single-view video, as well as creating long video using auto-regression. Our approach builds on existing video diffusion models by adding multi-view attention layers to generate consistent multi-view outputs. Doing so ensures that the output domain (perspective images) remains close to the original training distribution of the pretrained video model (as opposed to directly generating equirectangular projections), which helps in maintaining video quality while generating multiple views. The resulting views are then seamlessly stitched together to create a cohesive panoramic video. We evaluate our model on a diverse set of data domains, including both real and synthetic videos, and demonstrate its superior performance and quality compared to previous approaches, both quantitatively and qualitatively. Additionally, a user study indicates that the majority of participants prefer our generated videos over those from other baseline models. In summary, we make the following contributions: • We identify the value of panoramic video generation by allowing users to input single-view videos as a condition – a widely available modality, and present a multi-view video architecture capable of generating plausible panoramic videos. • We demonstrate that our model can be jointly trained for text-conditioning, video-conditioning, and autoregressive settings by randomizing the conditioning type, leading to improved results and enabling the generation of long panoramic videos. • When extending the video model to multi-view, the number of generated image frames greatly increases. We overcome the inherent computational burden associated by randomly subselecting the number of views and frames and show that it gracefully generalizes to video with long duration and more views during inference. 2 Related Work 2.1 Image and Video Diffusion Models Diffusion models ( Ho et al. 2020 ) have demonstrated remarkable success in generating high-quality images ( Karras et al. 2022 ; Karras et al. 2024 ; Pernias et al. 2023 ; Hoogeboom et al. 2023 ; Ho et al. 2022 ) and videos ( Girdhar et al. 2023 ; Hong et al. 2022 ; Blattmann et al. 2023a ; Blattmann et al. 2023b ; Brooks et al. 2024 ; Guo et al. 2023 ; Chen et al. 2023 ; Gupta et al. 2023 ) from text prompts. To reduce the computational cost of generating high-dimensional data such as images and videos, latent diffusion models ( Rombach et al. 2022 ) (LDMs) proposed to first encode the data into a compressed latent space using a variational autoencoder (VAE) ( Kingma 2013a ) , and then conduct the diffusion in this lower dimensional space. These models have been proven highly effective for a wide range of downstream tasks such as inpainting ( Lugmayr et al. 2022 ) , controllable generation ( Zhang et al. 2023 ) , customized generation ( Ruiz et al. 2023 ) , and image/video editing ( Kawar et al. 2023 ; Molad et al. 2023 ) etc. 2.2 Multi-View Image Generation Building on the success of diffusion models for 2D image generation, they have been increasingly adapted also for multi-view image generation. However, due to the limited availability of real-world multi-view training data, several recent approaches ( Shi et al. 2023b ; Long et al. 2024 ; Liu et al. 2023b ; Liu et al. 2023a ) attempted to fine-tune pretrained image generation models like Stable Diffusion ( Rombach et al. 2022 ) to support multi-view generation. Such approaches can be roughly categorized into two categories: object-centric and scene-centric approaches. Object-centric models focus primarily on generating images of objects where all cameras are inward-facing,