HoloTime: Taming Video Diffusion Models for Panoramic 4D Scene Generation

Haiyang Zhou, Wangbo Yu, Jiawen Guan, Xinhua Cheng, Yonghong Tian, Li Yuan · 2025

Read HoloTime: Taming Video Diffusion Models for Panoramic 4D Scene Generation by Haiyang Zhou, Wangbo Yu, Jiawen Guan, Xinhua Cheng, Yonghong Tian, Li Yuan (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2504-21650.txt.

Introduction

Haiyang Zhou and five co-authors built HoloTime in two thousand twenty-five to turn a prompt or single image into an immersive panoramic four-dimensional scene. Virtual and augmented reality need scene-level moving assets, but diffusion models mostly make static scenes or single moving objects. Their frame first generates a panoramic video, then rebuilds it as a four-dimensional asset.

They curated 360World, a data set of panoramic videos fit for scene rebuilding. A two-stage animator turns panoramic stills into high-quality panoramic video. A space-time depth method then lifts the video into a point cloud and a Gaussian splatting scene. Comparisons show gains over prior methods in both video and scene quality.

Catalog record

CREATORHaiyang Zhou, Wangbo Yu, Jiawen Guan, Xinhua Cheng, Yonghong Tian, Li Yuan
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE44 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2504-21650

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

HoloTime: Taming Video Diffusion Models for Panoramic 4D Scene Generation Haiyang Zhou, Wangbo Yu, Jiawen Guan, Xinhua Cheng, Yonghong Tian, Li Yuan Abstract . The rapid advancement of diffusion models holds the promise of revolutionizing the application of VR and AR technologies, which typically require scene-level 4D assets for user experience. Nonetheless, existing diffusion models predominantly concentrate on modeling static 3D scenes or object-level dynamics, constraining their capacity to provide truly immersive experiences. To address this issue, we propose HoloTime , a framework that integrates video diffusion models to generate panoramic videos from a single prompt or reference image, along with a 360-degree 4D scene reconstruction method that seamlessly transforms the generated panoramic video into 4D assets, enabling a fully immersive 4D experience for users. Specifically, to tame video diffusion models for generating high-fidelity panoramic videos, we introduce the 360World dataset , the first comprehensive collection of panoramic videos suitable for downstream 4D scene reconstruction tasks. With this curated dataset, we propose Panoramic Animator , a two-stage image-to-video diffusion model that can convert panoramic images into high-quality panoramic videos. Following this, we present Panoramic Space-Time Reconstruction , which leverages a space-time depth estimation method to transform the generated panoramic videos into 4D point clouds, enabling the optimization of a holistic 4D Gaussian Splatting representation to reconstruct spatially and temporally consistent 4D scenes. To validate the efficacy of our method, we conducted a comparative analysis with existing approaches, revealing its superiority in both panoramic video generation and 4D scene reconstruction. This demonstrates our method’s capability to create more engaging and realistic immersive environments, thereby enhancing user experiences in VR and AR applications. More results are available on the project page: https://zhouhyocean.github.io/holotime/ . Figure 1. HoloTime accepts either a user-provided or model-generated panoramic image as input, and transforms it into an immersive 360-degree 4D scene, enabling a virtual roaming experience. 1. Introduction The advancement of diffusion models for text-to-image generation ( Ho et al., 2020 ; Rombach et al., 2022 ) has sparked a burgeoning interest in video and 3D content creation, leading to significant milestones. Building upon the foundation of 3D content generation, 4D content generation—which integrates the temporal dimension into 3D data—holds immense potential for applications and promises to revolutionize the fields of Augmented Reality (AR) and Virtual Reality (VR). Scene-level 4D content generation has the potential to provide users with a more liberating and immersive experience, allowing for greater freedom in virtual navigation. However, unlike the advancements in video and 3D content generation, 4D content generation has lagged behind due to the scarcity of high-quality annotated data, particularly for large-scale 4D scenes. Consequently, current 4D generation methods are primarily limited to generating object-level dynamics ( Bahmani et al., 2024 ; Jiang et al., 2024 ; Ren et al., 2024 ; Ling et al., 2024 ; Yin et al., 2024 ) or creating forward-facing scenes with restricted viewing freedom ( Wu et al., 2024 ) . To cultivate a 360-degree immersive experience, an effective strategy is to harness the 360° field of view provided by panoramic images. While established methods ( Zhou et al., 2025 ; Zhou et al., 2024 ; Ma et al., 2024b ) already utilize panoramic images for 3D generation, introducing dynamics to these images for 4D generation is not straightforward due to the lack of large-scale panoramic video training data. To address these challenges, we introduce HoloTime , a 360-degree 4D scene generation framework. It accepts a user-provided or model-generated ( Labs, 2023 ) panoramic image as input, and leverage a Panoramic Animator to transform the panoramic image into a panoramic video featuring realistic visual dynamics and a fixed camera pose. Following this, we present a Panoramic Space-Time Reconstruction technique that reconstructs a high-fidelity 4D scene based on the generated video, thereby enabling an immersive user experience. To build the framework, we first curated a large-scale dataset of fixed-camera panoramic videos, accompanied by corresponding prompt descriptions, which we term the 360World dataset . We then train the Panoramic Animator on this dataset to convert panoramic images into panoramic videos, employing a two-stage motion-guided generation process to enhance its performance. After generating the panoramic video, we utilize the Panoramic Space-Time Reconstruction technique for high-fidelity 4D scene reconstruction. Specifically, we begin by employing a panoramic optical flow estimation model ( Shi et al., 2023 ) in conjunction with a narrow field-of-view depth estimation model ( Ke et al., 2024 ) to perform space-time depth estimation, ensuring depth consistency across both temporal and spatial dimensions. This aligned depth is used to convert the panoramic video into dynamic point clouds, optimizing a spatially and temporally consistent 4D-GS representation, which ultimately provides an immersive experience in virtual and augmented reality environments. Our contributions are encapsulated as follows: • We present Panoramic Animator , accompanied by a two-stage motion-guided generation strategy that seamlessly transforms panoramic images into dynamic panoramic videos, all while preserving the spatial characteristics of the original images, facilitating downstream 4D reconstruction tasks. • We introduce Panoramic Space-Time Reconstruction , which employs cutting-edge techniques to enable the temporal and spatial alignment of depth estimation for panoramic videos, enabling a holistic 4D scene reconstruction through the 4D-GS representations. • We contribute the 360World dataset , the first comprehensive collection of panoramic videos captured with fixed cameras. We believe this dataset can not only helps address gaps in 360-degree 4D scene generation but also holds promise for advancing future 4D generation efforts. 2. Related Works 2.1. Diffusion-based Image and Video Generation. As supported by previous works ( Ho et al., 2020 ; Rombach et al., 2022 ; Saharia et al., 2022 ; Ramesh et al., 2022 ; Nichol et al., 2022 ) , diffusion models have markedly enhanced the capacity to generate 2D images. The application of diffusion models also extends to the domain of video generation. VDM ( Ho et al., 2022b ) first introduced the space-time factorized U-Net architecture, extending the original 2D U-Net for video modeling and generation. Subsequently, many methods ( Ho et al., 2022a ; Singer et al., 2022 ; Zhou et al., 2023 ; Wang et al., 2023b ; Guo et al., 2024 ; Blattmann et al., 2023b ) have followed this similar architecture, achieving significant progress in generating high-quality video content from text. Some models ( Blattmann et al., 2023a ; Zhang et al., 2023 ; Xing et al., 2024 ) utilize image control for video generation to achieve a higher degree of customization and consistency in the content produced. Recent studies ( Kondratyuk et al., 2024 ; Ma et al., 2024a ; Yang et al., 2024b ) have employed the DiT (Diffusion Transformer) architecture for video generation, also achieving impressive performance. Although current video generation is largely confined to perspective videos, 360DVD ( Wang et al., 2024 ) has conducted explorations to panoramic video text-driven generation through training an adapter. Furthermore, Imagine360 ( Tan et al., 2024 ) is capable of lifting input perspective videos to panoramic videos. 2.2. 3D Scene Generation. The emergence of NeRF ( Mildenhall et al., 2021 ) and 3D Gaussian Splatting (3D-GS) ( Kerbl et al., 2023 ) has propelled significant advancements in the field of 3D reconstruction ( Barron et al., 2021 ; Tancik et al., 2022 ;

Read the complete text