TiP4GEN: Text to Immersive Panorama 4D Scene Generation Conference: Proceedings of the 33rd ACM International Conference on Multimedia; October 27–31, 2025; Dublin, Ireland Proceedings of the 33rd ACM International Conference on Multimedia (MM ’25), October 27–31, 2025, Dublin, Ireland DOI: 10.1145/

Ke Xing, Hanwen Liang, Dejia Xu, Yuyang Yin, Konstantinos N. Plataniotis, Yao Zhao, Yunchao Wei · 2025

Read TiP4GEN: Text to Immersive Panorama 4D Scene Generation Conference: Proceedings of the 33rd ACM International Conference on Multimedia; October 27–31, 2025; Dublin, Ireland Proceedings of the 33rd ACM International Conference on Multimedia (MM ’25), October 27–31, 2025, Dublin, Ireland DOI: 10.1145/ by Ke Xing, Hanwen Liang, Dejia Xu, Yuyang Yin, Konstantinos N. Plataniotis, Yao Zhao, Yunchao Wei (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2508-12415.txt.

Introduction

Xing and co-authors introduce TiP4GEN, a system that turns text into moving 360-degree scenes for virtual reality. Demand for rich immersive scenes is rising fast. Past generators made static scenes or moving ones from a narrow view. A true panorama that wraps the viewer was missing. TiP4GEN joins panorama video generation with dynamic scene rebuilding. The result is a 4D scene that can be entered from any angle.

A dual-branch model drives the video step. One branch handles the global panorama, the other the local view. Cross-attention runs both ways to keep them in sync. Fine control over content is a stated aim of the frame.

Catalog record

CREATORKe Xing, Hanwen Liang, Dejia Xu, Yuyang Yin, Konstantinos N. Plataniotis, Yao Zhao, Yunchao Wei
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE45 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2508-12415

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

TiP4GEN: Text to Immersive Panorama 4D Scene Generation Conference: Proceedings of the 33rd ACM International Conference on Multimedia; October 27–31, 2025; Dublin, Ireland Proceedings of the 33rd ACM International Conference on Multimedia (MM ’25), October 27–31, 2025, Dublin, Ireland DOI: 10.1145/3746027.3754704 ISBN: 979-8-4007-2035-2/2025/10 CCS: Computing methodologies Computer vision Ke Xing, Hanwen Liang, Dejia Xu, Yuyang Yin, Konstantinos N. Plataniotis, Yao Zhao, Yunchao Wei Abstract . With the rapid advancement and widespread adoption of VR/AR technologies, there is a growing demand for the creation of high-quality, immersive dynamic scenes. However, existing generation works predominantly concentrate on the creation of static scenes or narrow perspective-view dynamic scenes, falling short of delivering a truly 360-degree immersive experience from any viewpoint. In this paper, we introduce TiP4GEN , an advanced text-to-dynamic panorama scene generation framework that enables fine-grained content control and synthesizes motion-rich, geometry-consistent panoramic 4D scenes. TiP4GEN integrates panorama video generation and dynamic scene reconstruction to create 360-degree immersive virtual environments. For video generation, we introduce a Dual-branch Generation Model consisting of a panorama branch and a perspective branch, responsible for global and local view generation, respectively. A bidirectional cross-attention mechanism facilitates comprehensive information exchange between the branches. For scene reconstruction, we propose a Geometry-aligned Reconstruction Model based on 3D Gaussian Splatting. By aligning spatial-temporal point clouds using metric depth maps and initializing scene cameras with estimated poses, our method ensures geometric consistency and temporal coherence for the reconstructed scenes. Extensive experiments demonstrate the effectiveness of our proposed designs and the superiority of TiP4GEN in generating visually compelling and motion-coherent dynamic panoramic scenes. Our project page is at https://ke-xing.github.io/TiP4GEN/ . Figure 1. We introduce TiP4GEN, an advanced text-to-dynamic panorama scene generation framework capable of producing 360-degree immersive and high-fidelity 4D scenes. Given a global text prompt and four local text prompts, TiP4GEN generates visually pleasant, motion-rich panoramic videos with fine-grained content control and then explicitly reconstructs panorama 4D scenes with high spatial-temporal geometry consistency. 1. Introduction As VR/AR technologies advance at an accelerated pace and gain broader acceptance, immersive reality solutions are finding growing applications across industries such as gaming, entertainment, and education. This growth drives a rising demand for high-quality, immersive dynamic panorama scene generation. In recent years, diffusion-based generative models have advanced significantly from image and video generation ( Rombach et al. 2022 ; Blattmann et al. 2023a ) to 3D and 4D ( Poole et al. 2022 ; Yin et al. 2023 ; Wu et al. 2024 ) asset creation. While recent work has expanded from object-centric asset generation to scene synthesis, most research has focused mainly on static 3D scenes ( Zhou et al. 2024 ; Yang et al. 2024 ; Chung et al. 2023 ; Liang et al. 2024a ) , with relatively less attention to dynamic 4D scenes. This is primarily due to the scarcity of large-scale 4D scene datasets. Also, existing 4D scene generation methods ( Lee et al. 2024 ; Xu et al. 2024 ; Wang et al. 2024c ; Liang et al. 2024b ) are often constrained to narrow perspectives and struggle to maintain global consistency in full 360-degree views, thereby restricting their applicability to immersive scene experiences. Recent works have made attempts to enhance immersion and achieved promising results by generating panoramic video from text ( Wang et al. 2024a ) or perspective videos ( Tan et al. 2024 ) , or by animating panoramic images ( Li et al. 2024 ) . However, there are several limitations with current approaches. First, although panoramic scenes offer a much wider field of view than perspective scenes, prior works often generate simplistic scene compositions where a single object (e.g., a mountain or a lake) dominates the entire scene, lacking fine-grained control over local view variations. These methods typically rely on a single global text prompt, enforcing uniform content distribution and leading to homogeneous, less diverse scenes. Second, the spherical visual and motion patterns in panoramic video pose challenges for directly finetuning traditional perspective video diffusion models for panorama generation ( Tan et al. 2024 ) . Unlike perspective video generation models benefiting from training on web-scale datasets, the scarcity of panoramic video data makes it non-trivial to train a well-generalized model capable of producing content-diverse and motion-coherent panoramic videos. Furthermore, in the 4D construction stage, prior work ( Li et al. 2024 ) assumes a static camera and models per-timestamp geometry independently without explicit camera pose estimation, making it difficult to maintain temporal coherence, especially when the scene undergoes substantial movements. To address these limitations, we propose TiP4GEN , an advanced text-to-dynamic panorama scene generation framework that enables fine-grained content control and generates motion-rich and geometry-consistent 4D scenes. To support novel view synthesis and deliver immersive viewing experiences, our framework adopts a two-stage approach: first generating semantically rich panoramic videos and then reconstructing the dynamic 4D scenes. In the video generation phase, considering the limited panoramic video datasets, we take advantage of pretrained perspective video diffusion models and propose a Dual-branch Generation Model . This model comprises a panorama branch, which maintains global consistency, and a perspective branch, which enhances local detail and diversity by leveraging pretrained priors. A bidirectional cross-attention mechanism is adopted between the two branches to facilitate information exchange and improve the whole scene coherence. The perspective branch generates multiple views at the same time, each guided by a distinct local text prompt, enabling fine-grained control over panoramic scene composition. In the dynamic scene reconstruction phase, we introduce Geometry-aligned Reconstruction Model , which employs 3D Gaussian Splatting (3DGS) ( Kerbl et al. 2023 ) sets to represent the panoramic scenes across timestamps. Individually reconstructing the scene of each frame overlooks the temporal continuity in video sequences, resulting in inconsistencies in the reconstructed geometry across time. Moreover, within a single panoramic frame, multiple perspective views should also share a unified geometry under the same metric space. To address these issues and enhance spatial-temporal coherence, we introduce spatial and temporal alignment modules that leverage estimated depths ( Ranftl et al. 2021 ; Zhang et al. 2024a ) to align geometry consistently across both viewpoints and timestamps. To mitigate the inconsistency caused by camera motion, we also estimate camera poses to initialize the scene poses, followed by per-scene GS optimization. Our contributions can be summarized as follows: • We propose TiP4GEN , a novel text-to-dynamic panorama scene generation framework that synthesizes immersive 360-degree, motion-rich, and geometry-consistent 4D scenes with fine-grained content control. • We introduce a Dual-branch Generation Model that jointly leverages panoramic and perspective branches to integrate global scene consistency and local detail diversity, with a bidirectional spatial-temporal cross-attention mechanism enabling coherent content fusion. • We develop a Geometry-aligned Reconstruction Model that reconstructs dynamic panoramic 4D scenes using Gaussian Splatting, incorporating spatial-temporal alignment and camera pose initialization to largely enhance the geometry consistency across time and views. • Extensive experiments demonstrate that our method significantly outperforms existing methods in generating semantically rich, motion-coherent, and geometrically consistent dynamic 4D panoramic scenes. 2. Related Works 2.1. Video Diffusion Model In recent years, diffusion models ( Ho et al. 2020 ; Song et al. 2020 ; Sohl-Dickstein et al. 2015 ; Song and Ermon 2019 ) have achieved remarkable success in the field of image generation (

Read the complete text