π³ \mathcal{X} - S c e n e : Large-Scale Driving Scene Generation with High Fidelity and Flexible Controllability
Yu Yang, Alan Liang, Jianbiao Mei, Yukai Ma, Yong Liu, Gim Hee Lee Β· 2025
Read π³ \mathcal{X} - S c e n e : Large-Scale Driving Scene Generation with High Fidelity and Flexible Controllability by Yu Yang, Alan Liang, Jianbiao Mei, Yukai Ma, Yong Liu, Gim Hee Lee (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2506-13558.txt.
Introduction
Yang and co-authors (2025) present X-Scene, a system that generates large driving scenes for autonomous vehicle development. Diffusion models already synthesize driving data well over time, but building big three-dimensional scenes that stay coherent in space is less explored. X-Scene accepts control at two levels: fine layout direction from user input or text, and broad semantic guidance from user intent enriched by a language model.
The pipeline first builds a 3D semantic map of the scene, then renders matching images and video from several viewpoints, keeping the views aligned. Local scenes are then stretched into large ones by outpainting that extends geometry and imagery from what already exists. The results are lifted into a high-quality 3D representation used for simulation and exploration. In experiments, the authors report clear gains in both control and fidelity over prior scene generation methods.
Catalog record
| CREATOR | Yu Yang, Alan Liang, Jianbiao Mei, Yukai Ma, Yong Liu, Gim Hee Lee |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 69 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2506-13558
Browse the full library Β· About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
π³ \mathcal{X} - S c e n e : Large-Scale Driving Scene Generation with High Fidelity and Flexible Controllability Yu Yang, Alan Liang, Jianbiao Mei, Yukai Ma, Yong Liu, Gim Hee Lee Abstract Diffusion models are advancing autonomous driving by enabling realistic data synthesis, predictive end-to-end planning, and closed-loop simulation, with a primary focus on temporally consistent generation. However, the generation of large-scale 3D scenes that require spatial coherence remains underexplored. In this paper, we propose π³ \mathcal{X} - S c e n e , a novel framework for large-scale driving scene generation that achieves both geometric intricacy and appearance fidelity, while offering flexible controllability. Specifically, π³ \mathcal{X} -Scene supports multi-granular control, including low-level conditions such as user-provided or text-driven layout for detailed scene composition and high-level semantic guidance such as user-intent and LLM-enriched text prompts for efficient customization. To enhance geometrical and visual fidelity, we introduce a unified pipeline that sequentially generates 3D semantic occupancy and the corresponding multiview images, while ensuring alignment between modalities. Additionally, we extend the generated local region into a large-scale scene through consistency-aware scene outpainting, which extrapolates new occupancy and images conditioned on the previously generated area, enhancing spatial continuity and preserving visual coherence. The resulting scenes are lifted into high-quality 3DGS representations, supporting diverse applications such as scene exploration. Comprehensive experiments demonstrate that π³ \mathcal{X} -Scene significantly advances controllability and fidelity for large-scale driving scene generation, empowering data generation and simulation for autonomous driving. 1 Introduction Recent advancements in generative AI have profoundly impacted autonomous driving, with diffusion models (DMs) emerging as pivotal tools for data synthesis and driving simulation. Some approaches utilize DMs as data machines, producing high-fidelity driving videos [ 1 , 2 , 3 , 4 , 5 , 6 , 7 , 8 , 9 , 10 , 11 , 12 , 13 , 14 ] or multi-modal synthetic data [ 15 , 16 , 17 , 18 ] to augment perception tasks, as well as generating corner cases (e.g., vehicle cut-ins) to enrich planning data with uncommon yet critical scenarios. Beyond this, other methods employ DMs as world models to predict future driving states, enabling end-to-end planning [ 19 , 20 , 21 ] and closed-loop simulation [ 22 , 23 , 24 , 25 , 26 , 27 ] . All these efforts emphasize long-term video generation through temporal recursion , encouraging DMs to produce coherent video sequences for downstream tasks. However, large-scale scene generation with spatial expansion , which aims to build expansive and immersive 3D environments for arbitrary driving simulation, remains an emerging yet underexplored direction. A handful of pioneering works have explored 3D driving scene generation at scale. For example, SemCity [ 28 ] generates city-scale 3D occupancy grids using DMs, but the lack of appearance details limits its practicality for realistic simulation. UniScene [ 18 ] and InfiniCube [ 29 ] extend this by generating both 3D occupancy and images, but require a manually defined large-scale layout as a conditioning input, complicating the generation process and hindering flexibility. In this work, we explore the potential solution to large-scale scene generation with spatial expansion, which faces the following three main challenges: 1) Flexible Controllability : Enabling versatile control through both low-level conditions (e.g., layouts) for precise scene composition and high-level prompts (e.g., user-intent text descriptions) for efficient, intuitive customization. For instance, as shown in Fig. 1 , users can provide a brief scene description, which the system elaborates into a plausible scene by fully harnessing the generative modelβs creative capacity; 2) High-Fidelity Geometry and Appearance : generating intricate geometry alongside photorealistic appearance, which is essential to ensure both the structural integrity and visual realism of the 3D scene; 3) Large-Scale Consistency : maintaining spatial coherence across interconnected regions to ensure global consistency throughout the extended scene. To address these challenges, we propose \mathcal{X} -Scene , a novel framework for large-scale driving scene generation. \mathcal{X} -Scene offers: 1) Multi-Granular Controllability : \mathcal{X} -Scene empowers users to guide generation with varying levels of detail, accommodating both fine-grained layouts for precise control and high-level text prompts for efficient scene customization. To enhance the expressiveness of text-based control, textual prompts are initially enriched by LLMs to form detailed scene narratives. These narratives then inform a text-driven layout generation module that automatically establishes spatial arrangements, guiding subsequent scene synthesis. This dual-control paradigm effectively supports users requiring meticulous, layout-based precision alongside those preferring rapid, prompt-driven customization, thereby broadening accessibility. 2) Geometrical and Visual Fidelity : \mathcal{X} -Scene achieves high fidelity by employing a unified pipeline that sequentially generates 3D semantic occupancy and the corresponding multiview images. This process ensures both structural accuracy in the 3D geometry and photorealistic visual appearance, promoting inherent consistency and robust alignment between the geometric (occupancy) and visual (image) modalities. 3) Consistent Large-Scale Extrapolation : To enable the creation of expansive environments, \mathcal{X} -Scene progressively extrapolates new scene content conditioned on adjacent, previously synthesized regions. This consistency-aware outpainting mechanism meticulously preserves spatial continuity, facilitating the seamless and coherent extension of the 3D driving scene well beyond a single local area. Moreover, to support a diverse array of downstream applications, including realistic driving simulations and immersive free-roam exploration within the generated environments, we further process the synthesized semantic occupancy and multi-view images. Specifically, we reconstruct them into 3D Gaussian (3DGS) [ 30 ] representations, a technique adept at faithfully preserving both intricate geometric structures and high-fidelity visual appearance. By unifying these capabilities, \mathcal{X} -Scene advances the state of the art in large-scale, high-fidelity, and controllable driving scene synthesis, empowering data generation and simulation for autonomous driving. The main contributions of our work are summarized as follows: β’ We propose \mathcal{X} -Scene , a novel framework for large-scale 3D driving scene generation with multi-granular controllability, geometrical and visual fidelity, and consistent large-scale extrapolation, supporting a wide range of downstream applications. β’ We design a flexible multi-granular control mechanism that synergistically combines high-level semantic guidance (LLM-enriched text prompts) with low-level geometric specifications (user-provided or text-driven layout), enabling scene creation tailored to diverse user needs. β’ We present a unified generation and extrapolation pipeline that ensures robust geometric fidelity and photorealistic visual appearance, while also achieving seamless large-scale scene expansion by maintaining spatial and semantic coherence across extrapolated regions. β’ Extensive experiments show \mathcal{X} -Scene achieves superior performance in generation quality and controllability, enabling diverse applications from data augmentation to driving simulation. 2 Related Works 2.1 Driving Image and Video Generation Diffusion models [ 31 , 32 , 33 , 34 ] have revolutionized image generation by iteratively refining Gaussian noise into high-quality images. Building on this technique, they have significantly advanced autonomous driving by enabling image and video generation for a wide range of downstream applications. For example, several methods synthesize realistic driving images [ 35 , 36 , 1 ] or videos [ 2 , 3 , 4 , 5 , 6 , 7 , 8 , 9 , 10 , 11 , 12 , 13 , 14 ] from 3D box or layout conditions to support perception tasks through data augmentation. Other approaches [ 37 , 38 ] focus on generating rare yet critical driving events, such as lane changes or vehicle cut-ins, to enhance planning tasks