AnyScene: Towards Highly Controllable Driving Scene Generation at Anywhere and Beyond
Haiming Zhang, Junfei Zhou, Feng Jiang, Jingzhong Li, Zhenglong Guo, Penglin Dai, Jifeng Dai, Yan Xie, Benjin Zhu · 2026
Read AnyScene: Towards Highly Controllable Driving Scene Generation at Anywhere and Beyond by Haiming Zhang, Junfei Zhou, Feng Jiang, Jingzhong Li, Zhenglong Guo, Penglin Dai, Jifeng Dai, Yan Xie, Benjin Zhu (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2605-26113.txt.
Introduction
Zhang and eight colleagues generate steerable driving scenes from any top-down layout. Fake data matter for self-driving cars, above all for rare safety-critical cases, but current occupancy-guided methods steer weakly and lean on a fixed frame. AnyScene is a single occupancy-first system: a diffusion transformer reads top-down and occupancy traits jointly, step by step. That gives tight control from varied layouts and supports long runs. Later uses include sparse-view three-dimensional rebuilding.
Catalog record
| CREATOR | Haiming Zhang, Junfei Zhou, Feng Jiang, Jingzhong Li, Zhenglong Guo, Penglin Dai, Jifeng Dai, Yan Xie, Benjin Zhu |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 66609 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2605.26113
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
AnyScene: Towards Highly Controllable Driving Scene Generation at Anywhere and Beyond Haiming Zhang, Junfei Zhou, Feng Jiang, Jingzhong Li, Zhenglong Guo, Penglin Dai, Jifeng Dai, Yan Xie, Benjin Zhu Abstract Generating high-fidelity and controllable synthetic data is critical for advancing end-to-end autonomous driving, particularly for addressing the long tail of rare safety-critical scenarios. Existing occupancy-guided methods typically rely on shallow conditioning mechanisms and reference-frame-dependent video synthesis, which limits fine-grained controllability from arbitrary BEV layouts and restricts their applicability for scalable simulation. In this paper, we propose AnyScene , a unified occupancy-centric framework for driving scene generation. AnyScene generates semantic occupancy sequences from BEV layouts through a Spatial-Temporal Occupancy Diffusion Transformer that jointly tokenizes BEV and occupancy features in an autoregressive manner. This design enables precise controllability from cross-dataset and user-defined BEV inputs while naturally supporting long-horizon generation. Building upon the generated occupancy, a Geometry-Grounded View Expansion module treats occupancy as the canonical spatial representation and synthesizes temporally consistent multi-view driving videos in a reference-free and autoregressive fashion, supporting flexible camera configurations at inference time. Extensive experiments demonstrate that AnyScene achieves state-of-the-art performance in both occupancy and video generation. It exhibits strong generalization to unseen and customized layouts, and provides measurable benefits for downstream tasks such as sparse-view 3D reconstruction. 1 Introduction Despite substantial progress in L2+ assisted driving 19 ; 68 ; 28 ; 34 , achieving reliable Level-4 autonomy remains hindered by the long-tail distribution of rare yet safety-critical scenarios. These scenarios occur infrequently in real-world operations and are severely underrepresented in existing public datasets 4 ; 37 ; 49 . As a result, scalable synthetic data generation has become essential for systematically addressing this long-tail challenge. To support effective closed-loop training and evaluation of end-to-end autonomous driving systems, a next-generation driving simulator must simultaneously satisfy two fundamental requirements: precise controllability , which enables faithful instantiation of user-specified scene configurations, and anywhere-deployability , which requires robust generalization to unseen geographies, traffic patterns, and cross-dataset or user-defined layouts. Satisfying both requirements concurrently remains a central and unresolved challenge. Existing approaches to driving scene synthesis each exhibit structural limitations that prevent them from meeting these dual requirements. Direct end-to-end video generation models 17 ; 36 ; 15 ; 13 ; 63 ; 47 can produce visually compelling sequences from coarse prompts, yet their implicit mappings often result in inconsistent geometry across views and time, making fine-grained control difficult. Reconstruction-based simulators 50 ; 54 ; 40 ; 53 ; 67 ; 10 achieve high visual fidelity but are inherently limited to replaying logged trajectories due to per-scene optimization. These shortcomings motivate the use of intermediate representations that decouple geometry from appearance. Among such representations, 3D semantic occupancy has emerged as a promising choice because it encodes scene geometry and semantics in a structured, view-independent manner, allowing direct and editable control over scene elements 5 ; 22 ; 64 ; 44 . In autonomous driving, BEV layouts provide a particularly suitable conditioning signal, as they naturally align with perception and planning outputs 19 ; 14 ; 27 . Building on these advantages, recent occupancy-guided methods such as UniScene 27 and InfiniCube 33 have explored this direction. However, their practical capabilities remain limited in three key aspects. First, the mapping from BEV layouts to occupancy is fragile. Existing pipelines typically rely on shallow cross-attention adapters and over-engineered triplane factorizations 60 ; 26 , which weaken fine-grained control and often lead to missing agents, blurred layout edits, or mode-collapsed attributes. Second, conditioning signals are typically provided at low frequency ( 2 Hz) 4 ; 60 ; 39 or at coarse quality 27 , both of which are incompatible with the higher frame rates required by practical video generation pipelines. Third, multi-view video synthesis in current methods 27 ; 33 ; 60 structurally depends on reference frames and a fixed camera rig, preventing the generation of entirely new scenes from arbitrary BEV inputs and limiting flexibility in camera configuration. To address these limitations, we propose AnyScene , a unified occupancy-centric framework for driving scene generation. Given an arbitrary BEV layout, AnyScene first produces a high-fidelity semantic occupancy sequence through a BEV-conditioned generative model. This model employs a compact 2D BEV-based Occupancy VAE to encode occupancy into latent representations, and a Spatio-Temporal Occupancy Diffusion Transformer (STOccDiT) that jointly tokenizes BEV layout features and occupancy latents. By adopting an autoregressive generation process with a causal temporal attention mask, STOccDiT enables precise and high-frequency control from arbitrary, cross-dataset, and user-defined BEV inputs while naturally supporting long-horizon occupancy sequences. Building upon the generated occupancy, AnyScene further synthesizes temporally and cross-view consistent multi-view driving videos through Geometry-Grounded View Expansion (GGVE). This module treats the occupancy sequence as the canonical spatial representation and renders explicit geometric conditioning signals to guide video generation in a reference-free and autoregressive manner, allowing flexible control over camera counts and poses at inference time. To support high-frequency evaluation aligned with practical video pipelines, we additionally introduce nuCraftv2 , a 12 Hz resampling of nuScenes with synchronized BEV layouts and high quality dense semantic occupancy. We validate AnyScene through comprehensive experiments, demonstrating state-of-the-art performance across occupancy generation quality, multi-view video fidelity, and geometry-grounded controllability. Additional qualitative results further showcase the flexibility and versatility of AnyScene in video synthesis across arbitrary camera rigs, novel trajectories, scene editing, and scene reconstruction. In summary, the main contributions of this work are as follows: • We propose BEV-Conditioned Controllable Occupancy Generation, which consists of a BEV-based Occupancy VAE and a Spatio-Temporal Occupancy Diffusion Transformer (STOccDiT). The VAE adopts a compact 2D BEV representation for high-fidelity latent encoding, while STOccDiT jointly tokenizes BEV layouts and occupancy latents to generate occupancy in an autoregressive manner. This module enables precise, high-frequency control from arbitrary, cross-dataset, and user-defined BEV inputs. • We design Geometry-Grounded View Expansion (GGVE), a novel paradigm that treats the generated occupancy sequence as the canonical spatial anchor. By rendering explicit geometric conditioning signals from occupancy, GGVE enables autoregressive, reference-free multi-view video synthesis with arbitrary camera counts and poses, overcoming the reference-frame dependency that limits existing occupancy-guided methods. • We curate nuCraftv2, a 12 Hz high-frequency benchmark with synchronized BEV layouts and high-quality dense semantic occupancy. Built upon this benchmark, AnyScene achieves state-of-the-art performance in both occupancy and video generation, and demonstrates tangible benefits for applications. 2 Related Works 2.1 Controllable Occupancy Generation Semantic occupancy has emerged as a structured volumetric representation that encodes both geometry and semantics, making it suitable for controllable scene generation. Early efforts on layout-conditioned occupancy synthesis include UrbanDiffusion 61 , which generates semantic occupancy from BEV maps using a 3D diffusion model. More recent works have further explored controllable generation from BEV or layout conditions, such as UniScene 27 , which proposes an occupancy-centric pipeline for generating semantic occupancy from BEV layouts; InfiniCube 33 , which enables unbounded and controllable dynamic 3D scene generation; WoVoGen 32 , which develops a world volume-aware diffusion model for controllable multi-camera driving scene generation; X-Scene 57 , which advances large-scale generation with flexible multi-granular control; and ConsistentCity 69 , which improves temporal consistency under BEV conditioning via a semantic flow-guided Occupancy DiT. However, most existing methods still struggle to achieve strong controllability from arbitrary or user-defined BEV inputs. Our work addresses this