ScenDi: 3D-to-2D Scene Diffusion Cascades for Urban Generation

Hanlei Guo, Jiahao Shao, Xinya Chen, Xiyang Tan, Sheng Miao, Yujun Shen, Yiyi Liao, Zhejiang University · 2026

Read ScenDi: 3D-to-2D Scene Diffusion Cascades for Urban Generation by Hanlei Guo, Jiahao Shao, Xinya Chen, Xiyang Tan, Sheng Miao, Yujun Shen, Yiyi Liao, Zhejiang University (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2601-15221.txt.

Introduction

Hanlei Guo, Jiahao Shao, and co-authors generate city scenes by way of flat images. Their 2026 arXiv paper deals with 3D urban scene generation with diffusion models. Making a whole city block that looks real is much harder than making one object.

Pure 3D diffusion tends to lose fine surface detail. Pure 2D diffusion keeps the detail but gives up camera control. The authors present ScenDi, a cascade that goes from 3D to 2D in stages. The design aims to keep both the fine detail and the control over camera viewpoints.

Catalog record

CREATORHanlei Guo, Jiahao Shao, Xinya Chen, Xiyang Tan, Sheng Miao, Yujun Shen, Yiyi Liao, Zhejiang University
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE36 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2601-15221

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

ScenDi: 3D-to-2D Scene Diffusion Cascades for Urban Generation Hanlei Guo, Jiahao Shao, Xinya Chen, Xiyang Tan, Sheng Miao, Yujun Shen, Yiyi Liao, Zhejiang University Abstract Recent advancements in 3D object generation using diffusion models have achieved remarkable success, but generating realistic 3D urban scenes remains challenging. Existing methods relying solely on 3D diffusion models tend to suffer a degradation in appearance details, while those utilizing only 2D diffusion models typically compromise camera controllability. To overcome this limitation, we propose ScenDi, a method for urban scene generation that integrates both 3D and 2D diffusion models. We first train a 3D latent diffusion model to generate 3D Gaussians, enabling the rendering of images at a relatively low resolution. To enable controllable synthesis, this 3DGS generation process can be optionally conditioned by specifying inputs such as 3d bounding boxes, road maps, or text prompts. Then, we train a 2D video diffusion model to enhance appearance details conditioned on rendered images from the 3D Gaussians. By leveraging the coarse 3D scene as guidance for 2D video diffusion, ScenDi generates desired scenes based on input conditions and successfully adheres to accurate camera trajectories. Experiments on two challenging real-world datasets, Waymo and KITTI-360, demonstrate the effectiveness of our approach. Figure 1 : ScenDi generates high-quality urban scenes using a 3D-to-2D Scene Diffusion cascade, with optional condition signals like text and layout for controllable 3D space generation. Our method provides flexible camera control, even though our training data primarily consists of forward-moving trajectories. 1 Introduction Generating 3D urban scenes from scratch, either unconditionally or with coarse guidance such as text prompts or layout maps, is an important step toward building open-world virtual environments for gaming and driving simulation. Unlike image-to-video synthesis 17 ; 52 , which extends an input view given viewpoints, 3D scene generation aims to create entire environments with consistent geometry and realistic appearance that can be rendered from free viewpoints. Such capability is crucial for enhancing data diversity, supporting scene composition, and enabling scalable scene generation. A majority line of work addresses this task by directly performing generation in the 3D space 78 ; 25 ; 2 ; 32 . While such approaches offer explicit spatial modeling, their performance is constrained by the low resolution of 3D representations. Besides, generating 3D urban scenes that directly render to high-fidelity images is challenging due to the scarcity of such 3D GT data. More recently, another line of methods 13 ; 40 ; 27 ; 38 ; 85 attempt to enhance the performance by first generating 3D semantic voxels, and then render 2D conditions (e.g., depth or semantic maps) for video generation, allowing for producing high-fidelity details thanks to the powerful pre-trained 2D models. While this approach benefits from 3D structure, the final appearance is still entirely generated in 2D, requiring the model to learn a complex mapping from depth or semantics to RGB images, which can make training less efficient and lack consistency when the same place is revisited. This raises a fundamental question: How much of the generation process should occur in 3D space, and how much should be delegated to 2D? In this work, we propose ScenDi 1 1 1 ScenDi means “descend” in Italian, reflecting our cascaded generation process moving from 3D to 2D. , a cascaded 3D-to-2D Scen e Di ffusion framework that leverages the complementary strengths of both modalities. Our key insight is that 3D generation should take the lead in establishing geometric and coarse appearance priors, while 2D diffusion focuses on refining details and enhancing distant regions. With this cascaded design, ScenDi achieves high-fidelity urban scene generation without sacrificing camera controllability, outperforming the counterpart that generates appearance purely in 2D space conditioned on geometry in training efficiency and loop consistency. Our method begins by training a 3D latent diffusion model to generate coarse 3D scenes, which comprises a novel Voxel-to-3DGS VQ-VAE learned from 2D supervision and a 3D diffusion model, leveraging an off-the-shelf depth estimator to form the input voxel grids. This enables sampling 3D Gaussian Splatting (3DGS) scenes that renders to multi-view consistent images. However, rendered images from the 3D LDM often lack high-frequency details due to limited 3D resolution and fail to capture distant regions. To address this, we condition a 2D video diffusion model on the rendered RGB images, leveraging its ability to refine details and synthesize regions beyond predefined ranges. Experimental results on real-world autonomous driving datasets KITTI-360 and Waymo demonstrate that our 3D-to-2D diffusion cascades allow for generating high-fidelity urban scenes while preserving accurate camera control. 2 Related Work Direct 3D Content Generaion: One line of research focus on generating 3D scenes relying solely on 3D backbones. Early works 6 ; 2 ; 71 ; 78 ; 73 that adopt GANs 18 suffer from inherent issues such as training instability and model corruption. A growing trend to leverage 3D diffusion models in image on tasks emerged due to their superb training stability. Methods lying on this line enables the availability of explicit 3D representation, thus leads to the camera controllable viewpoints over the generated scenes. Some works 53 ; 29 ; 70 ; 84 ; 82 construct 3D ground-truth datasets and perform training directly in 3D space. However, the natural scarcity of high-quality 3D scene data, such as 3D Gaussians that require per-scene optimization, undermines these works’ scalability and limits their ability to generalize to diverse real-world scenarios. In contrast, we propose to use a VQ-VAE that directly maps voxel grids obtained from off-the-shelf depth priors to 3D Gaussians. Similar to us, several methods 1 ; 74 ; 8 ; 25 ; 3 also train 3D diffusion models that predict 3D representations with 2D multi-view image supervisions. Although they can avoid heavy dataset pre-processing, their results often lose fine appearance details. Our method addresses this challenge by using 3D-to-2D diffusion cascades. The last line of methods 31 ; 10 ; 60 ; 68 ; 75 distills information from pretrained 2D diffusion models to optimize 3D representation based on Score Distillation Sampling (SDS) 48 . These methods require a long training time to generate a single scene and usually suffer from over-saturation. Multi-view Image Generation: In contrast to direct 3D generation, some works leverage 2D diffusion models for multi-view generation, aiming to exploit the powerful generative prior of 2D diffusion models trained on internet-scale data. Despite the impressive progress in this direction, many methods target only on object-level generation 57 ; 33 ; 28 ; 34 ; 36 ; 62 . There are also attempts for street scene image/video generation 16 ; 66 ; 69 ; 17 . While most methods show superior visual fidelity compared with 3D generative methods, they struggle to provide flexible camera control due to the lack of explicit 3D information. Recently, there has been a growing trend towards gradually reconstructing 3D scenes while generating novel views 12 ; 81 ; 80 . However, these methods have not yet demonstrated the ability to handle large forward camera motions typical in driving scenarios. Urban Scene Generation: A line of methods 17 ; 69 ; 67 ; 52 ; 9 ; 19 ; 76 tackles image-to-video generation for urban scenes. Despite promising results, noticeable visual degradation often occurs as the methods extend far from

Read the complete text