BEV-VAE: Multi-view Image Generation with Spatial Consistency for Autonomous Driving

Zeming Chen Hang Zhao · 2025

Read BEV-VAE: Multi-view Image Generation with Spatial Consistency for Autonomous Driving by Zeming Chen Hang Zhao (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2507-00707.txt.

Introduction

Zeming Chen and Hang Zhao present a 2025 model that dreams up matched camera views for self-driving cars. A car's cameras must agree on one 3D scene, yet most generators treat the views as a loose photo set. Their model builds the scene in a bird's-eye map first, then renders each view from it.

A compact autoencoder learns that shared map space across views. A diffusion transformer then grows new scenes in the map, which can be read out from any camera setup and steered by a 3D layout if one is given. On two big driving data sets, the views held together in 3D better than past methods.

Catalog record

CREATORZeming Chen Hang Zhao
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE46 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2507-00707

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

BEV-VAE: Multi-view Image Generation with Spatial Consistency for Autonomous Driving Zeming Chen Hang Zhao Abstract Multi-view image generation in autonomous driving demands consistent 3D scene understanding across camera views. Most existing methods treat this problem as a 2D image set generation task, lacking explicit 3D modeling. However, we argue that a structured representation is crucial for scene generation, especially for autonomous driving applications. This paper proposes BEV-VAE for consistent and controllable view synthesis. BEV-VAE first trains a multi-view image variational autoencoder for a compact and unified BEV latent space and then generates the scene with a latent diffusion transformer. BEV-VAE supports arbitrary view generation given camera configurations, and optionally 3D layouts. Experiments on nuScenes and Argoverse 2 (AV2) show strong performance in both 3D consistent reconstruction and generation. The code is available at https://github.com/Czm369/bev-vae . † † footnotetext: {}^{\text{\textdagger}} Corresponding to: [email protected] 1 Introduction Multi-view image generation is becoming increasingly important in autonomous driving, as it enables controllable synthesis of diverse scenes such as the addition or removal of vehicles based on 3D layouts. This capability facilitates the creation of rare or hard-to-collect scenarios and provides a scalable, flexible means of augmenting data for training end-to-end driving models. Recent methods Magicdrive ; DrivingDiffusion ; DriveWM ; Panacea based on fine-tuned Stable Diffusion model multi-view image generation as a set of 2D synthesis tasks with adjacent-view consistency constraints. While these approaches can achieve a certain degree of spatial coherence, they rely on view-dependent cross-attention in image space to implicitly model 3D structure, lacking a unified and structured scene representation. Consequently, they struggle to support novel view synthesis from arbitrary camera poses and cannot perform controllable generation directly conditioned on 3D layouts. Moreover, using 2D projections of 3D bounding boxes as conditions inevitably leads to the loss of depth information. Projections of different objects may overlap in image space, especially in crowded scenes, introducing occlusion ambiguity. As a result, the generative model must simultaneously learn to produce spatially consistent images across views and align them with these ambiguous 2D conditions, making the training process more complex and less geometrically grounded. In contrast, our approach adopts a fundamentally different paradigm by performing generation in a Bird’s-Eye-View (BEV) latent space, as shown in Fig. 1 . Instead of modeling each view separately, BEV-VAE encodes a unified latent representation that captures both semantic content and structured 3D spatial geometry. This shared BEV representation ensures spatial consistency across all views, as the same spatial location corresponds to consistent content regardless of camera perspective. Novel views can be synthesized simply by modifying camera poses at decoding time, without the need for retraining. Furthermore, object layouts can be explicitly edited using 3D binary occupancy maps, which are spatially aligned with the BEV latent space. This alignment enables precise and interpretable control over object quantity, position, and category, and avoids the ambiguity and lack of depth information introduced by 2D projections of 3D bounding boxes. In this paper, we propose BEV-VAE, a multi-view image generation method with a unified representation of the 3D scene. BEV-VAE explicitly constructs a spatially aligned latent space in bird’s-eye view (BEV) during the encoding stage, This structured BEV space enables high-fidelity reconstruction with strong cross-view alignment, supports novel view synthesis by manipulating camera poses without retraining, and allows controllable generation conditioned on 3D object layouts, such as varying object quantity, position, or category—offering a more scalable and interpretable solution for autonomous driving applications. Experiments on nuScenes and Argoverse 2 (AV2) show strong reconstruction and generation performance. BEV-VAE is the first to generate all 7 surround-view images on AV2, demonstrating its robustness and practicality. Our contributions are as follows. \bullet We propose a framework that constructs spatially aligned BEV representations from multi-view images, enabling high-fidelity reconstruction with strong cross-view consistency. \bullet We demonstrate that the learned BEV latent space supports novel view synthesis by manipulating camera poses, validating its structured 3D nature and spatial coherence. \bullet We instantiate diffusion-based generation directly in the BEV space, allowing controllable synthesis conditioned on 3D object layouts, such as quantity, location and category. Figure 1 : Comparison of two paradigms for multi-view image generation. (a) Image latent space generation relies on 2D projections of 3D objects to guide image synthesis and cross-view attention to enforce spatial consistency. (b) BEV latent space generation is conditioned on 3D occupancy to produce a unified representation, from which all views are decoded, naturally preserving spatial consistency and enabling novel view synthesis by adjusting camera poses. 2 Related Work 2.1 Bird’s-Eye-View Perception Autonomous driving relies on Bird’s Eye View (BEV) to unify multi-view image information. The construction of the BEV feature follows two approaches: bottom-up and top-down. Bottom-up methods LSS ; BEVDet ; BEVFusion estimate the depth required to lift 2D features into 3D space before fusing them into BEV. In contrast, top-down methods BEVFormer ; UniAD use deformable attention and query mechanisms to efficiently aggregate features by dynamically sampling key regions. In top-down methods, deformable attention (DA) plays a pivotal role in enhancing computational efficiency and focusing on relevant areas. Let q,p , and v represent the query, reference points, and value features, respectively. M denotes the number of attention heads and K is the total number of sampled keys. The mechanism is calculated by: \operatorname{DA}(q,p,v)=\sum_{m=1}^{M}\mathcal{W}_{m}\sum_{k=1}^{K}\mathcal{A}_{mk}\cdot\mathcal{V}_{mk}, where m indexes the attention head, and k indexes the sampled keys. The \mathcal{W}_{m}\in\mathbb{R}^{C\times C/M} are learnable weights with dimension C , and \mathcal{V}_{mk} are the features at location p+\Delta p_{mk} , which are extracted by bilinear interpolation. \Delta p_{mk} and \mathcal{A}_{mk} denote the sampling offset and attention weight of the k^{\text{th}} sampling point in the m^{\text{th}} attention head, respectively. Both \Delta p_{mk} and \mathcal{A}_{mk} are obtained via linear projection over the query q , and A_{mk} is normalized by softmax to ensure \sum_{k=1}^{K}\mathcal{A}_{mk}\!=\!1 . 2.2 Variational Autoencoder Variational AutoEncoder (VAE) formulates image generation as probabilistic inference by introducing a latent variable z and optimizing the Evidence Lower Bound (ELBO) to jointly learn a Gaussian posterior and a reconstruction decoder. However, modeling z as a Gaussian limits the sharpness and fidelity of generated images. To address this, VQVAE VQVAE introduces a discrete codebook to enhance diversity, while VQGAN VQGAN integrates GAN GAN training with VQVAE, leveraging adversarial, perceptual PerceptualLoss , and reconstruction losses for realistic image generation. ViT-VQGAN ViT-VQGAN further improves global context modeling and codebook efficiency by employing ViT ViT as both encoder and decoder, enhancing generative performance. In addition to modeling the latent variable z as a discrete distribution, diffusion models transform data into a standard Gaussian distribution by progressively adding noise and then reverse this process via denoising. DDPM DDPM employs a Markov chain for iterative noise addition and recovery, while DDIM DDIM accelerates sampling with deterministic inference. LDM LDM performs diffusion in a compressed latent space to enhance efficiency and quality. DiT DiT further integrates diffusion with Transformer architectures, improving high-resolution generation and expanding applicability. 2.3 Autonomous Driving Multi-view Generation Multi-view generation inherently represents a 3D scene through 2D images. BEVGen BEVGen uses autoregressive generation to produce multi-view images based on BEV layouts. It constructs direction vectors for cameras and BEV layouts, maps them to the BEV ego-vehicle coordinate system via camera parameters, and integrates their inner product as an attention bias to

Read the complete text