ArbiViewGen: Controllable Arbitrary Viewpoint Camera Data Generation for Autonomous Driving via Stable Diffusion Models

Yatong Lan, Jingfeng Chen, Yiru Wang, Lei He · 2025

Read ArbiViewGen: Controllable Arbitrary Viewpoint Camera Data Generation for Autonomous Driving via Stable Diffusion Models by Yatong Lan, Jingfeng Chen, Yiru Wang, Lei He (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2508-05236.txt.

Introduction

Yatong Lan and co-authors teach a diffusion model to imagine what a car's cameras would see from viewpoints no camera occupied. Self-driving stacks could train on such views, but no ground truth exists for them, which starves the models. Their system, ArbiViewGen, stitches the real camera views together with a two-step matching method, keeping only the patches where the stitching is confident.

Training then runs in a loop: the model must rebuild the original views from its own stitched images, which enforces consistency between viewpoints without any outside labels. Only ordinary multi-camera images and their positions are needed. The authors claim the first controllable system of its kind across vehicle setups.

Catalog record

CREATORYatong Lan, Jingfeng Chen, Yiru Wang, Lei He
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE39 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2508-05236

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

ArbiViewGen: Controllable Arbitrary Viewpoint Camera Data Generation for Autonomous Driving via Stable Diffusion Models Yatong Lan, Jingfeng Chen, Yiru Wang, Lei He Abstract Arbitrary viewpoint image generation holds significant potential for autonomous driving, yet remains a challenging task due to the lack of ground-truth data for extrapolated views, which hampers the training of high-fidelity generative models. In this work, we propose ArbiViewGen a novel diffusion-based framework for the generation of controllable camera images from arbitrary points of view. To address the absence of ground-truth data in unseen views, we introduce two key components: Feature-Aware Adaptive View Stitching (FAVS) and Cross-View Consistency Self-Supervised Learning (CVC-SSL). FAVS employs a hierarchical matching strategy that first establishes coarse geometric correspondences using camera poses, then performs fine-grained alignment through improved feature matching algorithms, and identifies high-confidence matching regions via clustering analysis. Building upon this, CVC-SSL adopts a self-supervised training paradigm where the model reconstructs the original camera views from the synthesized stitched images using a diffusion model, enforcing cross-view consistency without requiring supervision from extrapolated data. Our framework requires only multi-camera images and their associated poses for training, eliminating the need for additional sensors or depth maps. To our knowledge, ArbiViewGen is the first method capable of controllable arbitrary view camera image generation in multiple vehicle configurations. Introduction The automotive industry has witnessed the emergence of end-to-end autonomous driving technology as a predominant development direction. However, the heterogeneous configurations of multi-source sensor systems have introduced coupling challenges. Models trained with different sensor combinations are difficult to transfer and reuse across platforms. Current autonomous driving systems typically employ multi-camera surround-view configurations as the core perception module, but there are significant differences among vehicle types in terms of the number of cameras, installation positions, and fields of view. These configuration discrepancies result in severely compromised cross-platform data reusability, necessitating extensive data collection and annotation efforts for each new vehicle model, which leads to high development costs and long cycles. To address this issue, arbitrary view camera image generation technology has emerged. By generating high-quality images from arbitrary poses using a limited set of existing camera views, it is possible to achieve data reuse across different vehicle types and reduce the development cost for new models. However, unlike general scene reconstruction, data collection in autonomous driving scenarios is usually limited to a single driving trajectory, resulting in sparsity and homogeneity of observed data in 3D space. This is particularly problematic for novel view synthesis, where there is a severe lack of ground truth in extrapolated views: when the rendered viewpoint deviates from the recorded trajectory, it is impossible to obtain ground truth images for direct supervised training, which has become a core bottleneck restricting the development of this technology. Despite rapid progress, existing multiview image generation methods remain fundamentally limited by their reliance on ground-truth supervision at target viewpoints, a resource that is inherently scarce in autonomous driving scenarios. Current approaches can be roughly grouped into two categories: diffusion-based generation methods and 3D reconstruction-based synthesis methods. Diffusion-based methods (e.g., MVDiffusion ( Tang et al. 2023 ) , SyncDreamer ( Liu et al. 2023b ) , FreeVS ( Wang et al. 2024 ) , DiST-4D ( Guo et al. 2025 ) ) typically adopt an end-to-end paradigm that learns mappings between input-output view pairs, but their performance often suffers in scenarios with sparse or incomplete viewpoint coverage, which is common in driving datasets. 3D reconstruction-based methods (e.g., 3D Gaussian Splatting ( Kerbl et al. 2023 ) , NeRF ( Mildenhall et al. 2020 ) ) utilize explicit or implicit scene geometry to enable novel view synthesis, yet their two-stage reconstruction-rendering pipeline is highly sensitive to the spatial sparsity of input views, leading to artifacts and degraded quality in extrapolated regions. These limitations underscore the core challenge: the absence of ground-truth data from novel viewpoints prevents most existing methods from being trained effectively in such scenarios, especially in real-world autonomous driving settings. We break the dependency on ground-truth supervision at novel viewpoints by introducing CVC-SSL, a self-supervised framework that allows closed-loop training for arbitrary-view generation. Our approach constructs pseudo-novel views via geometric image stitching and employs a diffusion model to reconstruct original camera images from these synthetic views. The reconstruction errors serve as self-supervised signals, allowing the model to learn cross-view geometric relationships and visual consistency without requiring ground-truth data at extrapolated poses. Notably, our method requires only six camera images and their corresponding pose information to achieve end-to-end model training, establishing for the first time a controllable arbitrary viewpoint generation system for multi-vehicle architectures. For extrapolated viewpoints, we propose a quantitative evaluation strategy based on projecting colored LiDAR point clouds to novel views to obtain sparse ground-truth pixels. The main contributions of this study are as follows. • A pure visual image stitching algorithm is developed by combining geometric transformation with hierarchical feature matching. It enables the automatic construction of high-quality pseudo-ground truth data for extrapolated views via precise alignment and texture fusion, offering reliable supervision for training. • A self-supervised learning paradigm based on cyclic reconstruction is introduced which establishes bidirectional mappings across views. This design effectively overcomes the lack of ground truth supervision in novel viewpoints and substantially enhances generation quality. • To enable quantitative evaluation, a novel image quality assessment strategy is proposed, which projects colored point clouds—sampled from real images—into target views. This establishes the first end-to-end evaluation framework for controllable arbitrary-view generation across diverse vehicle architectures. Figure 1: Pipeline of ArbiViewGen for controllable arbitrary-view image generation. FAVS generates pseudo ground-truth views via geometry-guided feature stitching. CVC-SSL trains a latent diffusion model with cross-view consistency to generate multi-view images from arbitrary poses using only 6-camera inputs and pose information—without requiring ground-truth extrapolated views. Related Work Diffusion-based Generation Methods Diffusion-based novel view synthesis methods have gained significant attention due to their ability to model complex scene distributions. However, most of these techniques rely heavily on ground-truth supervision at specific training viewpoints, which severely restricts their ability to generalize to unseen or extrapolated views. This limitation becomes particularly evident in autonomous driving scenarios, where such detailed supervision is often unavailable for novel viewpoints. Notable methods include Zero-1-to-3 ( Liu et al. 2023a ) , which integrates camera pose embeddings to enhance the synthesis of new perspectives, though it is primarily designed for object-centric scenes; StreetCrafter ( Yan et al. 2025 ) , which leverages a LiDAR-conditioned video diffusion approach to generate novel views, yet depends heavily on the availability of additional sensors and is less robust when tasked with synthesizing views at large angular extrapolations; DiST-4D ( Guo et al. 2025 ) , which incorporates metric depth information to facilitate 4D scene synthesis, but its reliance on precise depth data limits its applicability in the absence of such data; and DriveX ( Yang et al. 2024 ) and Drive-1-to-3 ( Lin et al. 2024 ) , which enhance the synthesis quality within constrained camera setups, yet still fail to guarantee geometric consistency when applied to novel viewpoints lacking supervision. The underlying challenge these methods face is their dependence on explicit ground-truth supervision, which fundamentally constrains their ability to perform well in situations where data is sparse or entirely missing from new viewpoints. In contrast, our framework addresses

Read the complete text