Driving Scene Synthesis on Free-form Trajectories with Generative Prior

Zeyu Yang, Zijie Pan, Yuankun Yang, Xiatian Zhu, Li Zhang · 2024

Read Driving Scene Synthesis on Free-form Trajectories with Generative Prior by Zeyu Yang, Zijie Pan, Yuankun Yang, Xiatian Zhu, Li Zhang (2024) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2412-01717.txt.

Introduction

Zeyu Yang and four co-authors present DriveX in 2024, a system that synthesizes driving views along paths no camera ever drove. Simulators need such free-form views to test self-driving policies in a closed loop. Older methods interpolate along recorded routes but crumble on novel ones, starved of viewpoints.

DriveX distills a video diffusion model's know-how into a 3D Gaussian scene while it trains. The diffusion model cleans up the rough novel-path renders; the cleaned videos then supervise the scene model, and the pseudo-truth improves in rounds as renders sharpen. The loop yields faithful, real-time views beyond the recorded roads.

Catalog record

CREATORZeyu Yang, Zijie Pan, Yuankun Yang, Xiatian Zhu, Li Zhang
YEAR2024
COLLECTIONPAPERS
FORMATTXT
FILE SIZE43 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2412-01717

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Driving Scene Synthesis on Free-form Trajectories with Generative Prior Zeyu Yang, Zijie Pan, Yuankun Yang, Xiatian Zhu, Li Zhang Abstract Driving scene synthesis along free-form trajectories is essential for driving simulations to enable closed-loop evaluation of end-to-end driving policies. While existing methods excel at novel view synthesis on recorded trajectories, they face challenges with novel trajectories due to limited views of driving videos and the vastness of driving environments. To tackle this challenge, we propose a novel free-form driving view synthesis approach, dubbed DriveX , by leveraging video generative prior to optimize a 3D model across a variety of trajectories. Concretely, we crafted an inverse problem that enables a video diffusion model to be utilized as a prior for many-trajectory optimization of a parametric 3D model ( e.g . , Gaussian splatting). To seamlessly use the generative prior, we iteratively conduct this process during optimization. Our resulting model can produce high-fidelity virtual driving environments outside the recorded trajectory, enabling free-form trajectory driving simulation. Beyond real driving scenes, DriveX can also be utilized to simulate virtual driving worlds from AI-generated videos. Figure 1 : Though conventional reconstruction method can well fit views on the recorded trajectory, they struggle to extrapolate onto the views along the novel trajectories ( top ). In contrast, our method significantly improves the synthesis quality on these novel views ( bottom ), thereby achieving driving scene synthesis on free-form trajectories. 1 Introduction Building virtual driving worlds engagible with driving policies plays a pivotal role in developing robust autonomous driving systems. It enables automatic synthesis of diverse driving data, including safety-critical long-tail scenarios, edge cases, and providing closed-loop evaluation for end-to-end autonomous driving systems. Recent advancements in novel view synthesis Mildenhall et al. 2020 ; Kerbl et al. 2023 have driven reconstruction-based driving simulation Xie et al. 2023 ; Chen et al. 2023 ; Yan et al. 2024 due to its ability to synthesize photorealistic sensor data. These methods aim to reconstruct driving environments from individual driving sequences, typically involving single-trajectory videos from surrounding cameras with sparse views, minimal overlap, and textureless areas. Despite impressive results in fitting training views and interpolating along original trajectories, they still struggle to extrapolate substantially novel views far from the recorded trajectory, leading to limited flexibility and usefulness for driving simulation. To address these problems, a couple of latest works Hwang et al. 2024 ; Yu et al. 2024b explored image generative prior as extra novel view regularization. However, they are largely limited in design due to (i) the need for fine-tuning an off-the-shelf image generation models to obtain the capability of view transformation, suffering from both additional computational cost and per-scene data scarcity, and (ii) absence of the desired spatiotemporal knowledge within image models. To tackle these issues, a concurrent work Zhao et al. 2024a uses a pretrained video diffusion model to generate novel trajectory training images conditioned on initial recorded frames. Without sufficient content constraint but with intrinsic hallucination risk, this approach however tends to drift away from the original scene in scene generation, making troubles for the subsequent model optimization. To address the challenges mentioned above, we propose a novel driving scene synthesis approach for free-form trajectories, termed DriveX , by leveraging video generative prior with rich spatiotemporal knowledge during the model ( e.g . , Gaussian splatting) optimization. This is achieved by designing an inverse problem that enables a video diffusion model to serve as a prior for optimizing the Gaussian over a variety of trajectories. During training, we render novel trajectory views using the in-training model. To identify potential artifacts from trajectory extrapolation, we compare each rendered view with recorded images wrapped under the same view and geometric conditions, allowing us to obtain reliable regions for each rendered image. The inverse problem is then crafted as restoring each rendered view via utilizing its corresponding reliable regions as conditioning inputs, which can be effectively executed by the video diffusion model. The output from this process is subsequently fed into the optimization as supervision to update the target model. To seamlessly exploit the generative prior, we iteratively conduct this process during Gaussian optimization. As shown in Fig. 1 , our approach substantially improve the view synthesis quality in novel trajectory compared to the traditional reconstruction methods. The contributions of this paper are as follows: (i) We propose leveraging video generative prior with rich spatiotemporal knowledge for generalizable driving scene synthesis from single-trajectory recorded videos. (ii) We introduce DriveX, a novel driving scene synthesis framework for free-form trajectories that innovatively constructs an inverse problem to enable the use of a video diffusion model as a prior. (iii) Extensive experiments demonstrate that DriveX significantly outperforms existing state-of-the-art alternatives in driving scene synthesis using single-trajectory recorded videos. Additionally, we showcase the superiority of our method in rendering novel trajectories of virtual driving worlds from AI-generated videos, even when faced with inherent content inconsistencies. This approach enables a more economically scalable large-scale simulation without the need to collect videos for each scene. 2 Related works Reconstruction-based driving scene synthesis Early approaches Xie et al. 2023 ; Yang et al. 2024 ; Tancik et al. 2022 ; Wang et al. 2023b ; Turki et al. 2023 for dynamic urban scenes reconstruction focus on extending neural radiance fields (NeRF) Mildenhall et al. 2020 in large scale unbounded scenes.These methods are restricted with low efficiency and blurry rendering performance. Recently, another line of works introduced 3D Gaussian splatting Kerbl et al. 2023 into this task to produce real-time high-fidelity rendering. Some of them Yan et al. 2024 ; Zhou et al. 2024 leverage its favorable explicit property to decouple the whole scene into the static background and moving foreground Gaussians, while others modeling the motion of 3D Gaussians with periodic functions Chen et al. 2023 or deformation fields Huang et al. 2024 . However, driving scenes featured with sparse views, little overlaps, and large textureless areas, making existing methods struggle to generalize to substantively novel views. Generative-based driving scene synthesis Recently, propelled by the significant advancement of the diffusion model Ho et al. 2020 ; Song et al. 2020 ; Rombach et al. 2022 ; Song et al. 2023 ; Blattmann et al. 2023 , many works employ video generators to construct world models LeCun 2022 . Some of them introduce this paradigm into autonomous driving Wang et al. 2023a ; Zhao et al. 2024b ; Li et al. 2023 ; Gao et al. 2023 ; Wang et al. 2024 ; Gao et al. 2024 ; Hu et al. 2023 ; Lu et al. 2025 for simulating the plausible visual outcomes under versatile driving controls. Despite the remarkable diversity of generated results, this approach is constrained by the absence of underlying 3D model. Consequently, they cannot promise geometry and texture consistency for the generated many-trajectory videos of the same scene. Figure 2 : The schematic illustration of DriveX . Top: Gaussian optimization with both recorded trajectory and generated novel trajectory as supervision. Bottom: Generative prior integration. Given a novel view rendered image from Gaussians \mathcal{G}_{t} , video diffusion is applied to generate its refined version with condition of unreliability mask. We omit the loss on recorded trajectory for clarity. Scene reconstruction with diffusion prior Leveraging generative

Read the complete text