Seeing Beyond Views: Multi-View Driving Scene Video Generation with Holistic Attention

Hannan Lu 1 Xiaohe Wu 1 Shudong Wang 2 Xiameng Qin 2 Xinyu Zhang 3 Junyu Han 2 Wangmeng Zuo 1 Ji Tao 2 1 Harbin Institute of Technology 2 Changan Automobile 3 The University of Adelaide · 2024

Read Seeing Beyond Views: Multi-View Driving Scene Video Generation with Holistic Attention by Hannan Lu 1 Xiaohe Wu 1 Shudong Wang 2 Xiameng Qin 2 Xinyu Zhang 3 Junyu Han 2 Wangmeng Zuo 1 Ji Tao 2 1 Harbin Institute of Technology 2 Changan Automobile 3 The University of Adelaide (2024) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2412-03520.txt.

Introduction

Hannan Lu and co-authors generate multi-camera driving videos for training self-driving systems, in a 2024 study. Fake scenes must stay consistent across views and across time, which split attention schemes often fail to hold, above all with fast objects. Their network, CogDriving, uses a diffusion transformer with a single holistic attention block. It binds space, time, and viewpoint at once.

A tiny controller, using about one percent of the usual control net's size, steers layout from bird's-eye maps. Training also ups the weight on key objects such as cars and people. On the nuScenes test set the system scored well on video quality. The authors pitch it as a source of realistic training scenes.

Catalog record

CREATORHannan Lu 1 Xiaohe Wu 1 Shudong Wang 2 Xiameng Qin 2 Xinyu Zhang 3 Junyu Han 2 Wangmeng Zuo 1 Ji Tao 2 1 Harbin Institute of Technology 2 Changan Automobile 3 The University of Adelaide
YEAR2024
COLLECTIONPAPERS
FORMATTXT
FILE SIZE42 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2412-03520

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Seeing Beyond Views: Multi-View Driving Scene Video Generation with Holistic Attention Hannan Lu 1 Xiaohe Wu 1 Shudong Wang 2 Xiameng Qin 2 Xinyu Zhang 3 Junyu Han 2 Wangmeng Zuo 1 Ji Tao 2 1 Harbin Institute of Technology 2 Changan Automobile 3 The University of Adelaide Abstract Generating multi-view videos for autonomous driving training has recently gained much attention, with the challenge of addressing both cross-view and cross-frame consistency. Existing methods typically apply decoupled attention mechanisms for spatial, temporal, and view dimensions. However, these approaches often struggle to maintain consistency across dimensions, particularly when handling fast-moving objects that appear at different times and viewpoints. In this paper, we present CogDriving, a novel network designed for synthesizing high-quality multi-view driving videos. CogDriving leverages a Diffusion Transformer architecture with holistic-4D attention modules, enabling simultaneous associations across the spatial, temporal, and viewpoint dimensions. We also propose a lightweight controller tailored for CogDriving, i.e . , Micro-Controller, which uses only 1.1% of the parameters of the standard ControlNet, enabling precise control over Bird’s-Eye-View layouts. To enhance the generation of object instances crucial for autonomous driving, we propose a re-weighted learning objective, dynamically adjusting the learning weights for object instances during training. CogDriving demonstrates strong performance on the nuScenes validation set, achieving an FVD score of 37.8, highlighting its ability to generate realistic driving videos. The project can be found at https://luhannan.github.io/CogDrivingPage/ . 1 Introduction Figure 1 : Decoupled Attention Gao et al. 2024a vs Holistic-4D Attention. Our Holistic-4D attention establishes straightforward cross-dimensional relationships, leading to explicit transmission of visual information and enhanced cross-view consistency. With the advancements in autonomous driving, Bird’s Eye View (BEV) perception techniques have become fundamental and achieved significant improvement in critical tasks such as 3D object detection Huang et al. 2021 ; Li et al. 2022 , map segmentation Jiang et al. 2023 ; Liu et al. 2023a , multi-object tracking Hu et al. 2022 ; Zhang et al. 2022 ; Zhang et al. , and 3D lane detection Huang et al. 2023 ; Chen et al. 2022 . The success of BEV perception techniques heavily relies on learning from multi-view videos. However, the collection and annotation of such data remain labor-intensive and challenging, especially in extreme weather conditions and rare scenarios. To resolve this challenge, recent methods Yang et al. 2023 ; Gao et al. 2024b ; Gao et al. 2024a ; Li et al. 2023 ; Wen et al. 2024 have increasingly turned to generative models, particularly diffusion-based models, to synthesize multi-view driving videos conditioned on BEV layouts. In contrast to conventional video generation models that focus primarily on maintaining temporal coherence within individual videos, i.e . , cross-frame consistency, multi-view video generation requires addressing both cross-frame and cross-view consistency simultaneously. The cross-view consistency refers to ensuring that the content captured across different camera perspectives within a given scene remains consistent. To this end, recent diffusion-based multi-view video generation methods Wen et al. 2024 ; Li et al. 2023 ; Gao et al. 2024a typically decouple the view dimension from the spatial-temporal dimensions and incorporate additional attention modules to model dependencies across different camera views. In practice, the consistency of cross-frame and cross-view is coupled together as shown in Fig. 1 . For example, when a vehicle moves from the Front-Left view at the t -th frame to the Front-Right view at the (t+2) -th frame, the method with decoupled attention Gao et al. 2024a fails to maintain a consistent appearance between frames. This limitation arises from the decoupling of attention modules, which leads to extensive implicit transmission for cross-dimensional associations. This significantly increases the learning complexity, making it challenging to maintain the consistency of content across dimensions. Therefore, seeing beyond views is crucial to multi-view consistency across frames, and a method capable of establishing holistic relationships between spatial, temporal, and viewpoint dimensions is essential for generating high-quality multi-view videos. In this work, we present CogDriving , a novel network designed for synthesizing consistent multi-view driving videos. CogDriving introduces a simple yet effective diffusion transformer equipped with holistic-4D attention module to simultaneously model associations across spatial, temporal, and view dimensions. By establishing complete and straightforward cross-dimensional relationships, it significantly simplifies the learning process and enhances the ability to generate consistent content across all dimensions. As illustrated in Fig. 1 , the direct cross-dimensional associations established through the holistic-4D attention mechanism in CogDriving ensure the consistency of the vehicle’s appearance across different frames and views. Alongside consistency, enabling controllability is a fundamental requirement in multi-view driving scene video synthesis. Controllability ensures that the generated multi-view content conforms to the BEV layouts that characterize the scene (road maps, 3D bounding boxes, camera poses, etc.). Prevailing methods Yang et al. 2023 ; Gao et al. 2024b ; Li et al. 2023 ; Wen et al. 2024 typically utilize ControlNet Zhang et al. 2023a , a widely adopted paradigm for conditional generation that duplicates the diffusion U-Net to process conditional information. In our case, duplicating the network with 4D attention to process conditional inputs is impractical, as it introduces excessive computational overhead and hinders network optimization and convergence. Inspired by the lightweight encoder design and cross-normalization mechanism introduced in ControlNeXt Peng et al. 2024 , we propose the Micro-Controller tailored to the multi-view generation tasks for the integration of conditioning controls. Specifically, to handle the various types of input conditions in driving video generation, we employ separate encoders and independent cross-normalization for each type of condition. Additionally, to achieve spatial-temporal alignment between each condition embedding and the multi-view latents, we deploy an encoder architecture with 3D convolution and temporal down-sampling layers. In street-view videos, object instances, such as cars and pedestrians, typically occupy a smaller spatial area relative to the background, such as roads, etc . This imbalance in spatial distribution leads the model to prioritize learning the background content generation rather than rendering object instances, which are key elements in autonomous driving systems. To enable the model to balance the generation of object instances and background content, we propose a re-weighted learning objective to dynamically assign learning importance to object instances during training. Without increasing computational overhead during inference, such a learning objective effectively enhances the capability of instance generation. Our main contributions can be summarized as follows: • We propose CogDriving, an innovative DiT equipped with holistic-4D attention module for multi-view video generation, which simultaneously models associations across spatial, temporal, and view dimensions. • We introduce a lightweight control branch for controllable generation, i.e . , Micro-Controller, which is tailored to our DiT with 4D attention, and contains only 1.1% of the parameters of the ControlNet branch, yet achieving competitive control over generated results. • We devise a re-weighted learning objective that emphasizes the supervised learning of object instances. • CogDriving attains an FVD score of 37.8 on the nuScenes dataset, highlighting its proficiency in generating high-quality driving-scene videos. Notably, experiments in BEV segmentation and 3D object detection demonstrate that our synthesized videos provide a substantial performance enhancement to the state-of-the-art BEV perception models, thereby validating its practical applications. 2 Related work 2.1 Video Generation Video generation has consistently attracted attention in the field of computer vision. Early studies primarily relied on autoregressive models Wu et al. 2021

Read the complete text