DriveWeaver: Point-Conditioned Video Inpainting for Controllable Vehicle Insertion in Autonomous Driving Simulation

Junzhe Jiang Zipei Ma Zijie Pan Li Zhang · 2026

Read DriveWeaver: Point-Conditioned Video Inpainting for Controllable Vehicle Insertion in Autonomous Driving Simulation by Junzhe Jiang Zipei Ma Zijie Pan Li Zhang (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2606-31918.txt.

Introduction

Jiang and three others present DriveWeaver, a system that inserts vehicles into driving-simulation video with point clouds as the control signal. The sim needs diverse corner cases, so foreground vehicles must follow set trajectories inside recorded scenes. Tools based on pre-built 3D assets often clash with the background lighting and cannot scale beyond a small asset library.

DriveWeaver instead inpaints the masked insertion area, conditioned on the vehicle's point cloud. It gives footage that stays consistent over time and blends with its surroundings. A global-to-local inpainting plan preserves each inserted vehicle's identity over long sequences. Full 3D Gaussian forms are also pulled out for real-time rendering. Across datasets the method beats baselines on visual realism and geometric consistency.

Catalog record

CREATORJunzhe Jiang Zipei Ma Zijie Pan Li Zhang
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE37 KB
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2606.31918

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

DriveWeaver: Point-Conditioned Video Inpainting for Controllable Vehicle Insertion in Autonomous Driving Simulation Junzhe Jiang Zipei Ma Zijie Pan Li Zhang Abstract A pivotal step in autonomous driving simulation involves inserting foreground vehicles with predefined trajectories into simulated scenes. This process enhances scene diversity and facilitates the creation of various corner cases for testing and improving autonomous driving models. However, existing methods often rely on pre-reconstructed 3D assets, which frequently lead to lighting inconsistencies between the inserted foreground and the background. Moreover, the reliance on limited, manually-curated 3D assets hinders large-scale deployment. To address these challenges, we propose DriveWeaver , a novel framework for controllable vehicle insertion in autonomous driving simulation. Specifically, for a masked target insertion area, DriveWeaver performs video inpainting conditioned on vehicle point clouds to generate high-quality, temporally consistent vehicles. This video-inpainting-based approach ensures seamless blending between the foreground and background, while the readily available point cloud conditions enable superior generalization. To support long-term generation, we further design a global-to-local hierarchical inpainting strategy, ensuring the consistent identity and appearance of the inserted vehicles. Meanwhile, we extract explicit 3D Gaussian representations of the inserted vehicles through an urban reconstruction pipeline to enable real-time rendering for autonomous driving simulation. Extensive experiments across diverse datasets demonstrate that our method outperforms existing baselines in visual realism and geometric consistency, providing a robust tool for scalable autonomous driving scene augmentation. 1 Introduction Autonomous driving simulators have garnered significant attention due to their ability to evaluate driving algorithms in a safe and cost-effective manner [ 6 , 53 , 3 , 28 , 19 , 61 ] . Traditional simulators [ 6 , 36 ] employ manually-crafted CAD models to render autonomous driving scenarios. While offering high controllability, these simulators suffer from a pronounced domain gap compared to the real world. Recently, leveraging readily available real-world urban imagery, scene reconstruction techniques have provided a reliable solution for high-fidelity autonomous driving simulation. With advancements in neural rendering [ 47 , 52 , 2 , 49 , 4 , 30 ] which further augmented by generative priors to refine novel view synthesis [ 50 , 58 , 54 ] , contemporary methods have achieved high-quality, large-scale novel view synthesis of recorded scenes. Despite achieving photo-realistic rendering, these methods are primarily restricted to faithfully reproducing captured environments. The lack of the capability for flexible scene editing poses a significant barrier to evaluating driving algorithms in rare, long-tail corner cases. Figure 1 : Controllable high-fidelity vehicle insertion via DriveWeaver. Our point-conditioned inpainting framework provides flexible scene editing for autonomous driving simulation. It supports (a) diverse vehicle insertion options, (b) seamless insertion of vehicles with lighting in complex weather and illumination conditions and (c) flexible point-conditioned insertion using point clouds from diverse sources. To achieve high-fidelity foreground insertion for constructing safety-critical corner cases, several recent studies [ 47 , 61 , 64 ] leverage large-scale autonomous driving datasets [ 7 , 38 , 13 ] to pre-reconstruct 3D vehicle assets. These assets are then composed and rendered into background scenes according to predefined trajectories. To mitigate the visual inconsistencies between the inserted objects and the environment, various post-processing techniques have been employed, such as selecting vehicles with matching illumination and synthesizing shadows. Despite these efforts, a persistent stylistic gap remains between the foreground and background. Furthermore, these pipelines heavily rely on labor-intensive processes, including manual pose calibration for unannotated datasets [ 7 ] and the heuristic selection of lighting-compatible assets. Such dependencies on manual intervention significantly limit the scalability and generalizability across diverse real-world scenarios. Alternative approaches [ 24 , 3 , 29 , 19 ] employ manually-modeled 3D mesh assets as foreground objects and utilize diffusion models [ 35 ] as evaluators to provide guidance signals for physics-based inverse rendering. Through differentiable rendering, these methods enable the end-to-end optimization of lighting parameters. While achieving superior realism in vehicle insertion, these frameworks require per-scene optimization via diffusion models, which is computationally expensive. Furthermore, the acquisition of high-quality 3D meshes remains costly, making these methods difficult to adapt efficiently to diverse scenarios or to satisfy the requirements of large-scale deployment. In this paper, we present DriveWeaver , a novel point-conditioned video inpainting diffusion model. As shown in Fig. 1 , our framework achieves precise control over the synthesis of high-quality, geometrically consistent foreground vehicles using cost-effective point cloud data. Specifically, we leverage point cloud renderings as pixel-level conditions to provide rigorous geometric grounding for vehicle poses. By integrating these renderings as conditional inputs, DriveWeaver exploits the powerful generative priors of video diffusion models to fix missing structural details and ensure seamless environmental harmony. This pixel-level conditioning paradigm enables cross-dataset generalization during inference, allowing point clouds from diverse sources to guide the generation process. Consequently, our approach offers a scalable and versatile solution for foreground vehicle insertion in autonomous driving simulations. Furthermore, the superior geometric consistency of the synthesized videos enables the extraction of explicit 3D Gaussian representations for the inserted foreground vehicles via urban reconstruction pipelines. This capability facilitates real-time rendering, which is crucial for the large-scale training and evaluation of autonomous driving models. Our main contributions are summarized as follows: (i) We propose a novel video inpainting diffusion framework DriveWeaver designed for highly scalable and generalizable foreground vehicle insertion in autonomous driving simulation. Leveraging point cloud renderings as pixel-level conditions, DriveWeaver generates foreground vehicles with strong geometric consistency while preserving environmental illumination and style coherence. (ii) To support long-term generation, we develop a global-to-local hierarchical inpainting strategy that ensures the long-term temporal consistency of inserted foreground vehicles. Furthermore, the generated results can be directly reconstructed into 3D Gaussian representations, supporting efficient rendering for downstream autonomous driving model training and evaluation. (iii) Extensive experiments conducted on two large-scale benchmark datasets (Waymo [ 38 ] and PandaSet [ 46 ] ) demonstrate that DriveWeaver significantly outperforms existing state-of-the-art methods in terms of visual realism and geometric consistency. 2 Related work Neural rendering for autonomous driving simulation. Recent advancements in neural rendering have fundamentally transformed autonomous driving simulation by enabling high-fidelity, data-driven scene reconstruction directly from sensor logs. Early neural simulators [ 52 , 32 , 39 , 47 , 3 ] primarily relied on neural radiance fields (NeRF) [ 31 ] to parameterize large-scale urban environments. To address the computational bottlenecks and representation limitations of NeRF, recent approaches [ 2 , 49 , 62 , 4 , 30 , 51 ] have increasingly adopted 3D Gaussian splatting (3DGS) [ 21 ] for highly efficient rendering. Despite their remarkable reconstruction capabilities, these purely physical-based rendering methods often suffer from visual artifacts or blurring in unobserved regions when rendering extreme novel views, which frequently arise in autonomous driving simulation [ 62 , 64 , 19 , 10 ] . To alleviate this, contemporary frameworks [ 43 , 50 , 58 , 54 ] leverage the powerful generative priors of large-scale diffusion models to fix missing details and refine rendering quality. Although these neural rendering and generative frameworks achieve impressive realism when reconstructing recorded data, they lack the ability to edit scenes, which limits their applicability for evaluating driving algorithms in rare long-tail corner cases within autonomous driving simulators. This limitation motivates us to develop a point-conditioned video inpainting model that enables flexible editing

Read the complete text