R3D2: R ealistic 3D Asset Insertion via D iffusion for Autonomous Driving Simulation
William Ljungbergh, Bernardo Taveira, Wenzhao Zheng, Adam Tonderski, Chensheng Peng, Fredrik Kahl, Christoffer Petersson, Michael Felsberg, Kurt Keutzer, Masayoshi Tomizuka, Wei Zhan · 2025
Read R3D2: R ealistic 3D Asset Insertion via D iffusion for Autonomous Driving Simulation by William Ljungbergh, Bernardo Taveira, Wenzhao Zheng, Adam Tonderski, Chensheng Peng, Fredrik Kahl, Christoffer Petersson, Michael Felsberg, Kurt Keutzer, Masayoshi Tomizuka, Wei Zhan (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2506-07826.txt.
Introduction
William Ljungbergh and co-authors drop photorealistic virtual cars into filmed streets for self-driving tests. Testing driverless systems needs endless dangerous scenarios, but hand-built simulators look fake and rebuilt real scenes freeze their objects in place. Their tool, R3D2, is a one-step diffusion model that inserts complete three-dimensional assets into neural reconstructions of real drives.
Trained on assets lifted from real traffic footage and restaged in virtual scenes, it learns to paint the shadows and lighting that sell the illusion, in real time. Judges and metrics rated the insertions markedly more realistic. The team shows text-prompted asset creation and objects carried between scenes, aiming at scalable, trustworthy validation.
Catalog record
| CREATOR | William Ljungbergh, Bernardo Taveira, Wenzhao Zheng, Adam Tonderski, Chensheng Peng, Fredrik Kahl, Christoffer Petersson, Michael Felsberg, Kurt Keutzer, Masayoshi Tomizuka, Wei Zhan |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 44 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2506-07826
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
R3D2: R ealistic 3D Asset Insertion via D iffusion for Autonomous Driving Simulation William Ljungbergh, Bernardo Taveira, Wenzhao Zheng, Adam Tonderski, Chensheng Peng, Fredrik Kahl, Christoffer Petersson, Michael Felsberg, Kurt Keutzer, Masayoshi Tomizuka, Wei Zhan Abstract Validating autonomous driving (AD) systems requires diverse and safety-critical testing, making photorealistic virtual environments essential. Traditional simulation platforms, while controllable, are resource-intensive to scale and often suffer from a domain gap with real-world data. In contrast, neural reconstruction methods like 3D Gaussian Splatting (3DGS) offer a scalable solution for creating photorealistic digital twins of real-world driving scenes. However, they struggle with dynamic object manipulation and reusability as their per-scene optimization-based methodology tends to result in incomplete object models with integrated illumination effects. This paper introduces R3D2, a lightweight, one-step diffusion model designed to overcome these limitations and enable realistic insertion of complete 3D assets into existing scenes by generating plausible rendering effects—such as shadows and consistent lighting—in real time. This is achieved by training R3D2 on a novel dataset: 3DGS object assets are generated from in-the-wild AD data using an image-conditioned 3D generative model, and then synthetically placed into neural rendering-based virtual environments, allowing R3D2 to learn realistic integration. Quantitative and qualitative evaluations demonstrate that R3D2 significantly enhances the realism of inserted assets, enabling use-cases like text-to-3D asset insertion and cross-scene/dataset object transfer, allowing for true scalability in AD validation. To promote further research in scalable and realistic AD simulation, we will release our dataset and code, see project page . Figure 1 : R3D2 enables realistic insertion of 3D assets from different origins into existing scene reconstructions. From left to right, assets are sourced from: the same scene, a different scene, a different dataset, and a text-to-3D generative model. Bottom: naive insertion, top: insertion + R3D2. 1 Introduction Ensuring safety remains one of the most important—if not the most important—aspects of autonomous driving (AD) development. To achieve this, autonomous vehicles (AVs) must be validated across a wide range of scenarios, covering environmental variations as well as diverse road-user appearances and behaviors. However, validating AVs at this scale in the real world is costly, time-consuming, and potentially dangerous in the case of safety-critical testing. Therefore, virtual environments have emerged as a crucial tool for testing and validating AVs in a safe, controlled, and scalable manner. Although traditional simulators (e.g., CARLA [ 7 ] ) offer high controllability, the creation of environments and assets for these platforms is resource-intensive and often results in a domain gap with real-world AV data. In remedy, neural rendering has emerged as a scalable alternative for creating highly realistic and controllable digital clones of real world driving data. Recent advances in Neural Radiance Fields (NeRFs) [ 21 ] and 3D Gaussian Splatting (3DGS) [ 14 ] have demonstrated the ability to efficiently reconstruct driving scenes with minimal sim-to-real gap [ 16 ] . Such reconstructions can be used to create a large amount of safety-critical scenarios [ 18 ] to validate a full-stack AD system. Despite achieving impressive reconstruction quality, current neural rendering techniques like 3DGS still rely on per-scene optimization, which limits their ability to generate truly diverse and novel scenarios. For instance, road users are typically observed from limited viewpoints in driving data, leading to incomplete or partially reconstructed 3D models. Consequently, if these actors are moved or viewed from novel perspectives not present in the original recordings, significant visual artifacts can emerge. Furthermore, environmental effects such as illumination and shadows are often integrated into these partial reconstructions, restricting the reuse of assets in different scenes or under varying lighting conditions. Addressing these challenges could further enhance the scalability and versatility of neural rendering techniques in generating truly novel scenarios. Replacing foreground objects with high-quality 3D assets offers a promising path toward overcoming the aforementioned limitations of per-scene neural reconstructions. Yet, without scene-specific illumination and shadow effects, these assets often fail to integrate realistically into the environments they are placed in. Physically-based rendering (PBR) with ray tracing provides a principled solution to this challenge, but requires detailed knowledge of material properties (e.g., albedo, roughness, specularity) and scene lighting (e.g., source intensity, color, and direction)—information that is typically unavailable in neural reconstruction pipelines. Although there are works that seek to recover such parameters [ 41 , 15 ] , the problem remains fundamentally ill-posed and error-prone given the sparse, incomplete observations common in driving data. We explore an alternative approach—leveraging large generative models (LGMs) as priors to bypass the need for accurate scene-level modeling and physical attribute estimation while still enabling realistic insertion of assets into virtual environments. LGMs, when trained on internet-scale real world data, have proven effective in various applications including image in-painting [ 29 ] , text-to-3D generation [ 45 ] , and in improving the fidelity of neural rendering-based reconstruction methods [ 17 ] . However, they struggle at our particular task, either not maintaining overall scene consistency or failing to generate realistic rendering effects. To overcome these limitations, we propose R3D2, a lightweight, one-step diffusion model trained to enable realistic insertion of foreign 3D assets into existing scene reconstructions by generating plausible rendering effects like shadows and lighting. To train R3D2, we construct a novel dataset of 3DGS object assets derived from in-the-wild AD data using an image-conditioned large generative 3D model [ 45 ] . While these assets closely match the shape and appearance of its corresponding real-world objects, they initially lack realistic integration when placed into new scenes. We then leverage state-of-the-art AD scene reconstruction methods [ 9 ] to capture the original environment, replacing existing objects with our generated 3DGS assets. This process yields training pairs: scenes with realistically shaped and placed assets that still exhibit a sim-to-real gap in rendering. R3D2 is then trained to bridge this gap by learning the distribution of rendering effects from the original scenes, effectively realizing photorealistic inserted 3DGS assets. Beyond quantitative evaluations, we demonstrate R3D2’s utility through highly useful applications, including text-to-3D asset insertion and the flexible transfer of objects between different scenes and even datasets. Our contributions can be summarized as follows: • Propose a lightweight diffusion model, R3D2, capable of generating rendering effects in real time, drastically improving realism of inserted assets. • Demonstrate powerful asset manipulation capabilities while also generalizing to novel applications, including text-to-3D generation and seamless cross-dataset asset insertion. • Construct and release the novel dataset used to train R3D2, comprising both 3D assets and image pairs where the original actors are replaced with the aforementioned assets. 2 Related Work Simulation for autonomous driving. Simulation is indispensable for the development and testing of autonomous driving systems. Conventional methods [ 7 , 34 ] achieve high visual fidelity via Physically-Based Rendering (PBR) and ray tracing but rely on manually designed 3D assets and environments. Neural Radiance Fields (NeRFs) [ 21 ] and its adaptations to autonomous driving [ 39 , 49 , 38 , 48 ] bypass the need for manual modeling by enabling the generation of photorealistic 3D representations from real-world data. Compared to NeRF’s ray-tracing formulation, 3D Gaussian Splatting (3DGS) [ 14 ] uses rasterization to increase the rendering speed by orders of magnitude, allowing real-time simulation of full AD sensor suites [ 47 , 52 , 9