SynthDrive : Scalable Real2Sim2Real Sensor Simulation Pipeline for High-Fidelity Asset Generation and Driving Data Synthesis
Zhengqing Chen, Ruohong Mei, Xiaoyang Guo, Qingjie Wang, Yubin Hu, Wei Yin, Weiqiang Ren, Qian Zhang · 2025
Read SynthDrive : Scalable Real2Sim2Real Sensor Simulation Pipeline for High-Fidelity Asset Generation and Driving Data Synthesis by Zhengqing Chen, Ruohong Mei, Xiaoyang Guo, Qingjie Wang, Yubin Hu, Wei Yin, Weiqiang Ren, Qian Zhang (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2509-06798.txt.
Introduction
Driving data for rare road events is synthesized from real scenes by Zhengqing Chen and seven co-authors in a 2025 autonomous driving paper. Sensor simulation is essential because dangerous or unusual cases are hard to capture on real roads. Game-engine simulators lack diversity, while learning-based ones cover only narrow object types like vehicles and demand heavy multi-sensor data.
Their system, SynthDrive, follows a real-to-simulation-to-real path built on three-dimensional generation. It automates the mining of real assets, the generation of new ones, and the synthesis of rare-case driving data, aiming at objects beyond a single category. The abstract presents the pipeline as scalable and aimed at high-fidelity sensor data for perception training. Broad, automated asset generation is the key claim: rare scenarios can be manufactured in variety rather than waited for on the road.
Catalog record
| CREATOR | Zhengqing Chen, Ruohong Mei, Xiaoyang Guo, Qingjie Wang, Yubin Hu, Wei Yin, Weiqiang Ren, Qian Zhang |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 30 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2509-06798
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
SynthDrive : Scalable Real2Sim2Real Sensor Simulation Pipeline for High-Fidelity Asset Generation and Driving Data Synthesis Zhengqing Chen, Ruohong Mei, Xiaoyang Guo, Qingjie Wang, Yubin Hu, Wei Yin, Weiqiang Ren, Qian Zhang Abstract In the field of autonomous driving, sensor simulation is essential for generating rare and diverse scenarios that are difficult to capture in real-world environments. Current solutions fall into two categories: 1) CG-based methods, such as CARLA, which lack diversity and struggle to scale to the vast array of rare cases required for robust perception training; and 2) learning-based approaches, such as NeuSim, which are limited to specific object categories (vehicles) and require extensive multi-sensor data, hindering their applicability to generic objects. To address these limitations, we propose SynthDrive , a scalable “real2sim2real” system that leverages 3D generation to automate asset mining, generation, and rare-case data synthesis. Our framework introduces two key innovations: 1) Automated Rare-Case Mining and Synthesis. Given a text prompt describing specific objects, SynthDrive automatically mines image data from the Internet and then generates corresponding high-fidelity 3D assets, which eliminates the need for costly manual data collection. By integrating these assets into existing street-view data, our pipeline produces photorealistic rare-case data, supporting rapid scaling to diverse assets including irregular obstacles and temporary traffic facilities. 2) High-Fidelity 3D Generation. We propose a hybrid asset generation pipeline that combines a geometry-aware LRM, iterative mesh optimization, and an improved texture fusion algorithm. Our approach achieves 0.0164 Chamfer Distance on the GSO dataset, outperforming InstantMesh by 14.1% in geometry accuracy, and achieves 19.05 PSNR (vs 16.84) for texture quality. This enables fine geometry details and high-resolution texture generation, which is essential for perception model training. Experiments demonstrate that SynthDrive -generated data improves the performance of downstream perception tasks (2D and 3D detection on rare objects) by 2-4% mAP. SynthDrive greatly lowers the data production cost and improves the diversity for corner-case data generation, showcasing extensive potential applications in the field of autonomous driving. Fig. 1: Comparison of different approaches for autonomous driving simulation. Left: CARLA [ 1 ] , a CG-based platform with manually crafted assets, lacking realism and scalability. Middle: NeuSim [ 2 ] , a vehicle-only asset reconstruction approach requiring complex multi-modality images and LiDAR data. Right: Our approach, a fully automated data synthesis framework capable of generating diverse generic assets. The figure shows the automatically-constructed asset bank that includes traffic elements, vehicles, and road infrastructure. I INTRODUCTION The development of robust autonomous driving systems underscores the critical need for the coverage and diversity of training data and rare scenarios, particularly those involving safety-critical corner cases. However, relying solely on collecting data from the real world is insufficient in capturing all rare objects and corner cases [ 3 ] . Therefore, simulating these infrequent traffic scenarios is paramount for the development of robust and secure vehicles. Existing computer graphics-based software, such as Carla [ 1 ] , use graphic rendering engines to depict these corner cases. However, they heavily rely on manually crafted 3D assets to construct specific driving scenes. 3D assets in such systems typically encompass a limited range of textures and lack realism, which restricts their capability to fully replicate the complexity and diversity of real-world environments. Another line of research works [ 4 , 2 , 5 , 6 ] employ 3D reconstruction techniques instead of handcrafting to create 3D assets, which greatly improves the realism of 3D assets. The reconstructed 3D models are then inserted into existing driving scenes to create rare scenarios. To further improve realism, lighting estimation [ 7 , 8 , 9 ] and style transfer [ 10 ] can be additionally applied to ensure the correctness of environment lighting and realism. However, these methods investigate 3D reconstruction only for limited object categories like vehicles and require complex multi-sensor inputs, lacking flexibility and scalability to generic objects. Different from previous methods, recent image-to-3D methods [ 11 , 12 , 13 , 14 , 15 , 16 , 17 , 18 , 19 ] could generate diverse 3D assets from single input images, offering the potential to address the above issues in current data synthesis approaches. Inspired by the above observations, we introduce SynthDrive (Fig. 2 ), a “real2sim2real” data generation pipeline to overcome the realism and scalability issues in existing pipelines [ 1 , 2 , 5 , 6 ] . The core concept of this approach is fully-automated rare-data mining and synthesis. Our objectives address four critical gaps in existing methods: 1) Generic Object Reconstruction : Unlike NeuSim [ 2 ] and CADSim [ 5 ] , which are confined to limited object categories, our system supports arbitrary objects (e.g., portable traffic lights, fallen debris) through automated asset mining. 2) Low Data & Computation Cost : In contrast to [7] requiring complex sensor inputs, SynthDrive reconstructs assets from a single image within 1 minute, without the need for additional manual data collection. 3) High-Fidelity Geometry & Texture : the proposed hybrid image-to-3D pipeline incorporates robust geometric and texture details, which is more suitable for perception model training compared with [ 14 , 20 ] . To achieve these objectives, SynthDrive is built with three core modules: Asset Mining, 3D Asset generation, and Scenario Synthesis . Given a text prompt describing specific objects ( e.g. , “portable traffic lights” or “irregular obstacles”), our system automatically mines image data from existing driving data and web-collected sources using CLIP-guided [ 21 ] retrieval. This eliminates the need for manual data collection and ensures the coverage of diverse rare objects for simulation. The key of generic 3D asset reconstruction lies in our high-fidelity 3D asset generation module, which transforms single images into detailed 3D models. We propose a hybrid image-to-3D pipeline that combines a geometry-aware LRM, iterative mesh optimization, and an improved texture fusion algorithm. Specifically, starting with a single input image, we first extract high-quality texture and geometry cues using a multi-view diffusion model [ 16 ] and an image-to-normal diffusion model [ 22 ] . The multi-view texture cues are utilized to initialize a coarse 3D model with a feed-forward large reconstruction model [ 14 ] , and the geometric surface normal cues are utilized to refine the coarse model through a normal-supervised explicit mesh optimization process, enabled by differentiable surface normal rendering. To further improve texture quality, we up-sample the texture cues by diffusion-based image super-resolution [ 23 ] . The high-resolution images are then comprehensively mapped and fused across multiple views to produce the final textured 3D models, maintaining high-fidelity details. Finally, the generated 3D assets are integrated into existing driving videos or reconstructed simulation environments for further synthetic data generation and closed-loop simulations. A complex HDR lighting estimation and post-video harmonizer are employed to ensure the correctness of shadows and colors. The road surface geometry is also reconstructed to ensure the correct placement of inserted 3D assets. It offers enhanced realism and diversity for the simulated scenarios, which are crucial for the rapid development of autonomous driving systems. To sum up, our contributions are three-fold: • We introduce SynthDrive , a cost-effective and scalable “real2sim2real” pipeline for automatic asset library construction and data synthesis for autonomous driving. • We propose a novel single-view reconstruction approach with enhanced mesh geometry