Gaussian Splatting is an Effective Data Generator for 3D Object Detection

Farhad G. Zanjani, Davide Abati, Auke Wiggers, Dimitris Kalatzis, Hong Cai, Amirhossein Habibian · 2025

Read Gaussian Splatting is an Effective Data Generator for 3D Object Detection by Farhad G. Zanjani, Davide Abati, Auke Wiggers, Dimitris Kalatzis, Hong Cai, Amirhossein Habibian (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2504-16740.txt.

Introduction

Farhad G Zanjani and co-authors grow extra training scenes for self-driving cars inside reconstructions of real streets. Detectors learn 3D object finding from labeled examples, and labels are dear. Using Gaussian splatting, the team rebuilds a street in 3D, then parks extra virtual cars and objects into the geometry itself, where their positions and poses are known exactly by construction.

On the nuScenes benchmark, even a modest number of inserted objects lifted detection scores clearly, beating image-generating diffusion methods that paste from a bird's-eye plan. Variety in placement geometry helped more than variety in appearance. Two tempting tricks, hunting the hardest examples by loss or by hiding objects, did not pay off. Physical plausibility, the authors conclude, is the augmentation that counts.

Catalog record

CREATORFarhad G. Zanjani, Davide Abati, Auke Wiggers, Dimitris Kalatzis, Hong Cai, Amirhossein Habibian
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE40 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2504-16740

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Gaussian Splatting is an Effective Data Generator for 3D Object Detection Thanks: Qualcomm AI Research is an initiative of Qualcomm Technologies, Inc. Farhad G. Zanjani, Davide Abati, Auke Wiggers, Dimitris Kalatzis, Hong Cai, Amirhossein Habibian Abstract We investigate data augmentation for 3D object detection in autonomous driving. We utilize recent advancements in 3D reconstruction based on Gaussian Splatting for 3D object placement in driving scenes. Unlike existing diffusion-based methods that synthesize images conditioned on BEV layouts, our approach places 3D objects directly in the reconstructed 3D space with explicitly imposed geometric transformations. This ensures both the physical plausibility of object placement and highly accurate 3D pose and position annotations. Our experiments demonstrate that even by integrating a limited number of external 3D objects into real scenes, the augmented data significantly enhances 3D object detection performance and outperforms existing diffusion-based 3D augmentation for object detection. Extensive testing on the nuScenes dataset reveals that imposing high geometric diversity in object placement has a greater impact compared to the appearance diversity of objects. Additionally, we show that generating hard examples, either by maximizing detection loss or imposing high visual occlusion in camera images, does not lead to more efficient 3D data augmentation for camera-based 3D object detection in autonomous driving. 1 Introduction The accurate detection of street objects in 3D is a central perception problem in automotive compute vision, and it represents a key requirement which modern assisted and automated driving solutions build on top of. One of the key challenges in deploying accurate detectors is still the collection of training data. Indeed, automotive applications involve complicated acquisition pipelines, typically comprising complex setups with multiple sensors and cameras and significant effort for labeling 3D objects [ 4 , 47 , 30 ] . For this reason, the idea to generate synthetic scenes to support the training of 3D perception models is a long standing in automotive [ 9 , 39 , 5 ] , and its appeal only increases as the research community continues to deliver improvements in generative models [ 28 , 3 , 49 ] . Modern strategies involve generating scenes compliant with a desired 3D layout ( i.e . , a set of 3D bounding boxes) by means of diffusion models. To achieve this, the layout is first transformed a control signal [ 38 , 2 ] or integrated into a text prompt [ 15 ] , and it is then fed to the model for scene rendering. Although this procedure showed great potential in synthesizing realistic scenes, it only involves an implicit conditioning on the desired layout, meaning that it offers no guarantees on whether the desired objects will be generated, nor on whether they will appear in the specified location and pose. The negative impact of such image/layout discrepancies to augmenting detector training remains unclear. Moreover, diffusion models are still extremely computationally intensive, and their application to automotive settings is currently limited to low resolution scenarios. Figure 1 : Image augmentation through 3D scene reconstruction and physically plausible object placement for 3D object detection. In this work, we therefore focus on augmentation via a 3D reconstruction approach, which allows to maintain explicit control over synthetic objects placement, resulting in a high geometric fidelity to their pose and positions in the scene. More specifically, we explicitly model the scene using a dynamic 3D Gaussian Splatting (3D-GS) approach [ 7 ] , and insert 3D assets into the scene in physically plausible locations by applying explicit geometric transformations. These synthetically augmented frames will have high location/pose accuracy by design while the photometric diversity is upper-bounded by the size of the 3D asset library. An overview of our approach is illustrated in Fig. 1 . We show that the resulting augmented frames boost the performance of 3D detectors in both monocular and multi-view settings. We further question the importance of appearance diversity by ablating the number of synthetic assets, finding that our solution outperforms diffusion-based baselines even in an extreme setup where a single object per category is used to augment the full dataset. Finally, we explore some alternative strategies for augmentation including varying the added object’s orientation and placing the object in hard locations for the detector. In this respect, our experiments show that random orientations and locations are more rewarding, possibly because they improve the variations in the data. Our contributions are as follows: • We investigate the potential of advanced 3D driving scene reconstruction to improve 3D object detection performance. To the best of our knowledge, this is the first study to successfully explore 3D detector augmentation using 3D Gaussian Splatting methods. • Through extensive experiments, we demonstrate that geometric diversity plays a more crucial role in 3D augmentation than photometric diversity, and that introducing even a limited number of 3D objects can outperform state-of-the-art diffusion-based data augmentation methods. 2 Related work 3D object detection in street scenes. In autonomous driving, 3D object detection is crucial, and approaches differ by sensors and temporal inputs. Monocular 3D detection, relying on single camera inputs, offers cost-effective but face challenges in depth perception. FCOS3D [ 35 ] is one of the camera-based monocular object detector shows good robustness in compared to other detectors [ 40 ] . Bird’s-eye-view (BEV) object detection aims at detecting objects in BEV space given multi-view 2D images [ 23 , 22 , 42 ] . SparseBEV [ 23 ] is one of the most efficient solutions to date, consisting of a query-based one stage detector with multiple decoder layers. In this paper, we use FCOS3D and a non-temporal configuration of SparseBEV to evaluate the effectiveness of the proposed data augmentation. 3D reconstruction for autonomous driving. Recent advancements in NeRF-based [ 24 ] scene reconstruction methods for dynamic autonomous driving, such as SUDS [ 34 ] and EmerNeRF [ 43 ] , have demonstrated impressive reconstruction capabilities. However, these methods model all dynamic elements using a single dynamic field, which limits their controllability and practicality as simulators. By decomposing the scene into separate components (represented as nodes in a scene graph), it becomes possible to control them individually. This approach is widely adopted in methods like Neural Scene Graphs [ 25 ] , UniSim [ 46 ] , MARS [ 37 ] , NeuRAD [ 33 ] , and ML-NSG [ 12 ] . Simulating and image rendering of a large-scale autonomous driving dataset with hundreds of scenes using NeRF-based models is computationally heavy and almost impractical. Recent 3D Gaussian splatting-based (3D-GS) [ 17 ] works, such as StreetGaussians [ 41 ] , DrivingGaussians [ 52 ] , and HUGS [ 50 ] , utilize the fast differentiable rendering pipeline of 3D-GS to address the computational hurdle. However, these methods typically handle only rigid objects due to the limitations of time-independent representations or deformation-based techniques. To address these limitations, OmniRe [ 7 ] proposes a Gaussian scene graph that incorporates various Gaussian representations for both rigid and non-rigid objects. This provides additional flexibility and controllability for diverse actors. In our work, we rely on OmniRe as the scene model to interact with the scene, and we explore its suitability to generate training data for 3D detectors. Diffusion methods for autonomous driving. Diffusion-based models, which model visual context

Read the complete text