Scene-Aware Location Modeling for Data Augmentation in Automotive Object Detection
Jens Petersen, Davide AbatiAmirhossein Habibian Auke WiggersQualcomm AI Research {jpeterse,dabati,ahabibia,auke}@qti.qualcomm.com · 2025
Read Scene-Aware Location Modeling for Data Augmentation in Automotive Object Detection by Jens Petersen, Davide AbatiAmirhossein Habibian Auke WiggersQualcomm AI Research {jpeterse,dabati,ahabibia,auke}@qti.qualcomm.com (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2504-17076.txt.
Introduction
Jens Petersen and co-authors argue that where objects sit in a scene matters as much as how they look for training car detectors. Generative augmentation usually swaps objects into old layouts. Or it pastes them at random, unrealistic spots.
Their model learns likely placements from real scenes. It predicts where a new object can plausibly stand. A generative painter then fills that spot in. Layout realism lifts the detector's training value. Two automotive benchmarks set new state-of-the-art results. The gains beat realism-only and diversity-only rivals. Scene-aware placement is the missing third leg. It serves any detection task with structured scenes and set, fixed, known layouts.
Catalog record
| CREATOR | Jens Petersen, Davide AbatiAmirhossein Habibian Auke WiggersQualcomm AI Research {jpeterse,dabati,ahabibia,auke}@qti.qualcomm.com |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 52 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2504-17076
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Scene-Aware Location Modeling for Data Augmentation in Automotive Object Detection Jens Petersen, Davide AbatiAmirhossein Habibian Auke WiggersQualcomm AI Research {jpeterse,dabati,ahabibia,auke}@qti.qualcomm.com Abstract Generative image models are increasingly being used for training data augmentation in vision tasks. In the context of automotive object detection, methods usually focus on producing augmented frames that look as realistic as possible, for example by replacing real objects with generated ones. Others try to maximize the diversity of augmented frames, for example by pasting lots of generated objects onto existing backgrounds. Both perspectives pay little attention to the locations of objects in the scene. Frame layouts are either reused with little or no modification, or they are random and disregard realism entirely. In this work, we argue that optimal data augmentation should also include realistic augmentation of layouts. We introduce a scene-aware probabilistic location model that predicts where new objects can realistically be placed in an existing scene. By then inpainting objects in these locations with a generative model, we obtain much stronger augmentation performance than existing approaches. We set a new state of the art for generative data augmentation on two automotive object detection tasks, achieving up to 2.8 × 2.8\times higher gains than the best competing approach ( + 1.4 +1.4 vs. + 0.5 +0.5 mAP boost). We also demonstrate significant improvements for instance segmentation. Figure 1 : An original scene (top left) and three augmented frames using different location modeling and augmentation strategies. Generated objects are indicated by green bounding boxes. Our approach proposes locations that fit the original scene, resulting in novel compositions with high visual realism and challenging occlusion cases. Approaches that reuse original locations, even with minor modifications such as in GeoDiffusion [ 6 ] , generate frames with visual appearance diversity but limited location diversity. Approaches that add objects in random locations such as X-Paste [ 49 ] disregard the realism of the resulting layout and, in turn, of the generated frames. 1 Introduction Generative Data Augmentation describes the use of generative models to create synthetic data that extends the training corpus of a learning model. The appeal of “free” training data has long motivated related work [ 3 , 17 ] , but with the recent progress in large generative image models [ 20 , 34 , 35 ] the interest in this field has increased drastically, with promising successes in image classification [ 19 , 50 ] and object detection [ 49 , 36 , 15 ] . This includes automotive scenes [ 6 , 14 ] , the focus of this work, where the benefit of generative data augmentation is especially large, as edge case scenarios are often safety-critical and costly to acquire. Existing methods for training data augmentation usually concern themselves with improving the quality of generated objects, but they often neglect reasoning about their locations. Some approaches reuse original object locations from real frames [ 36 , 14 , 27 ] , possibly with minor modifications [ 6 , 38 ] , which results in augmented frames that are visual variations of the same scene. Alternatively, other methods add new objects in random locations [ 49 ] , completely ignoring the original scene composition, which results in unrealistic generations ( Fig. 1 ). In this work, we argue that object locations should also be considered a key component of data augmentation. To demonstrate this, we propose a scene-aware probabilistic location model that, given an existing scene, predicts where a new object should be placed. Specifically, our model parses the scene to extract depth and drivable space, and it factorizes the joint probability of object categories, their locations, and their dimensions into a series of simpler conditional densities, which can be sampled from with ancestral sampling. We then combine our location model with an inpainting diffusion model [ 34 ] to render objects in the predicted locations, yielding augmented frames that are both realistic and different from existing scenes. The result is a generative data augmentation technique that outperforms state-of-the-art approaches by a large margin, with a performance boost of up to 2.8\times w.r.t. the best competing approach ( +1.4 vs. +0.5 mAP boost). By modifying the inpainting model to produce both RGB and instance masks, we further demonstrate substantial gains in the instance segmentation setting. In summary, our contributions are the following: • We propose a scene-aware probabilistic location model that augments street scene layouts by placing new objects in realistic locations. • We combine our location model with an inpainting diffusion model to produce augmented frames for object detector training, where our performance boost is up to 2.8\times higher compared to state-of-the-art approaches. • By enabling the diffusion model to predict instance masks for generated objects, we further demonstrate substantial performance gains on instance segmentation. 2 Related Work Generative data augmentation. The use of synthetic instances to augment training data for vision tasks has recently become a common strategy [ 1 , 15 , 39 , 48 ] . For object detector augmentation specifically, synthetic objects need to be rendered precisely in specified locations in a frame. Most works reuse object locations from real data [ 36 , 27 ] , change only the background while leaving objects intact [ 30 ] , or perform minor modifications such as translation or removal of bounding boxes [ 6 , 14 , 13 ] . The focus of these works is therefore mostly on improving the realism of generated data. Another popular augmentation strategy is “cut-and-paste” [ 11 ] : placing segmented or generated objects in real backgrounds in random locations. Cut-and-paste approaches have shown to be very effective in both object detection and instance segmentation [ 15 , 49 , 12 , 40 ] , despite the low realism of the resulting image. These results indicate that adding new objects in new locations may be just as important as creating realistic images, echoing earlier findings [ 10 ] . Our work achieves both by adding new objects in realistic locations to augment frames. Layout generation and location modeling. Predicting object locations is related to layout completion and generation. Dedicated methods typically solve these tasks by modeling interactions at the bounding box level without additional context [ 23 , 41 , 16 , 26 , 22 ] , and are usually applied to design documents or other highly structured data. Approaches that take scene information into account to determine object locations do exist [ 45 , 51 ] , but often require paired training datasets of (empty) images and feasible object placements, which are not readily available for automotive scenes. They may also require an image of the segmented object to place [ 51 , 46 ] , or have only been shown to work for specific object categories such as cars and pedestrians [ 29 ] . In this work, we instead want to determine the location before such an object is available. Finally, some approaches reason about object locations in 3D space [ 24 , 9 , 37 ] , but this requires detailed 3D annotations, which are usually much harder to obtain than 2D annotations. Figure 2 : Overview of our augmentation pipeline. (A) We first use the location model to predict realistic