A 2-Stage Model for Vehicle Class and Orientation Detection with Photo-Realistic Image Generation

Youngmin Kim 1 , Donghwa Kang 1 , Hyeongboo Baek 2 · 2025

Read A 2-Stage Model for Vehicle Class and Orientation Detection with Photo-Realistic Image Generation by Youngmin Kim 1 , Donghwa Kang 1 , Hyeongboo Baek 2 (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2506-01338.txt.

Introduction

Youngmin Kim, Donghwa Kang, and Hyeongboo Baek detect vehicle type and facing from synthetic training images. Models taught on fake images often falter on real ones, and class counts in the data are skewed. Their answer is a two-stage model fed by photo-realistic image generation.

In their 2025 paper, a meta table first stores each image with its class and object spots. Style transfer then turns synthetic shots into real-looking ones and folds them into the table. A classifier reads the merged set to name type and orientation, and stored spots complete the detection. The method placed high in the IEEE BigData Challenge on this task, proof the bridge from fake to real can hold.

Catalog record

CREATORYoungmin Kim 1 , Donghwa Kang 1 , Hyeongboo Baek 2
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE16 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2506-01338

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

A 2-Stage Model for Vehicle Class and Orientation Detection with Photo-Realistic Image Generation Thanks: 1 Co-leading authors. 2 Corresponding author. PubID: pubid: 978-1-6654-8045-1/22/$31.00 ©2022 IEEE Youngmin Kim 1 , Donghwa Kang 1 , Hyeongboo Baek 2 Abstract We aim to detect the class and orientation of a vehicle by training a model with synthetic data. However, the distribution of the classes in the training data is imbalanced, and the model trained on the synthetic image is difficult to predict in real-world images. We propose a two-stage detection model with photo-realistic image generation to tackle this issue. Our model mainly takes four steps to detect the class and orientation of the vehicle. (1) It builds a table containing the image, class, and location information of objects in the image, (2) transforms the synthetic images into real-world images style, and merges them into the meta table. (3) Classify vehicle class and orientation using images from the meta-table. (4) Finally, the vehicle class and orientation are detected by combining the pre-extracted location information and the predicted classes. We achieved 4 th place in IEEE BigData Challenge 2022 Vehicle class and Orientation Detection (VOD) with our approach. Our code and project material will be available at https://github.com/inu-RAISE/VOD_Challenge . I Introduction A large amount of data is required to train using a Transformer [ 1 ] or Convolution Neural Networks (CNN) [ 2 ] based model, which is in the spotlight in the field of computer vision. In addition, to apply this model to the real world using this model, it is necessary to collect and construct data from the real environment. However, this work has a challenge because it takes a lot of time and cost. To solve this challenge, we study using virtual synthetic data, which is relatively inexpensive, and then study to improve the model that can operate in a real-world driving environment. Our proposed approach follows: First, a table containing the object information of the image is built, and the location information of the object is extracted. After that, the styles of the synthesized object images are transformed into photo-realistic images. Using the transformed photo-realistic images and synthetic images, the vehicle class and orientation are classified, and location information is combined to achieve detection in real-world images. In this way, we achieved an accuracy that is about 8\%p higher than that of the existing object detection model trained only on synthetic data. This paper is summarized as follows: • We propose a two-stage model that detects the class and orientation of a vehicle by building a meta-table containing information about each object in images. • We propose a method for better prediction in the real world with photo-realistic image generation. In this paper, we introduce our proposed approach to detect vehicle class and orientation, and the rest of the paper is organized as follows: In Section 2, we survey synthetic datasets and related studies on object detection. In Section 3, we introduce our detailed approach. We share our experimental environment and experimental results in Sections 4 and 5, respectively. Finally, we conclude our paper in Section 6 by discussing our findings and the limitations of our paper. Fig. 1: Number of annotations by class Fig. 2: Overview of our proposed VOD approach. II Related Works II-A Synthetic Dataset Collecting labels that fit images and objects in the real world is an expensive and challenging process in object detection using deep learning. For this reason, a relatively inexpensive synthetic dataset is used to collect a relatively inexpensive dataset and perform object detection. Kumar et al . [ 3 , 4 , 5 ] collected a dataset that includes vehicle class and orientation and location information to identify traffic flow. This image dataset has a total of 63,066 synthetic images for training at 1920 x 1080 resolution. Also, a total of 3,000 real-world images were provided for testing at 1920 x 1080 and 1280 x 720 resolutions, a number of 1,500, respectively. There are 12 classes in the dataset, consisting of four types of vehicles (car, truck, motorcycle, cycle) and three orientations (back, front, side). The number of annotations for each class is shown in Fig. 1 . The FCAV dataset [ 6 ] is an image of a video game-based simulation engine as virtual data, and was created based on GTA V [ 7 ] . This dataset contains images of various situations because it was collected assuming driving in 4 weather conditions (sun, fog, rain, haze) and 4-time ranges (day, night, morning, dusk). There are a total of 205,879 training datasets for FCAV, consisting of three classes (person, car, and motorbike). For each class, there are 58 objects for a person, 1,125,624 objects for a car class, and 15,902 objects for a motorbike. We use only the training dataset, and the person class is excluded. II-B Object Detection The object detection task is to detect object instances of a specific class in images to perform both class classification and object localization. The methods for object detection can be categorized into two main types: There is a single-stage model that performs class classification and object localization simultaneously, and a two-stage model proceeds with class classification after object localization. The single-stage model has a relatively fast inference speed, and representative models include SSD [ 8 ] and YOLO [ 9 ] . On the other hand, the two-stage model has relatively high performance, and the representative model is the RCNN series [ 10 , 11 , 12 ] . We used YOLOv5 1 1 1 https://github.com/ultralytics/yolov5 , the latest version of YOLO, which has high accuracy and fast inference time on the COCO dataset [ 13 ] . III Proposed Approach To avoid learning bias due to imbalanced classes in the training data set, we used a method to solve the problem by separating it into regional proposals and vehicle class classification. Also, we transform the synthetic image into a photo-realistic image to make better predictions from real-world images. For this, we crop the image by object using the label of the training data and define training meta-table , \mathcal{M}=\{\mathcal{M}_{1},\cdots,\mathcal{M}_{n}\} , with information about the number of n objects. It is built with the object’s images dictionary, \mathcal{I}=[\mathcal{I}_{1},\cdots,\mathcal{I}_{n}] , classes dictionary, \mathcal{C}=[\mathcal{C}_{1},\cdots,\mathcal{C}_{n}] , and location dictionary, \mathcal{L}=[\mathcal{L}_{1},\cdots,\mathcal{L}_{n}] , according to \mathcal{I},\mathcal{C},\mathcal{L}\subset\mathcal{M} . Our approach using meta-table is illustrated in Figure 1 and summarized in the following four steps. 1) It stacks the image, class, and location information of the object extracted from the training image in the meta-table . 2) Transform synthetic images to photo-realistic images and creates a new meta-table , \mathcal{M}_{\text{cut}} . 3) The image classification model classifies the class and orientation of the vehicle by the image and class in the final meta-table \mathcal{M}^{*} , which combines \mathcal{M}_{\text{cut}} and the meta-table \mathcal{M} of the training data. 4) Finally, the vehicle class and orientation are detected by combining the predicted class and the location information of the \mathcal{M}^{*} . Fig. 3: Examples of transform result from synthetic image to photo-realistic image III-A Meta-table Representation III-A 1 Training It builds the training image dictionary, \mathcal{I}=[\mathcal{I}_{1},\cdots,\mathcal{I}_{n}] , location dictionary of training data, \mathcal{L}=[\mathcal{L}_{1},\cdots,\mathcal{L}_{n}] , and class dictionary, C=[\mathcal{C}_{1},\cdots,\mathcal{C}_{n}] , of the object with

Read the complete text