From Gaming to Research: GTA V for Synthetic Data Generation for Robotics and Navigations
Matteo Scucchia, Matteo Ferrara, Davide Maltoni · 2025
Read From Gaming to Research: GTA V for Synthetic Data Generation for Robotics and Navigations by Matteo Scucchia, Matteo Ferrara, Davide Maltoni (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2502-12303.txt.
Introduction
Matteo Scucchia, Matteo Ferrara, and Davide Maltoni turn a hit video game into a lab in a 2025 paper. Robot sight tools need vast photo sets from many weathers and places to cope in the real world. Real sets cost time and cash and can be flat-out out of reach.
Their set is drawn from the game world of Grand Theft Auto V, whose streets ape real ones in fine grain. It serves two tasks: mapping a route while you move, and knowing a place by sight. Fake sets can then be made on tap, in any light or storm. Vision teams gain cheap ground truth with full control.
Catalog record
| CREATOR | Matteo Scucchia, Matteo Ferrara, Davide Maltoni |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 44 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2502-12303
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
From Gaming to Research: GTA V for Synthetic Data Generation for Robotics and Navigations Matteo Scucchia, Matteo Ferrara, Davide Maltoni Abstract In computer vision, the development of robust algorithms capable of generalizing effectively in real-world scenarios more and more often requires large-scale datasets collected under diverse environmental conditions. However, acquiring such datasets is time-consuming, costly, and sometimes unfeasible. To address these limitations, the use of synthetic data has gained attention as a viable alternative, allowing researchers to generate vast amounts of data while simulating various environmental contexts in a controlled setting. In this study, we investigate the use of synthetic data in robotics and navigation, specifically focusing on Simultaneous Localization and Mapping (SLAM) and Visual Place Recognition (VPR). In particular, we introduce a synthetic dataset created using the virtual environment of the video game Grand Theft Auto V (GTA V), along with an algorithm designed to generate a VPR dataset, without human supervision. Through a series of experiments centered on SLAM and VPR, we demonstrate that synthetic data derived from GTA V are qualitatively comparable to real-world data. Furthermore, these synthetic data can complement or even substitute real-world data in these applications. This study sets the stage for the creation of large-scale synthetic datasets, offering a cost-effective and scalable solution for future research and development. 1 Introduction In the context of computer vision, numerous tasks require extensive datasets to train robust algorithms capable of generalizing effectively in real-world scenarios. Techniques based on Deep Learning necessitate substantial data volumes, often acquired under diverse environmental conditions (especially in outdoor scenarios) to ensure adaptability to varying contexts, including lighting conditions, weather variations, seasonal changes, and human interactions. The acquisition of such datasets poses hard challenges: it is a laborious task, time- and energy-consuming, often also costly, and not always feasible. To address these challenges, the generation of synthetic data has emerged as a promising solution. By creating data synthetically, researchers can simulate diverse environmental conditions in a controlled manner and generate vast amounts of data, thus substituting or integrating real-world datasets for algorithm training and improvement. For this purpose, synthetic data must be qualitatively close to real data, and this is not trivial, since real-world data can be difficult to replicate in a simulation, being subject to complex physical rules. The aim of this study is to demonstrate that synthetic data obtained from the famous video game Grand Theft Auto V (GTA V) [ 1 ] can be used in conjunction with or even in place of real datasets in robotics and navigation tasks. For this purpose, data were acquired from GTA V for Simultaneous Localization and Mapping (SLAM) and Visual Place Recognition (VPR). Several experiments were conducted to demonstrate the usability of these synthetic data in both tasks. The advantages of these acquisition methods are remarkable. For instance, a notable challenge in robotics is the long-term or lifelong SLAM [ 2 ] . This refers to a robot’s ability to deal with very large and long-term explorations, where it can repeatedly traverse the same path under varying conditions, with the environment evolving over time. To achieve accurate pose estimation, the robot must relocalize itself when revisiting the same locations. This problem is known as Loop Closure Detection (LCD) or relocalization [ 3 ] . To the best of our knowledge, there are very few existing SLAM datasets in which a robot retraces the same paths multiple times over many hours, under different lighting and weather conditions, within a single acquisition. This would require extensive effort in the real world, but it can be easily achieved by leveraging GTA V. Furthermore, in a simulated environment when a path is retraced we can introduce local pose perturbation to adds variability in the camera view and make LCD more challenging. Even if the idea of acquiring datasets from simulators and games like GTA V is not new, in this work we specifically focuses on synthetic data generation for robotic tasks, where real data are still predominantly used, particularly for VPR. Furthermore, we developed a simple but effective automatic algorithm for generating a VPR dataset from GTA V acquisitions, eliminating the need for human supervision. This work paves the way for the creation of unprecedentedly large datasets for all the aforementioned robotic vision tasks, composed solely of synthetic data from GTA V, which can be easily and rapidly enriched at no cost. The main novelties and contributions of this work are: • The creation of a novel synthetic dataset acquired from GTA V, specifically designed for use in robotics and navigation tasks. • The development of an autonomous frame selection algorithm to generate a VPR dataset from synthetic data acquired with GTA V. • The description of the pipeline and tools used to create the dataset to allow other researcher to easily generate further datasets tuned for their needs. • A series of experiments to support and validate the use of synthetic data acquired from GTA V for SLAM and VPR applications. Code and data will be released at https://github.com/scumatteo/GTA-rn upon acceptance. The rest of the paper is organized as follows: Section 2 introduces the data that can be extracted from GTA V and the computer vision tasks that can be addressed utilizing such data. Section 3 discusses existing SLAM and VPR datasets and presents other datasets generated using GTA V. Section 4 details the generation pipeline, while Section 5 and 6 present the dataset and the automatic frame selection algorithm for VPR. Section 7 reports the experiments conducted for VPR and SLAM, along with key insights. Finally, Section 8 provides conclusions and outlines future work. 2 Background With the rapid growth of deep learning, artificial neural networks have begun to be used to solve a wide variety of computer vision tasks, with outstanding performance. To train deep neural networks, researchers started to acquire large datasets, usually task-specific. 2.1 Type of data While some of the data commonly used in computer vision tasks can be easily acquired in the real world through fairly inexpensive sensors (e.g., RGB images), the acquisition of other data (e.g., 3D point clouds) requires expensive devices. Other data, such as bounding boxes or segmented images, require manual labeling or the use of semi-automated software tools, leading to a significant investment of time and resources. Hence, the opportunity to create highly accurate and realistic data without any expense by utilizing GTA V offers a remarkable benefit. RGB image : an RGB image uses the RGB color model to represent colors by combining red, green, and blue channels. Each pixel is represented by a triplet (R, G, B), with intensity values from 0 (no intensity) to 255 (full intensity). In GTA V it is the image captured from the screen during the game. Depth image : also known as depth map, it is a type of image that encodes the distance from the sensor to the objects in the scene. Unlike a regular RGB image that captures color information, a depth image captures spatial information, representing the three-dimensional structure of the scene. Usually, each pixel in a depth image represents the distance in millimeters from the objects, and it is encoded as a 16-bit single channel image, so the maximum theoretical distance is 2^{16}