Automating 3D Dataset Generation with Neural Radiance Fields
Paul Schulz, Thorsten Hempel, Ayoub Al-Hamadi · 2025
Read Automating 3D Dataset Generation with Neural Radiance Fields by Paul Schulz, Thorsten Hempel, Ayoub Al-Hamadi (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2503-15997.txt.
Introduction
Paul Schulz, Thorsten Hempel, and Ayoub Al-Hamadi automate the making of 3D data sets in two thousand twenty-five. Finding objects in 3D is key for robots, augmented reality, and image search. Training needs large, well tagged sets, but they are slow and dear to make. Few public sets exist, with a narrow range of classes.
Their pipeline uses radiance fields to build high quality 3D models of any object. Those models feed a tool that makes fake training data. The pipeline is fast, simple, and largely self run. Pose nets trained on its data scored well in standard scenes. The work eases a key block in 3D vision.
Catalog record
| CREATOR | Paul Schulz, Thorsten Hempel, Ayoub Al-Hamadi |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 25 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2503-15997
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Automating 3D Dataset Generation with Neural Radiance Fields Paul Schulz, Thorsten Hempel, Ayoub Al-Hamadi Abstract 3D detection is a critical task to understand spatial characteristics of the environment and is used in a variety of applications including robotics, augmented reality, and image retrieval. Training performant detection models require diverse, precisely annotated, and large scale datasets that involve complex and expensive creation processes. Hence, there are only few public 3D datasets that are additionally limited in their range of classes. In this work, we propose a pipeline for automatic generation of 3D datasets for arbitrary objects. By utilizing the universal 3D representation and rendering capabilities of Radiance Fields, our pipeline generates high quality 3D models for arbitrary objects. These 3D models serve as input for a synthetic dataset generator. Our pipeline is fast, easy to use and has a high degree of automation. Our experiments demonstrate, that 3D pose estimation networks, trained with our generated datasets, archive strong performance in typical application scenarios. THIS WORK HAS BEEN SUBMITTED TO SPRINGER FOR PUBLICATION. COPYRIGHT MAY BE TRANSFERRED WITHOUT NOTICE, AFTER WHICH THIS VERSION WILL NO LONGER BE ACCESSIBLE. Figure 1 : Automated dataset generation with our pipeline. The process start with capturing images of an object, continues with 3D model creation and finishes with 3D dataset generation. 1 Introduction 3D scene perception and manipulation is a fundamental task in various fields where autonomous agents interact with their environment. Applications in autonomous driving, mobile robotics, medicine or manufacturing [ 1 , 2 , 3 , 4 ] and intuitive human-robot interaction systems [ 5 , 6 ] rely on this capability. To effectively perform scene manipulation, an agent has to estimate the pose of relevant objects within its environment. State-of-the-art pose estimation methods utilize neural networks that require large, diverse and precisely annotated 3D datasets for training. However, generating this data is a significant challenge, because collecting and annotating 3D datasets is labor-intensive and expensive, particularly when tailored datasets are required for specific applications. Recent works proposed approaches to automate 3D dataset generation, by incorporating 3D models of the objects of interest [ 7 , 8 , 9 , 3 , 10 , 11 , 12 ] . These methods either generate real or synthetic datasets. Automating real world dataset generation requires dedicated equipment and is limited in its field of application [ 10 , 11 , 12 ] . Synthetic dataset generation does not have these limitations, because it only requires 3D modeling software and the workflow can be easily adapted [ 7 , 8 , 9 , 3 ] . However, real-world and synthetic approaches assume that 3D models of the target objects are available. Creating these models with traditional methods, like 3D scanning or 3D modeling, is expensive and requires expert knowledge. Generating even a single high quality model for an individual target object may require, a textureless CAD model 3D scanning equipment and model refinement [ 13 ] We address these limitations by proposing a pipeline, which incorporates 3D model creation into the dataset generation process. To archive this, our pipeline creates 3D models from 2D images of target objects by utilizing Radiance Fields. Afterward, our pipeline incorporates these models in a state-of-the-art dataset generation workflow. In contrast to previous works, our pipeline generates datasets for the pose estimation of arbitrary objects without requiring 3D models. To verify our approach, we applied our pipeline to six objects of varying complexity and trained a 3D pose estimation model on our generated datasets. Furthermore, we tested the trained networks for two tasks: in-hand pose estimation and tabletop pose estimation. In summary, our contribution is an end-to-end dataset generation pipeline, which begins with arbitrary objects and ends with diverse, precisely annotated 3D datasets for these objects. 2 Related Works To the best of our knowledge, our method is the first to automate dataset generation by utilizing Radiance Fields for 3D model creation. Most similarly, Wong et al. [ 14 ] introduced a synthetic dataset generation pipeline, but in contrast to us, they use photogrammetry to create 3D models from 2D images. In the following, we provide an overview of synthetic dataset generation methods and Radiance Field frameworks that could potentially be integrated into our pipeline. 2.1 Automated Generation of Synthetic 3D Datasets Dataset Generation Methods Synthetic dataset generation takes place in virtual 3D environments, like game engines [ 8 ] or general 3D modeling software [ 7 ] . The dataset generation workflow, proposed by state-of-the-art publications, can be divided in two phases: scene composition and rendering + annotation. Scene composition constructs 3D scenes by placing objects in a virtual environment. These objects include target objects and additional content like distractors. The subsequent rendering + annotation process generates RGB images from the scenes, by applying 3D rendering software. Since all parameters of the 3D scene are known, the step also performs automated annotation [ 8 , 14 , 3 ] . Recent works can be divided in general and specific generation methods. General methods can be applied to arbitrary domains, specific method incorporate domain restrictions and are therefore applied in specific domains. Hodan et al. [ 7 ] created a general generator for the BOP challenge. Tremblay et al. [ 8 ] developed another general method for the pose estimation of arbitrary household objects. Both methods are suitable for arbitrary target objects, because they do not incorporate domain restrictions. In contrast to that, Sun et al. [ 15 ] developed a specific method, which creates datasets for the autonomous driving domain. Their framework incorporates special domain shifts for autonomous driving, like changing weather conditions. Other specific methods [ 9 , 3 ] tackle industrial tasks, like bin packing. These methods apply domain restrictions in the form of special industrial environment. Furthermore, McCormac et al. [ 16 ] applied a specific dataset generator to create a benchmark dataset for household scenes. Dataset Generation Frameworks Recent works also introduced dedicated frameworks to develop customized dataset generators [ 17 , 18 , 19 ] . These frameworks incorporate measurements to approach a central limitation of synthetic dataset generation, which is known as the simulation to reality gap (Sim2Real gap). The Sim2Real gap causes a performance drop, when networks trained on synthetic data are applied in the real world. The gap can be divided into the appearance gap and the content gap. The appearance gap is caused by differences in detailed, pixel-level scene appearances, resulting from a lack of realism [ 20 , 21 ] . The content gap is related to the context of simulated scenes. It appears when the scene content does not reflect the variety of objects and environments found in the real world [ 20 , 21 , 22 ] . Dataset generation frameworks incorporate photorealistic rendering and domain randomization to reduce the Sim2Real gap. Photorealistic rendering reduces the appearance gap by making objects and environments appear more realistic. Domain randomization reduces the content gap by randomizing the texture of objects of interest and/or the 3D environment [ 17 , 19 ] . 2.2 Radiance Fields In 2020, Mildenhall et al. [ 23 ] introduced Neural Radiance Fields (NeRF) for novel view synthesis. NeRFs combine the function approximation capability of multi layer