Point Cloud Segmentation of Agricultural Vehicles using 3D Gaussian Splatting

Alfred T. Christiansen, Andreas H. Højrup, Morten K. Stephansen, Md Ibtihaj A. Sakib, Taman S. Poojary, Filip Slezák, Morten S. Laursen, Thomas B. Moeslund, Joakim B. Haurum · 2025

Read Point Cloud Segmentation of Agricultural Vehicles using 3D Gaussian Splatting by Alfred T. Christiansen, Andreas H. Højrup, Morten K. Stephansen, Md Ibtihaj A. Sakib, Taman S. Poojary, Filip Slezák, Morten S. Laursen, Thomas B. Moeslund, Joakim B. Haurum (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2506-05009.txt.

Introduction

Alfred Christiansen and co-authors build synthetic data to train models that segment farm vehicles in three-dimensional point clouds. Their 2025 paper starts from a cost problem: real point clouds are pricey to gather and label by hand. Generic digital models may not match the machines that a working system must handle.

The pipeline uses a rendering method called 3D Gaussian Splatting to make realistic assets of several farm vehicles. The assets sit in a simulated scene scanned by a virtual LiDAR sensor. One segmentation model trained and tested only on synthetic data scored above ninety-one percent on a standard overlap measure. In some tests, synthetic training even beat training on real data, the authors report.

Catalog record

CREATORAlfred T. Christiansen, Andreas H. Højrup, Morten K. Stephansen, Md Ibtihaj A. Sakib, Taman S. Poojary, Filip Slezák, Morten S. Laursen, Thomas B. Moeslund, Joakim B. Haurum
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE40 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2506-05009

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Point Cloud Segmentation of Agricultural Vehicles using 3D Gaussian Splatting Alfred T. Christiansen, Andreas H. Højrup, Morten K. Stephansen, Md Ibtihaj A. Sakib, Taman S. Poojary, Filip Slezák, Morten S. Laursen, Thomas B. Moeslund, Joakim B. Haurum Abstract Training neural networks for tasks such as 3D point cloud semantic segmentation demands extensive datasets, yet obtaining and annotating real-world point clouds is costly and labor-intensive. This work aims to introduce a novel pipeline for generating realistic synthetic data, by leveraging 3D Gaussian Splatting (3DGS) and Gaussian Opacity Fields (GOF) to generate 3D assets of multiple different agricultural vehicles instead of using generic models. These assets are placed in a simulated environment, where the point clouds are generated using a simulated LiDAR. This is a flexible approach that allows changing the LiDAR specifications without incurring additional costs. We evaluated the impact of synthetic data on segmentation models such as PointNet++, Point Transformer V3, and OACNN, by training and validating the models only on synthetic data. Remarkably, the PTv3 model had an mIoU of 91.35%, a noteworthy result given that the model had neither been trained nor validated on any real data. Further studies even suggested that in certain scenarios the models trained only on synthetically generated data performed better than models trained on real-world data. Finally, experiments demonstrated that the models can generalize across semantic classes, enabling accurate predictions on mesh models they were never trained on. Figure 1 : Illustration shows the pipeline created for generating the synthetic point cloud dataset. Step (1) shows the process of gathering images of a combine harvester. Step (2) shows the combine harvester mesh, extracted using GOF. Step (3) shows an example of a combine harvester mesh employed in the Gazebo LiDAR simulation environment, where the LiDAR sensor is visualized as the red cylinder with rays depicting the LiDAR scans. Step (4) shows an example of a final, synthetically generated, point cloud of a combine harvester. 1 Introduction The demand for automation in the transportation sector for self driving vehicles has, in recent years, fueled research for having machines understand 3D environments Sun et al. 2020 . As a format to capture 3D information, point clouds are increasingly popular due to their efficiency and direct representation of LiDAR data. Commonly, for the models to understand the composition of the environment represented by the point cloud, semantic segmentation is used as it provides per-point class-wise information, thus getting an understanding of the complete environment Qi et al. 2017a ; Qi et al. 2017b ; Wang et al. 2019b ; Peng et al. 2024 ; Zhao et al. 2021 ; Wu et al. 2022 ; Wu et al. 2024 ; Li et al. 2018 . An example of this is autonomous driving, where point cloud semantic segmentation is used to identify vehicles, road signs, pedestrians, etc. Sun et al. 2020 . However, obtaining and annotating point clouds is a well-known tedious and expensive task, making the available training resources sparse compared to the training datasets for image recognition. This has led a wave of research into both using the point clouds more efficiently by augmenting them Chen et al. 2020 ; Li et al. 2020 , and methods for generating new, annotated point clouds synthetically using simulated environments relying on existing 3D assets Wang et al. 2019a ; Yue et al. 2018 ; Ma et al. 2020 . The issue with using existing 3D assets arises when models need training for niche tasks with a sparse amount of available 3D assets. As a method to mitigate this issue, we propose the use of 3D Gaussian splatting (3DGS) to generate 3D assets that can be used to generate synthetic point cloud datasets. 3DGS is used due to the ease of only needing a digital camera for capturing useful data, and a few hours of GPU time to get a mesh. In this project, the method is tested as a use case to generate and train point cloud semantic segmentation models in a self driving tractor scenario, with synthetic data acquired using the pipeline shown in Figure 1 . This is a motivating domain for the proposed pipeline, as the existing 3D data is sparse and the meshes needed would be hard to model from scratch. As a proof of concept, the class count is reduced to: tractor, combine harvester, and other, as these vehicles often carry out collaborative tasks. Our key contributions are as follows: • We present a novel way of generating high-quality LiDAR data from 3DGS. • We demonstrate that by training a model purely on synthetic data it is possible to achieve high performance on real world data. • We show that our method generalizes to objects not seen during training, highlighting its robustness and adaptability. • Our method allows seamless modification of LiDAR-specific settings and the choice of mesh models used in the simulation, enabling greater adaptability to different sensing conditions and environments. 2 Related work 2.1 Point Cloud Semantic Segmentation Segmentation of 3D points is a common problem in the field of 3D computer vision. Many methods have been proposed to tackle the challenge of segmenting the unstructured and irregular-spaced point clouds Qi et al. 2017a ; Qi et al. 2017b ; Wang et al. 2019b ; Peng et al. 2024 ; Zhao et al. 2021 ; Wu et al. 2022 ; Wu et al. 2024 ; Li et al. 2018 . These methods can be categorized into two categories: point-based models and voxel-based models Peng et al. 2024 . Point-based methods directly manipulate the unstructured point clouds using point-wise operations. One such method is PointNet Qi et al. 2017a , which introduced a novel deep learning architecture used for point cloud semantic segmentation. However, the network cannot capture local structures due to the lack of neighborhood processing, limiting its ability to recognize fine-grained patterns and generalize to complex scenes. This was improved in PointNet++ Qi et al. 2017b by partitioning the point set into local overlapping regions, and then recursively applying PointNet to obtain the local features. Dynamic Graph CNN, proposed by Wang et al . Wang et al. 2019b , is another method which expands upon the PointNet architecture, this time by implementing a new convolutional operation, EdgeConv, which allows the extraction of the local neighborhood features for each point. Another prominent point-based method is the transformer-based models, which leverages the attention mechanism Vaswani 2017 that has gained prominence across various domains, including point cloud segmentation. The Point Transformer V3 (PTv3) Wu et al. 2024 advances point cloud segmentation by emphasizing scalability and efficiency. Unlike its predecessors, PTv1 Zhao et al. 2021 and PTv2 Wu et al. 2022 , which introduced local attention mechanisms, grouped vector attention, improved position encoding, and partition-based pooling, PTv3 focuses on the benefits of scaling rather than architectural complexity. It replaces the K-Nearest Neighbors (KNN) query with serialized patterns like Z-order and Hilbert curves, significantly reducing computational overhead, as KNN accounted for 28% of PTv2’s forward time. Additionally, PTv3 refines the positional encoding, further enhancing the model’s efficiency and ability to handle large-scale point clouds. Voxel-based methods convert the point clouds into a voxel grid during data pre-processing. A recent paper by Peng et al .

Read the complete text