Boosting LiDAR-Based Localization with Semantic Insight: Camera Projection versus Direct LiDAR Segmentation
Sven Ochs, Philip Schörner, Marc René Zofka, J. Marius Zöllner · 2025
Read Boosting LiDAR-Based Localization with Semantic Insight: Camera Projection versus Direct LiDAR Segmentation by Sven Ochs, Philip Schörner, Marc René Zofka, J. Marius Zöllner (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2509-20486.txt.
Introduction
Self-driving systems locate themselves better when they know what surrounds them. Sven Ochs and three co-authors add that semantic knowledge to laser-based localization in a 2025 arXiv paper. Two routes supply the semantics: segment the laser data directly, or borrow labels from camera images. Diverse sensors make the direct route brittle, so the authors pursue the camera route.
Their method projects laser points into the camera's segmented image, attaching object labels to each point. A state-of-the-art depth network produced the camera segmentation, while an adaptive network handled the laser side. Satellite positioning with real-time corrections provided ground truth. Validation included a fifty-five kilometer drive through Karlsruhe, Germany, spanning city streets, multi-lane roads, and rural highways. The fused approach improved the precision and reliability of the localization pipeline.
Catalog record
| CREATOR | Sven Ochs, Philip Schörner, Marc René Zofka, J. Marius Zöllner |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 25 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2509-20486
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Boosting LiDAR-Based Localization with Semantic Insight: Camera Projection versus Direct LiDAR Segmentation Sven Ochs, Philip Schörner, Marc René Zofka, J. Marius Zöllner Abstract Semantic segmentation of LiDAR data presents considerable challenges, particularly when dealing with diverse sensor types and configurations. However, incorporating semantic information can significantly enhance the accuracy and robustness of LiDAR-based localization techniques for autonomous mobile systems. We propose an approach that integrates semantic camera data with LiDAR segmentation to address this challenge. By projecting LiDAR points into the semantic segmentation space of the camera, our method enhances the precision and reliability of the LiDAR-based localization pipeline. For validation, we utilize the CoCar NextGen platform from the FZI Research Center for Information Technology, which offers diverse sensor modalities and configurations. The sensor setup of CoCar NextGen enables a thorough analysis of different sensor types. Our evaluation leverages the state-of-the-art Depth-Anything network for camera image segmentation and an adaptive segmentation network for LiDAR segmentation [ 1 ] . To establish a reliable ground truth for LiDAR-based localization, we make us of a Global Navigation Satellite System (GNSS) solution with Real-Time Kinematic corrections (RTK). Additionally, we conduct an extensive 55 km drive through the city of Karlsruhe, Germany, covering a variety of environments, including urban areas, multi-lane roads, and rural highways. This multimodal approach paves the way for more reliable and precise autonomous navigation systems, particularly in complex real-world environments. I INTRODUCTION Accurate and robust localization is a fundamental requirement for autonomous vehicles in complex environments, such as urban canyons and parking spaces. LiDAR sensors, with their ability to provide high-resolution depth information, play a crucial role in modern localization systems. However, LiDAR-based localization methods often struggle in challenging conditions, such as feature-sparse environments, dynamic scenes, or adverse weather conditions. To address these limitations, semantic information can be integrated into localization frameworks, enhancing the robustness and accuracy of pose estimation. Recent advancements in machine learning, particularly deep neural networks (DNN), have led to significant improvements in the semantic segmentation of LiDAR point clouds. These methods can classify point cloud data into meaningful categories, providing valuable contextual information for localization. However, LiDAR-only semantic segmentation approaches face inherent challenges, including sensor sparsity, domain adaptation issues, and computational complexity [ 1 ] . (i) Segmentation of the camera image (ii) Segmentation through projection into a segmented camera image (iii) Direct segmentation of whole LiDAR scan (iv) Resulting SLAM-Map utilizing semantic information Fig. 1 : Overview of the essential components of our pipeline. Fig. 1i shows the segmentation result from one of the eight cameras. This segmentation is projected onto the LiDAR point cloud in Fig. 1ii , and the process is repeated for all cameras to obtain a complete segmentation. Fig. 1iii illustrates the direct segmentation of the LiDAR point cloud using the approach from [ 1 ] . Finally, Fig. 1iv presents the semantic maps generated from both segmentation strategies. In contrast, leveraging semantic information from camera images, it is possible to improve LiDAR-based localization without solely relying on point cloud segmentation. This paper proposes an alternative approach integrating semantic features from camera images into a LiDAR-based localization pipeline through projection techniques. By mapping semantic labels from camera images onto the LiDAR point cloud, we aim to enhance localization performance without requiring direct LiDAR point cloud segmentation. This paper evaluates the proposed method compared to state-of-the-art DNN LiDAR point cloud segmentation techniques. Specifically, we analyze the impact of semantic camera integration on localization accuracy and robustness to environmental variations. Our contributions are as follows: • An approach for LiDAR-based localization that integrates semantic camera information through projection. • A comparative analysis of our method with state-of-the-art machine learning-based semantic segmentation of LiDAR point clouds. • Experimental validation demonstrates the advantages and disadvantages of localization accuracy. The remainder of this paper is structured as follows: Section II provides an overview of related work, including machine learning-based LiDAR segmentation and camera-LiDAR fusion techniques. Section III details the proposed methodology, including the semantic projection pipeline. Section IV presents experimental results. Finally, Section V concludes the paper with potential future research directions. II RELATED WORK Fig. 2 : Overview of the pipeline used in our work. The top row illustrates the camera-based segmentation pathway, where the LiDAR point cloud is projected into the image domain to generate a labeled point cloud. The bottom row depicts the direct LiDAR-based semantic segmentation pathway, where the point cloud is segmented without projection. The resulting data is independently used for the same mapping process in both cases. II-A LiDAR-Segmentation Haidar et al. [ 2 ] evaluate the performance of various 3D semantic segmentation neural networks on resource-constrained NVIDIA Jetson platforms. Utilizing standardized training protocols and data augmentations, the study benchmarks these models using the SemanticKITTI and nuScenes datasets. Findings indicate that while some models achieve high accuracy, they often fail to meet real-time processing requirements on embedded systems. Conversely, models optimized for speed tend to compromise on segmentation accuracy. This highlights the ongoing challenge of balancing computational efficiency and performance in deploying real-time LiDAR semantic segmentation for autonomous vehicles. SalsaNext [ 3 ] by Cortinhal et al. proposes an enhanced neural network for real-time, uncertainty-aware semantic segmentation of 3D LiDAR point clouds in autonomous driving. It improves upon SalsaNet [ 4 ] with a Global Context Module, a refined encoder-decoder architecture, and pixel-shuffle upsampling for better segmentation. A combined loss function of weighted cross-entropy and a Lovász-Softmax optimizes the Jaccard index, improving accuracy. Bayesian uncertainty estimation provides confidence scores for each point’s prediction. TransRVNet by Cheng et al. [ 5 ] integrates transformer-based modules like Multi Residual Channel Interaction Attention Module, Residual Context Aggregation Module, and Balanced Non-square-Transformer Module. These modules effectively capture local and global contextual information within point cloud data. Experimental evaluations demonstrate that TransRVNet performs better than existing state-of-the-art methods on benchmark datasets. Nevertheless, they did not upload the results to the leaderboard of the SemanticKITTI semantic segmentation task. Yan et al. [ 6 ] introduce a training scheme that enhances 3D point cloud semantic segmentation by leveraging 2D image data during training. 2DPASS utilizes an auxiliary modal fusion and multi-scale fusion-to-single knowledge distillation (MSFSKD) approach to transfer rich semantic and structural information from 2D images to a pure 3D network. This enables the model to perform effective segmentation using only point cloud inputs during inference. Hou et al. [ 7 ] present an approach to improve lightweight LiDAR segmentation models by distilling knowledge from larger, more complex teacher models. This method addresses challenges inherent in point cloud data, such as sparsity and varying density, by transferring knowledge at both the point and voxel levels. Key components include point and voxel-wise output distillation to complement sparse supervision signals and a difficulty-aware sampling strategy emphasizing less frequent classes and distant objects. In addition, the approach employs inter-point and inter-voxel affinity distillation to capture structural information effectively. II-B Image Segmentation Segformer by Xie et al. [ 8 ] combines Transformers with lightweight multilayer perceptron (MLP) decoders for semantic segmentation tasks. SegFormer features a hierarchically structured Transformer encoder that produces multiscale features without relying on positional encoding, enhancing flexibility across varying image resolutions. It’s a simple MLP decoder that effectively aggregates information from different layers, integrating local and global attention