RGBDS-SLAM: A RGB-D Semantic Dense SLAM Based on 3D Multi Level Pyramid Gaussian Splatting

Zhenzhong Cao, Chenyang Zhao, Qianyi Zhang, Jinzheng Guang, Yinuo Song, Jingtai Liu · 2024

Read RGBDS-SLAM: A RGB-D Semantic Dense SLAM Based on 3D Multi Level Pyramid Gaussian Splatting by Zhenzhong Cao, Chenyang Zhao, Qianyi Zhang, Jinzheng Guang, Yinuo Song, Jingtai Liu (2024) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2412-01217.txt.

Introduction

Zhenzhong Cao and co-authors build a mapping system that rebuilds a scene's color, depth, and meaning together, using a pyramid of Gaussian splats. Their 2024 paper works in dense SLAM, where a moving camera must chart space in fine detail. Recent splat-based methods often lose detail or let the three maps drift apart.

Their method trains splats from stacked image pyramids, so fine and coarse detail both land in the model. A coupled optimization then lets each map sharpen the others during rendering. On two public indoor data sets the system beat the leading methods on the standard scores. The team promises open code, and frames the gain as detail kept without losing agreement across maps.

Catalog record

CREATORZhenzhong Cao, Chenyang Zhao, Qianyi Zhang, Jinzheng Guang, Yinuo Song, Jingtai Liu
YEAR2024
COLLECTIONPAPERS
FORMATTXT
FILE SIZE30 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2412-01217

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

RGBDS-SLAM: A RGB-D Semantic Dense SLAM Based on 3D Multi Level Pyramid Gaussian Splatting Zhenzhong Cao, Chenyang Zhao, Qianyi Zhang, Jinzheng Guang, Yinuo Song, Jingtai Liu Abstract High-fidelity reconstruction is crucial for dense SLAM. Recent popular methods utilize 3D gaussian splatting (3D GS) techniques for RGB, depth, and semantic reconstruction of scenes. However, these methods ignore issues of detail and consistency in different parts of the scene. To address this, we propose RGBDS-SLAM, a RGB-D semantic dense SLAM system based on 3D multi-level pyramid gaussian splatting, which enables high-fidelity dense reconstruction of scene RGB, depth, and semantics. In this system, we introduce a 3D multi-level pyramid gaussian splatting method that restores scene details by extracting multi-level image pyramids for gaussian splatting training, ensuring consistency in RGB, depth, and semantic reconstructions. Additionally, we design a tightly-coupled multi-features reconstruction optimization mechanism, allowing the reconstruction accuracy of RGB, depth, and semantic features to mutually enhance each other during the rendering optimization process. Extensive quantitative, qualitative, and ablation experiments on the Replica and ScanNet public datasets demonstrate that our proposed method outperforms current state-of-the-art methods, which achieves great improvement by 11.13% in PSNR and 68.57% in LPIPS. The open-source code will be available at: https://github.com/zhenzhongcao/RGBDS-SLAM . I INTRODUCTION Visual SLAM is a fundamental problem in the field of robotics, aimed at solving the problem of simultaneously locating a robot and constructing a map of its surrounding environment. Dense mapping is an important component of visual SLAM; on the one hand, it enables the robot to perceive its surroundings more comprehensively, and on the other hand, it provides a foundational map for downstream tasks such as grasping, manipulation, and interaction. However, traditional dense visual SLAM [ 1 , 2 , 3 , 4 , 5 , 6 ] relies solely on point clouds to reconstruct scenes, and due to the limited number of points and their discontinuous distribution, it faces significant bottlenecks and cannot achieve high-fidelity reconstructions of the environment. With the advent of NeRF (Neural Radiance Fields) [ 7 ] , scene representation based on implicit neural radiance fields has gradually become popular. Through training, the reconstruction accuracy has significantly improved, and many approaches have incorporated NeRF into SLAM [ 8 , 9 , 10 , 11 , 12 , 13 , 14 , 15 ] , achieving high-precision RGB, depth, and semantic Reconstructions. However, NeRF itself suffers from issues such as long training times and slow rendering speeds, meaning that NeRF-based SLAM solutions cannot run in real time, which contradicts the original goal of SLAM. 3D GS [ 16 ] technology, with its efficient optimization framework and real-time rendering capability, improves upon the shortcomings of NeRF. As a result, many 3D GS-based SLAM [ 17 , 18 , 19 , 20 , 21 , 22 , 23 , 24 ] solutions have emerged. However, these methods typically train using only raw image features, which are insufficient to fully capture the fine-grained details of certain scene parts, leading to poor reconstruction consistency. Moreover, when performing multi-feature reconstruction, these approaches do not effectively fuse and optimize the features through reasonable constraints, preventing them from mutually enhancing each other. To address the key issues of insufficient detail restoration, poor reconstruction consistency, ineffective fusion of multi-feature information, and real-time challenges in reconstruction, we propose the RGBDS-SLAM algorithm in this paper. First, we introduce a 3D multi-level pyramid gaussian splatting method, which constructs a multi-level image pyramid to extract rich detail information at different resolution levels and perform gaussian splatting training. This method significantly improves the scene’s detail restoration capability, and through stepwise optimization across levels, it ensures effective global consistency during reconstruction, providing a solid foundation for precise restoration of complex scenes. Second, we design a tightly coupled multi-features reconstruction optimization mechanism, which reasonably couples RGB, depth, and semantic features through various constraints. In the rendering optimization process, these three features collaborate and promote each other. Semantic information enhances depth understanding, depth information supports semantic refinement, and at the same time, the realism and consistency of RGB rendering are optimized, thereby comprehensively improving the accuracy and reliability of reconstruction. Finally, we develop a complete RGB-D Semantic Dense SLAM system, achieving high-quality dense reconstruction of scene RGB color, depth information, and semantic color. This system is based on the current classic ORB-SLAM3 algorithm [ 6 ] , capable of processing complex scenes in real time and meeting the dual requirements of speed and accuracy for online applications. Fig. 1: Overview of the proposed RGBDS-SLAM. Our method is an enhancement of ORB-SLAM3 [ 6 ] , taking RGB, depth, and semantic frames as input and outputting a map database with the point map, gaussian origin map, and gaussian semantic map. It consists of four threads: Tracking, LocalMapping, GaussianMapping, and LoopClosing. The main contributions of this work are as follows: • We introduce a 3D Multi-Level Pyramid Gaussian Splatting (MLP-GS) method, which extracts multi-level image pyramids for gaussian splatting training, restoring scene details and ensuring consistency during reconstruction. • We design a Tightly Coupled Multi-Features Reconstruction Optimization(TCMF-RO) mechanism, which promotes mutual improvement of RGB, depth, and semantic map reconstruction accuracy during the optimization rendering process. • We develop a complete RGB-D Semantic Dense SLAM system capable of high-quality dense reconstruction of scene RGB, depth, and semantic information, and the system can operate in real time. We will also open source our code once the paper is accepted. II Related Work II-A NeRF-based SLAM The development in neural implicit representations, particularly those based on NeRF, have significantly enhanced the performance of SLAM systems. Among them, NICE-SLAM [ 8 ] is the first solution to combine NeRF and SLAM, which incorporates multi-level local information by introducing a hierarchical scene representation, enabling efficient map construction and robust tracking. However, NICE-SLAM suffers from computational efficiency issues. Therefore, [ 9 ] , [ 10 ] , [ 11 ] and [ 12 ] have introduced voxel-based neural representations, coordinate and sparse parameters, hybrid representation of signed distance fields (SDF) and neural point cloud respectively to optimize and improve computational efficiency. The above solutions do not consider semantic mapping, so based on these solutions, NIDS-SLAM [ 13 ] introduce a novel approach for dense 3D semantic segmentation, based on 2D semantic color information of keyframes, which are able to accurately learn the dense 3D semantics of the scene online while simultaneously learning geometry. However, this work does not integrate semantic with other features of the environment, such as geometry and appearance. Therefore, DNS-SLAM [ 14 ] integrates multi-view geometry constraints with image-based feature extraction to improve appearance details and to output color, density, and semantic class information. SNI-SLAM [ 15 ] introduce cross-attention based feature fusion to incorporate semantic, appearance, and geometry features, thus improving the accuracy of mapping, tracking, and semantic segmentaion. Although these NeRF-based SLAM schemes achieve high-quality reconstruction effects, they suffer from poor scalability, low efficiency and poor real-time performance due to NeRF. II-B 3D GS-based SLAM The emergence of 3D GS have led to significant advancements in both general and semantic SLAM systems. [ 17 , 18 , 19 , 20 ] pioneered the introduction of 3D GS technology into SLAM systems, which are all committed to continuously expanding and optimizing gaussian

Read the complete text