PlanarSplatting: Accurate Planar Surface Reconstruction in 3 Minutes
Bin Tan, Rui Yu, Yujun Shen, Nan Xue · 2024
Read PlanarSplatting: Accurate Planar Surface Reconstruction in 3 Minutes by Bin Tan, Rui Yu, Yujun Shen, Nan Xue (2024) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2412-03451.txt.
Introduction
Bin Tan, Rui Yu, Yujun Shen, and Nan Xue rebuild indoor rooms from photos in about three minutes, in a 2024 paper. Indoor scenes are dominated by flat surfaces, walls, floors, and tabletops. Rather than reconstructing every point, their method takes three-dimensional planes as the main building blocks.
The system splats those planes into depth and surface-direction maps and optimizes them directly against the input views. Working with plane pieces from the start removes the usual pipeline of detecting planes in images and matching them across views. Speed comes from the compact representation and custom graphics-chip code. On two large benchmark collections covering hundreds of rooms, the method was both much faster and geometrically more accurate than rivals. The authors expect it to serve as a fast curation tool for surface reconstruction data.
Catalog record
| CREATOR | Bin Tan, Rui Yu, Yujun Shen, Nan Xue |
|---|---|
| YEAR | 2024 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 32 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2412-03451
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
PlanarSplatting: Accurate Planar Surface Reconstruction in 3 Minutes Bin Tan, Rui Yu, Yujun Shen, Nan Xue Abstract This paper presents PlanarSplatting , an ultra-fast and accurate surface reconstruction approach for multiview indoor images. We take the 3D planes as the main objective due to their compactness and structural expressiveness in indoor scenes, and develop an explicit optimization framework that learns to fit the expected surface of indoor scenes by splatting the 3D planes into 2.5D depth and normal maps. As our PlanarSplatting operates directly on the 3D plane primitives, it eliminates the dependencies on 2D/3D plane detection and plane matching and tracking for planar surface reconstruction. Furthermore, the essential merits of plane-based representation plus CUDA-based implementation of planar splatting functions, PlanarSplatting reconstructs an indoor scene in 3 minutes while having significantly better geometric accuracy. Thanks to our ultra-fast reconstruction speed, the largest quantitative evaluation on the ScanNet and ScanNet++ datasets over hundreds of scenes clearly demonstrated the advantages of our method. We believe that our accurate and ultrafast planar surface reconstruction method will be applied in the structured data curation for surface reconstruction in the future. The code of our CUDA implementation will be publicly available. Project page can be found here . Figure 1 : We introduce PlanarSplatting , a fast and accurate optimization-based planar reconstruction method for indoor scenes. Top (Planar Reconstruction) : We show our planar reconstruction results on the ScanNetV2 [ 2 ] dataset achieved in 3 minutes. Compared to prior art PlanarRecon [ 35 ] and AirPlanes [ 33 ] , our PlanarSplatting reconstructs more complete and detailed 3D planes. Bottom (Novel View Synthesis) : We show that our PlanarSplatting can be seamlessly integrated with recent Gaussian Splatting methods (e.g., 3DGS [ 12 ] and 2DGS [ 10 ] ) to achieve improved rendering results in indoor scenes while requiring significantly less optimization time. 1 Introduction We humans are long-term immersed in structured indoor scenes, ranging from bedrooms to offices. This fact has ignited a wealth of studies for reconstructing the indoor environment with various 3D structures such as lines [ 8 , 18 , 37 ] , planes [ 17 , 40 , 35 , 31 , 11 , 15 ] and blocks [ 7 , 21 ] . Among them, the 3D plane is the most common representation because of its simplicity and completeness in describing physical surfaces, thus motivating us to study the problem of 3D reconstruction for indoor scenes using 3D planes, i.e . , planar 3D reconstruction. Planar 3D reconstruction has been extensively studied for years as a model fitting problem, in which a 3D scene geometry ( e.g . , point clouds, or meshes) was assumed to be known and the main goal is fitting the scene in a set of 3D planes [ 28 , 23 , 36 ] . Recently, the paradigm has been gradually simplified in image-based solutions in single-view and multiview 3D reconstruction, eliminating the acquisition of known 3D scene geometry. To make the problem trackable, existing methods were extensively based on the image-level characterization of 2D/3D planes. That is to say, those methods have to detect 3D planes for each input image, match or track planes in across viewpoints, and finally reconstruct and merge 3D planes as the compact 3D representation of indoor scenes. Recent PlanarRecon [ 35 ] attempted to address these problems end-to-end by learning a 3D volume from monocular videos. Then, 3D planes can be detected, tracked, and fused in a consistent 3D space. In fact, image-based planar 3D reconstruction mostly followed keypoint-based 3D reconstruction pipelines and treated planes as a special kind of visual features. However, because of the essential difference between the local point features and regional plane masks in image space, we argue, the existing approaches did not fully leverage the advantages of planar representations. Some evidences could be observed from the results of PlanarRecon [ 35 ] , which are usually very coarse and lose many details of the scene. In this paper, we are going to address the issues remained in image-based planar 3D reconstruction, aiming at obtaining a complete, structural, and compact indoor scene reconstruction from multi-view images. Our main idea is approximating the indoor scenes with a collection of solid 3D planar primitives from multi-view input images, directly optimizing them to have consistent 3D planes, but eliminating any suboptimal precomputing of plane primitives ( e.g . , plane masks). We introduce PlanarSplatting that explicitly optimizes rectangular plane primitives in 3D space by differentiably splatting them into 2.5D depth and normal maps. Thanks to our well-designed plane splatting function, PlanarSplatting effectively leverage monocular geometry cues from modern foundational models [ 39 , 9 , 3 ] for accurate plane optimization. As shown at the top of Figure 1 , the final high-quality planar surface can be reconstructed by simply merging similar 3D plane primitives without any plane annotations for supervision or matching/tracking operations. As our PlanarSplatting is directly designed on the 3D plane primitives and efficiently implemented with CUDA, it can be seamlessly integrated with recent Gaussian Splatting (GS) methods for high-quality indoor novel view synthesis (NVS). As shown at the bottom of Fig. 1 , benefiting from our fast and accurate scene reconstruction (within 3 minutes), GS-based methods can be well-initialized and optimized without densification, leading to better rendering results and significantly less training time. It demonstrates the strong potential of our PlanarSplatting to promote the unity of reconstruction and novel view synthesis for the representation of indoor scenes. In the experiments, our PlanarSplatting shows its powerful ability for accurate indoor planar surface reconstruction on two real-world indoor datasets including ScanNetV2 [ 2 ] and ScanNet++ [ 38 ] on hundreds of scenes. Furthermore, we show that with the combination of our PlanarSplatting and GS-based methods ( e.g . , 3DGS [ 12 ] and 2DGS [ 10 ] ), we can effectively improve the rendering quality with less training time and fewer points. 2 Related Work Indoor Planar Reconstruction. Reconstructing indoor scenes with 3D plane primitives has been studied for a long time [ 4 , 5 , 34 , 6 , 27 ] . Traditionally, it is usually achieved by deducing and fitting the plane geometry directly from 3D data ( e.g . , point clouds [ 23 , 29 , 22 , 28 ] and line clouds [ 14 ] ), or from 2D single-view images with strict scene assumptions ( e.g . , the Manhattan World constrain) [ 20 , 15 , 24 ] which seriously limited their application. In recent years, some learnable-based methods are proposed to formulate this problem as single-view 3D plane segmentation [ 16 , 17 , 40 , 30 , 26 , 41 ] and cross-view plane instance matching [ 11 , 1 , 31 ] . Although impressive results have been achieved, these works are hard to extend to multiple views. PlanarRecon [ 35 ] was the first end-to-end work proposed to deal with holistic indoor plane recovery by learning a plane-related 3D volume from posed RGB videos. Then, 3D planes can be extracted, tracked, and fused from the learned 3D volume incrementally. Most recently, AirPlanes [ 33 ] developed a two-step method that