Gaussian Splatting for Efficient Satellite Image Photogrammetry

Luca Savant Aira, Gabriele Facciolo, Thibaud Ehret https://mezzelfo.github.io/EOGS/ · 2024

Read Gaussian Splatting for Efficient Satellite Image Photogrammetry by Luca Savant Aira, Gabriele Facciolo, Thibaud Ehret https://mezzelfo.github.io/EOGS/ (2024) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2412-13047.txt.

Introduction

Luca Savant Aira, Gabriele Facciolo, and Thibaud Ehret adapt in a 2024 paper Gaussian splatting for satellite image photogrammetry. Splatting builds 3D scenes fast, far faster than the neural field methods before it. Earth observation versions of those older methods can take a day to fit.

Their frame keeps splatting's speed while adding remote sensing fixes from prior work. Light calibration and shadow modeling carry over from a leading neural baseline. New parts add sparsity, view consistency, and opacity controls tuned for orbit views. The result hits top accuracy in minutes rather than a day of fitting. That speed opens rapid mapping from fresh satellite passes for users.

Catalog record

CREATORLuca Savant Aira, Gabriele Facciolo, Thibaud Ehret https://mezzelfo.github.io/EOGS/
YEAR2024
COLLECTIONPAPERS
FORMATTXT
FILE SIZE40 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2412-13047

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Gaussian Splatting for Efficient Satellite Image Photogrammetry Luca Savant Aira, Gabriele Facciolo, Thibaud Ehret https://mezzelfo.github.io/EOGS/ Abstract Recently, Gaussian splatting has emerged as a strong alternative to NeRF, demonstrating impressive 3D modeling capabilities while requiring only a fraction of the training and rendering time. In this paper, we show how the standard Gaussian splatting framework can be adapted for remote sensing, retaining its high efficiency. This enables us to achieve state-of-the-art performance in just a few minutes, compared to the day-long optimization required by the best-performing NeRF-based Earth observation methods. The proposed framework incorporates remote-sensing improvements from EO-NeRF, such as radiometric correction and shadow modeling, while introducing novel components, including sparsity, view consistency, and opacity regularizations. EOGS altitude EO-NeRF altitude Pansharpened image Ground-truth altitude Figure 1 : Using a limited number of satellite images of a given scene, the proposed EOGS method estimates the appearance and geometry of the scene. It achieves the same level of detail as EO-NeRF Marí et al. 2023 , such as the group of fans or the thin structures on top of the tall building on the left. However, EOGS requires only a few minutes of optimization, compared to the day-long training time required by EO-NeRF Marí et al. 2023 . 1 Introduction Since the mid-20th century, the number of active satellites in orbit has increased exponentially. Today, hundreds of satellites are dedicated to Earth observation, and this number is expected to continue growing Clauson et al. 2024 . These satellites frequently acquire optical images of the same areas at regular intervals. As a result, the availability of these datasets will continue to grow in the near future, requiring efficient algorithms to handle this expansion. One common task for these image datasets is photogrammetry Hartley and Mundy 1993 ; Lebègue et al. 2020 ; eoPortal 2024 , which aims to recover the 3D geometry ( e.g . Digital Surface Model - DSM), and the appearance ( e.g . an albedo map) of the Earth’s surface for this area using only the available 2D satellite images. Specifically, we focus on performing digital surface modeling from remote sensing images. Historically, binocular stereovision and tri-stereo methods have been used for this purpose. However, these methods rely on image acquisitions being nearly simultaneous and with specific relative positions, which is often impractical, with limited acquisition opportunities, and/or costly. Therefore, our goal is to develop a method capable of handling multi-date images captured from arbitrary satellite positions. More recently, multi-view stereo methods tailored for novel-view synthesis (NVS) have been used to solve this task as they naturally handle diverse camera positions. Indeed, as the NVS task also requires understanding the geometry of the scene, these methods can be used to recover a digital elevation model. These methods are usually based on the concept of radiance field, i.e . a representation of the scene modeling the radiance emitted by each point, of which NeRF Mildenhall et al. 2021 is the main representative. Among those, EO-NeRF Marí et al. 2023 has established itself as a state-of-the-art approach for digital surface modeling thanks to an improved shadow model. However, because it is NeRF-based, it also inherits NeRF’s computational slowness. In recent years, 3D Gaussian Splatting (3DGS) Kerbl et al. 2023 has been proposed as an alternative to NeRF. This method offers much faster training and rendering while reaching comparable reconstruction accuracy. In this work, we introduce EOGS, the Earth-observation Gaussian splatting, the first method for digital elevation modeling based on 3DGS. EOGS achieves accuracy comparable to previous state-of-the-art approaches while being approximately 300\times faster. Keys to the success of EOGS are the following contributions, all of which are compatible with the original 3DGS framework’s efficiency: • Approximating locally the pushbroom satellite sensors as affine cameras. • Introducing a shadow-mapping-based pipeline for rendering the shadows in a physically accurate manner. • Adding three new regularization terms that promote sparsity in the Gaussians opacities, view consistency, and completely opaque objects. This reduces the training time and improves the quality of the results. 2 Related Work 2.1 Stereovision for Earth Observation Stereovision is at the heart of many tools for 3D estimation from series of satellite images. Examples of such pipelines are Ames stereo pipeline Beyer et al. 2018 , MicMac Rupnik et al. 2017 , CARS Michel et al. 2020 , S2P de Franchis et al. 2014 , or CATENA Krauß et al. 2013 . Traditionally, these multi-view stereo methods are applied to well-chosen (either manually or automatically) image pairs. Since they process each pair independently (for example in the dense stereo matching step), a crucial step is the fusion of all the generated pairwise 3D models into a single one. The recent trend has been to replace classic dense matching methods, such as semi-global matching (SGM) Hirschmuller 2007 or more global matching (MGM) Facciolo et al. 2015 , with deep learning based methods such as PSM Chang and Chen 2018 , HSM Yang et al. 2019 or GA-Net Zhang et al. 2019 . A review of these methods and a comparative study for satellite images is performed in Marí et al. 2022a . 2.2 NeRF for Earth Observation Recently, Mildenhall et al . Mildenhall et al. 2021 have shown that it is possible to learn a volumetric model of a scene, called neural radiance fields (NeRF), using differentiable inverse rendering. Given a sparse set of views of the scene, NeRF learns in a self-supervised manner by maximizing the photoconsistency across the predicted renderings corresponding to the available viewpoints. After convergence, the volumetric model can then be used to render realistic novel views of the scene. In practice, this volumetric model is represented by an MLP that predicts, for each position p of the space, the local density of the scene \sigma(\textbf{p}) as well as its appearance ( i.e . color) c(\textbf{p}) . The rendering is performed using an approximation of the volumetric integral I from optical physics estimated using ray casting, I(\boldsymbol{\mathrm{o}},\boldsymbol{\mathrm{d}})=\int_{t_{n}}^{t_{f}}e^{-\int_{t_{n}}^{t}\sigma\left(\boldsymbol{\mathrm{r}}(s)\right)ds}c\left(\boldsymbol{\mathrm{r}}(t),\boldsymbol{\mathrm{d}}\right)\sigma\left(\boldsymbol{\mathrm{r}}(t)\right)dt (1) with \boldsymbol{\mathrm{o}} the camera center from which the ray r originates and \boldsymbol{\mathrm{d}} unit direction. NeRF-based methods were then extended to the remote sensing case, and in particular to perform multi-view and multi-date satellite photogrammetry, namely S-NeRF Derksen and Izzo 2021 , Sat-NeRF Marí et al. 2022b , and EO-NeRF Marí et al. 2023 . S-NeRF Derksen and Izzo 2021 exploits the solar direction, information typically available in the metadata of each observation or that can be easily retrieved knowing the location of the scene as well as the acquisition hour and date, to predict the direct sun light reaching each point in the scene. This is done by adding the solar direction as an input to the MLP and predicting the amount of sun light reaching a point as a new output. In this way, the shadows cast by buildings can be learned by the MLP and generated accordingly during the novel-view rendering step. Sat-NeRF Marí et al. 2022b extends S-NeRF by modeling the transient parts of the scenes ( e.g . cars, construction sites, or foliage) as done in NeRF-in-the-wild Martin-Brualla et al. 2021 and improves the camera representation of S-NeRF, from pinhole to RPC Tao and Hu 2001 ; Baltsavias and Stallmann 1992 . EO-NeRF Marí et al. 2023

Read the complete text