Dense Scene Reconstruction from Light-Field Images Affected by Rolling Shutter
Hermes McGriff, Renato Martins, Nicolas Andreff, Cédric Demonceaux · 2024
Read Dense Scene Reconstruction from Light-Field Images Affected by Rolling Shutter by Hermes McGriff, Renato Martins, Nicolas Andreff, Cédric Demonceaux (2024) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2412-03518.txt.
Introduction
Hermes McGriff, Renato Martins, Nicolas Andreff, and Cedric Demonceaux present a computer vision method from 2024. Light-field cameras record a scene from many slight angles at once, which lets software work out depth. But when the camera sensor reads the image line by line, moving shots warp, an effect called rolling shutter.
Their method fixes the warp in two stages. First it builds a rough three-dimensional shape of the scene from a few of the views, using small soft blobs to stand for points. Then it works out the camera motion that bent the shape and undoes the bend. Tests on varied scenes and motions show the fix works well. Since no good test set existed, the team also built and released a synthetic one, along with code and trained models.
Catalog record
| CREATOR | Hermes McGriff, Renato Martins, Nicolas Andreff, Cédric Demonceaux |
|---|---|
| YEAR | 2024 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 39 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2412-03518
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Dense Scene Reconstruction from Light-Field Images Affected by Rolling Shutter Hermes McGriff, Renato Martins, Nicolas Andreff, Cédric Demonceaux Abstract This paper presents a dense depth estimation approach from light-field (LF) images that is able to compensate for strong rolling shutter (RS) effects. Our method estimates RS compensated views and dense RS compensated disparity maps. We present a two-stage method based on a 2D Gaussians Splatting that allows for a “render and compare” strategy with a point cloud formulation. In the first stage, a subset of sub-aperture images is used to estimate an RS agnostic 3D shape that is related to the scene target shape “up to a motion”. In the second stage, the deformation of the 3D shape is computed by estimating an admissible camera motion. We demonstrate the effectiveness and advantages of this approach through several experiments conducted for different scenes and types of motions. Due to lack of suitable datasets for evaluation, we also present a new carefully designed synthetic dataset of RS LF images. The source code, trained models and dataset will be made publicly available at: https://github.com/ICB-Vision-AI/DenseRSLF . 1 Introduction Figure 1 : Dense depth and motion compensation example from a light-field image affected by RS distortions. The RS deformed central sub-aperture view (used as input) is shown in the top left . The top right view shows the corresponding view with no RS effect. Our method results are shown in the second row, with the rendered compensated appearance model ( bottom left ) and disparity ( bottom right ) generated from the set of learned 2D Gaussians. Light-field cameras ( a.k.a. plenoptic cameras) are capable, thanks to their optical system, of recovering a 4D light field (LF) from a single capture. LFs have gained a strong popularity in the vision and graphics community notably since the apparition of neural radiance fields (NeRFs, e.g., [ 20 , 29 , 3 ] ) and more recently 3D Gaussian Splatting [ 13 ] . In the LF literature, images are generally considered to be acquired from a global shutter sensor, i.e . , under the assumption that all pixels are measured at the same time instant. But most consumer available cameras are equipped with rolling shutter (RS) sensors. Figure 2 : Overview of the main components of our two-stage method . In the first stage, we estimate a 2D Gaussian Splatting representation from a sub-set of SAIs. In the second stage, using the ability to render and compare thanks to the 2D Gaussian Splatting representation, we estimate the motion parameters in order to recover an undistorted representation of the scene. The particularity of RS cameras is that the acquisition of pixels is sequential (line by line) and, in the presence of either a camera motion or dynamic objects during the acquisition, the obtained image appears deformed by what is called the RS effect. It prevents the disambiguation of the impact of motion and shape in the RS imaging process. Ait-Aider [ 1 ] showed that we could leverage these deformations for motion estimation if both object shape and 2D-3D correspondences are known. More recently, we have shown in [ 19 ] that in the case of an RS light field, it is possible to estimate jointly the shape and the motion without any prior knowledge of the scene. However, this approach still requires 2D point correspondences between subaperture views of the LF. This constraint limits the density of object points and generates sparse reconstructions of distinctive object regions such as corners. In this paper, we propose a new method that is capable of producing dense reconstructions without the need of priors neither in the shape of the scene nor in the velocity profile. Different than most RS correcting techniques for monocular setups, we only require one capture to estimate both depth and motion and do not rely on multiple acquisitions [ 16 , 15 , 11 ] . Our method explores the potential offered by a Gaussian Splatting scene representation adapted to LF images to avoid performing correspondences. The designed approach presents different changes compared to existing Gaussian Splatting techniques [ 14 ] such as the use of 2D Gaussians, which are motivated by the specificity of the Light Fields acquired with a plenoptic camera. The method presented in this paper is a two-stage technique to obtain a dense representation of the scene, in terms of depth and appearance, jointly with motion parameters in order to compensate for the RS effect. These motion parameters are able to model the 3D deformation from the “3D RS effect” and the motion of the camera. Our method outputs dense disparity maps and a compensated RS corrected light field as shown in Fig. 1 . Different experiments show the ability of this method to obtain dense 3D representations of the scene, even in the case of important RS effect distortions. The main technical contributions of this paper are as follows: • We present a new 2D Gaussian Splatting representation carefully designed to leverage geometric properties of LF images. This dense appearance and geometric representation is exploited in a differentiable rendering of the observed scene. • An optimization-based approach is then designed to account for eventual rolling-shutter effects. We estimate motion parameters to correct the geometry, with a simple modelling of the deformation in the 3D space induced by RS in the light-field. • Due to the lack of publicly existing LF datasets affected by rolling shutter, we present a new light-field dataset with rolling shutter effects. This dataset is designed to allow the evaluation on scenes with highly textured surfaces, while containing different levels of geometric details. We will publicly release the source code, trained models, dataset and test samples following the review process. 2 Related work Leveraging the capabilities of rolling-shutter and light-field cameras for scene analysis has generated growing interest, although most current methods tend to utilize each modality independently. Ait-Aider et al . [ 1 ] seminal work was the first to demonstrate the potential of leveraging the RS effect to estimate object motion, but they assumed that both object 3D shape and 2D-3D correspondences where known in order to estimate the motion. Following works like Ait-Aider et al . [ 2 ] , Saurer et al . [ 23 ] and Lao et al . [ 15 ] explored the duality of deformation and RS effects, particularly in the context of stereo vision. Recently, we [ 19 ] have proposed an optimization-based method that estimates the motion and scene structure for LF cameras affected by RS. Although that approach does not require either a prior shape of the scene or 2D-3D correspondences, it still required 2D-2D image correspondences between sub-aperture images (SAI). Consequently, the obtained 3D models are sparse, in contrast to the proposed method of this paper. Recently, 3D Gaussian Splatting [ 14 ] has been introduced with promising results for efficient differentiable rendering. Various extensions and applications have been proposed beyond graphics rendering due to its properties [ 12 , 18 ] notably by allowing the increase of the observed scene or varying point densities (which were challenges with neural fields such as NeRFs). In this context, Seisakri et al . [ 24 ] proposed a