ProbeSDF: Light Field Probes For Neural Surface Reconstruction

Briac Toussaint [email protected], Diego Thomas [email protected], Jean-Sébastien Franco [email protected] · 2024

Read ProbeSDF: Light Field Probes For Neural Surface Reconstruction by Briac Toussaint [email protected], Diego Thomas [email protected], Jean-Sébastien Franco [email protected] (2024) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2412-10084.txt.

Introduction

Toussaint, Thomas, and Franco present in 2024 a faster way to rebuild three dimensional surfaces from many views using neural rendering. Methods based on signed distance fields lead the field, but their appearance models are heavy. The team reformulates that model in a simpler, physically inspired way that splits how light varies by angle and by place, storing each part in its own grid at its own resolution.

Only four values per cell and a tiny network call inside one fused kernel are needed. Results improve on both surface and image quality measures, with much faster training and real time rendering. Tests span generic objects and human subjects across four hard data sets.

Catalog record

CREATORBriac Toussaint [email protected], Diego Thomas [email protected], Jean-Sébastien Franco [email protected]
YEAR2024
COLLECTIONPAPERS
FORMATTXT
FILE SIZE54 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2412-10084

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

ProbeSDF: Light Field Probes For Neural Surface Reconstruction Briac Toussaint [email protected], Diego Thomas [email protected], Jean-Sébastien Franco [email protected] Abstract SDF-based differential rendering frameworks have achieved state-of-the-art multiview 3D shape reconstruction. In this work, we re-examine this family of approaches by minimally reformulating its core appearance model in a way that simultaneously yields faster computation and increased performance. To this goal, we exhibit a physically-inspired minimal radiance parametrization decoupling angular and spatial contributions, by encoding them with a small number of features stored in two respective volumetric grids of different resolutions. Requiring as little as four parameters per voxel, and a tiny MLP call inside a single fully fused kernel, our approach allows to enhance performance with both surface and image (PSNR) metrics, while providing a significant training speedup and real-time rendering. We show this performance to be consistently achieved on real data over two widely different and popular application fields, generic object and human subject shape reconstruction, using four representative and challenging datasets. 1 1 1 Source code is available at https://gitlab.inria.fr/projects-morpheo/ProbeSDF Figure 1 : We design a new appearance model for neural surface approaches, which combines high resolution spatial features and lower resolution angular features for improved reconstruction quality, training and inference speed. We plot the chamfer distance as a function of training speed for several baselines on MVMannequins (top) and DTU (bottom). 1 Introduction Neural radiance fields have established a new milestone for the task of novel view synthesis and have led to a plethora of variants of broad applicability. We here take particular interest in SDF surface-based radiance fields, which have proven to be a highly performing variant for the tasks of 3D modeling from images Wang et al. 2021 ; Wu et al. 2022 , with wide reaching applications ranging from object and performance capture, to immersive and 3D content production. At the heart of such methods usually lies a parametrization of the light field, in the form of a 5-dimensional function. This function maps the spatial coordinates of a point in space and the observed direction to a color and a signed distance to the reconstructed surface. On one hand, the mapping can be global and use a complex scene-wide MLP as implicit 5D decoder, as is the case for most seminal approaches Mildenhall et al. 2020 ; Wang et al. 2021 . On the other hand, recent popular variants that target higher speed and memory efficiency use explicit representations. Global representations are based on a grid encoding Wu et al. 2022 ; Fridovich-Keil et al. 2022 ; Wang et al. 2023 where interpolated spatial and angular features are run through a global shallow MLP. Local representations use the sparse 3D Gaussian Splatting Kerbl et al. 2023 sample-based encoding approach, where each separate Gaussian comes with its own opacity and adjoining set of angular color features. While grids or local approaches use a single deep MLP to densely encode the complete light field function, they almost always co-locate storage and decoding of color and angular features. But is this really a necessary or even desirable feature of any light field encoding? We question these assertions by postulating that, contrary to the spatial components of the light field that mainly encode intrinsic surface texture, the angular components encode phenomena that are predominantly extrinsic to the surface, i.e. lighting and environment, and as such can be encoded at a lower spatial resolution as they are locally almost invariant to parallax. This hypothesis is in fact exploited by the rendering community, where angular and environment components contributing to the radiance equation used to compute rendered colors, are typically precomputed at sparse 3D locations coined light field probes Pettineo , and interpolated in-between. Leveraging this intuition, our approach borrows the spherical harmonic parametrization popular with explicit methods e.g . Kerbl et al. 2023 ; Fridovich-Keil et al. 2022 to represent angular components, but instead decorrelates high density spatial variation from lower density angular variation using a mid-resolution probe grid of angular features. Each of the components encode abstract features that are fed to a tiny MLP, which acts as a minimal BRDF decoder of spatial and angular varying components, while we explicitly store SDF values. This scheme advantageously replaces the core appearance and decoding model of SDF approaches Wang et al. 2021 ; Wu et al. 2022 ; Wang et al. 2023 ; Li et al. 2023 to predict ray colors. We are careful to also include power terms of the cosine angle between the view direction and the estimated surface normal, allowing our lightweight model to effectively account for increased surface reflectivity at grazing angles. This yields an unprecedented performance combination, superseding both 3D and image metric performances with four popular benchmarks for general object reconstruction Yao et al. 2020 ; Jensen et al. 2014 and human reconstruction Toussaint et al. 2024 ; Işık et al. 2023 while simultaneously achieving a significant training and rendering performance boost. In summary, our contributions are: • An explicit embedding of the angular dependency that is simultaneously local, smooth and efficient. • A simpler color prediction MLP that is agnostic to the surface position and orientation. 2 Related Works In this work we leverage disentangled spatial and angular parameterization of radiance towards high quality appearance and surface modeling. Here we consequently review closely related works on surface-based 3D scene representations and existing parametrizations of color. Neural Surface Reconstruction. Through the use of implicit surface representations, neural implicit fields have demonstrated increasing capabilities to generate high-quality 3D models Niemeyer et al. 2020 ; Yariv et al. 2020 ; Yariv et al. 2021 . In NeuS Wang et al. 2021 , Wang et al. proposed to parametrize the surface with a signed distance function that is modeled with a multi layer perceptron (MLP). The approach opened new frontiers in the generation of accurate 3D scenes from images but at the cost of prohibitive training times (several hours). Follow-up works have thus focused on boosting both accuracy and computation time by employing explicit grids of features and smaller MLPs. Notably, Wu et al. proposed in Voxurf Wu et al. 2022 to use a dense grid of voxels, and Neus2 was also proposed using hash grids Wang et al. 2023 . Recently, Millimetric humans Toussaint et al. 2024 have demonstrated remarkably fast and accurate reconstructions in the case of human datasets by using sparse and dynamic grids. Other reparametrizations target performance improvement, e.g . point-based with Fast Dipole Sums Chen et al. 2024b and PGSR Chen et al. 2024a both leveraging multi-view stereo priors, or by seeking mutual advantage of a hybrid volume/mesh representation Huang et al. 2024b . All the methods discussed above rely on co-located encodings of position and view direction to parametrize the appearance. As a consequence, these approaches either fail to model local radiance effects like inter-reflections, or require larger MLPs and training times, and may thus benefit from our refactored appearance model as drop-in replacement. Angular parametrization Deep MLPs have become ubiquitous to model view-dependent colors, taking as input the view direction Mildenhall et al. 2020 , the view and normal Yariv et al. 2020 ; Wang et al. 2021 ; Wu et al. 2022 ; Wang et al. 2023 or the

Read the complete text