High-Fidelity and Generalizable Neural Surface Reconstruction with Sparse Feature Volumes
Aoxiang Fan, Corentin Dumery, Nicolas Talabot, Hieu Le, Pascal Fua · 2025
Read High-Fidelity and Generalizable Neural Surface Reconstruction with Sparse Feature Volumes by Aoxiang Fan, Corentin Dumery, Nicolas Talabot, Hieu Le, Pascal Fua (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2507-05952.txt.
Introduction
Fan and co-authors present SVRecon, a 2025 method for rebuilding 3D surfaces from photos without tuning the system to each scene. Neural methods that render scenes usually optimize per scene, which is slow. General methods skip that step but lean on dense feature grids that eat memory and cap detail. SVRecon first predicts which voxels actually contain surface, then works only in those sparse regions.
A second stage renders from the sparse volume at resolutions up to 512 cubed on ordinary 32-gigabyte hardware. Custom sparse routines handle ray sampling, feature pooling, and queries. Across data sets, the surfaces came out smoother and more exact than prior general methods, above all with few input views. The authors frame learned sparsity as the key that unlocks detail.
Catalog record
| CREATOR | Aoxiang Fan, Corentin Dumery, Nicolas Talabot, Hieu Le, Pascal Fua |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 40 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2507-05952
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
High-Fidelity and Generalizable Neural Surface Reconstruction with Sparse Feature Volumes Aoxiang Fan, Corentin Dumery, Nicolas Talabot, Hieu Le, Pascal Fua Abstract Generalizable neural surface reconstruction has become a compelling technique to reconstruct from few images without per-scene optimization, where dense 3D feature volume has proven effective as a global representation of scenes. However, the dense representation does not scale well to increasing voxel resolutions, severely limiting the reconstruction quality. We thus present a sparse representation method, that maximizes memory efficiency and enables significantly higher resolution reconstructions on standard hardware. We implement this through a two-stage approach: First training a network to predict voxel occupancies from posed images and associated depth maps, then computing features and performing volume rendering only in voxels with sufficiently high occupancy estimates. To support this sparse representation, we developed custom algorithms for efficient sampling, feature aggregation, and querying from sparse volumes—overcoming the dense-volume assumptions inherent in existing works. Experiments on public datasets demonstrate that our approach reduces storage requirements by more than 50× without performance degradation, enabling reconstructions at 512 3 512^{3} resolution compared to the typical 128 3 128^{3} on similar hardware, and achieving superior reconstruction accuracy over current state-of-the-art methods. 1 Introduction Multi-view reconstruction has witnessed a performance boost with the emergence of neural implicit representation techniques, starting with the seminal work Neural Radiance Fields (NeRF) [ 27 ] . NeuS [ 37 ] is a popular representative that, along with similar methods [ 20 , 43 , 44 ] , excels at geometry recovery. These algorithms take posed images as input, and optimize a Signed Distance Function (SDF) based neural representation by constructing a color loss with input images through volume rendering. However some limitations exist, including the lack of cross-scene generalization abilities and the dependence on dense input views. These limitations remain in 3D Gaussian Splatting [ 14 ] based methods for geometry reconstruction [ 10 , 47 ] . Figure 1 : Two-Stage Approach . From multiple images, we predict occupancy field that is then supersampled to build a sparse scene representation. This lets us reconstruct high-fidelity surfaces by operating at previously unattainable 512^{3} resolution. Generalizable approaches to Neural Surface Reconstruction (GNSRs) [ 21 , 30 , 39 , 25 , 29 , 45 ] aim to remove these restrictions. By training on 3D ground truth data, the model learns to construct meaningful 3D scene representations, typically in the form of dense 3D feature volumes to model global information. Sampling can then be performed at arbitrary locations in 3D space through interpolation, enabling volume rendering networks to be trained to perform per-ray feature aggregations and infer scene colors and geometries. Owing to the learnable feature representations, these methods possess cross-scene generalization abilities while requiring only a few images as input. However, these methods have yet to achieve their full potential in practice. This stems from the poor scalability of their dense representations, which drastically constrain the operational resolution as shown in Tab. 1 . This adversely impacts reconstruction quality and detail preservation. Sparse scene representation seems like a natural answer to this problem. Since surfaces are mathematically of measure zero, they occupy only a small fraction of voxels in a discretized 3D space, making sparsity an ideal choice for high-resolution, memory-efficient reconstruction. However, this raises the question of how to construct a sparse representation efficiently for a new scene using only a few available images. For example, SparseNeuS [ 25 ] uses a coarse-to-fine scheme but has to query on densified feature volumes. Octree-like structures, such as those used in Neural Sparse Voxel Fields [ 24 ] , require per-scene training of a voxel-bounded implicit function, which is not feasible in the GNSR setting. Methods 128^{3} 192^{3} 256^{3} 512^{3} SparseNeuS [ 25 ] ✓ ✓ ✗ ✗ VolRecon [ 30 ] ✓ ✗ ✗ ✗ ReTR [ 21 ] ✓ ✗ ✗ ✗ Ours ( \mathit{SVR}econ ) ✓ ✓ ✓ ✓ Table 1 : Resolutions handled by GNSR algorithms at training time . Crosses indicate that the memory requirements were too large on a 32GB NVIDIA Tesla V100 GPU, with the same setting of batch size 1 and 1024 sampled rays for all methods. In this paper, we propose a novel and effective way to bring sparsity into GNSRs. Instead of relying on complex hierarchical structures like octrees, we use a simpler nested two-layer structure that enables GNSR to operate at significantly higher resolution than previous methods while maintaining generalization across scenes. The two layers are as follows. 1. Coarse Occupancy Field: As shown in the upper right corner of Fig. 1 , we can label voxels in a coarse grid as containing a surface element or not. This can be done by trianing a network given only a few posed images of the scene, making our method fully generalizable to unseen scenes. 2. Sparse High-Resolution Feature Volume: We can then construct a high-resolution feature volume, but only in the occupied regions. This allows for fine-grained neural surface reconstruction while keeping memory and computational costs at a minimum. To effectively exploit this representation, we had to reformulate the algorithms used by existing GNSRs because they assume a dense feature volume. Thus, we developed algorithms specifically designed to handle sparse feature volumes to perform ray sampling, feature aggregation, and query operations. These technical innovations are key to realizing the theoretical memory advantages of our approach while maintaining computational efficiency. Testing on public datasets demonstrates that our sparse representation method is highly effective, reducing the storage requirement by a factor of more than 50 without sacrificing reconstruction quality. This has enabled us to operate at an unprecedented resolution of 512^{3} on consumer-grade GPUs with 32GB of VRAM, thus improving the final reconstruction accuracy beyond the state-of-the art. In short, we propose a simple, practical, yet powerful implementation of sparsity for GNSR, making high-resolution scene reconstruction at 512^{3} achievable. Figure 2 : \mathit{SVR}econ pipeline. First, we build a low-resolution dense 3D feature volume from multi-view features and predict 3D occupancies. Second, we construct high-resolution sparse feature volumes where the predicted occupancy warrants it. Finally, volume rendering Transformers are used to aggregate per-ray sampled features and infer color and depth, enabling scene reconstruction. 2 Related Works Multi-View Stereo. Multi-view stereo (MVS) [ 6 , 9 , 34 , 33 , 7 , 19 , 17 , 18 ] has long been known to be effective for 3D reconstruction. Modern multi-view stereo methods can be categorized into volumetric ones [ 11 , 12 , 13 ] and depth-map ones [ 40 , 41 , 8 , 36 , 5 , 2 ] . Volumetric methods are directly related to our method, as they also operate on a volumetric 3D space. However, our scene representation and volume rendering formulation makes it possible to recover finer details. Depth map-based methods deliver excellent performance, along with low storage overhead. As proposed in the MVSNet approach [ 40 ] , these methods construct 3D cost volumes by correlating source image features with the reference image features to infer a depth map for each reference view. A filtering and fusion method is then used to produce a 3D point cloud for the whole scene. These methods rely on view-dependent frustums as