SERES: Semantic-Aware Neural Reconstruction from Sparse Views
2025
Read SERES: Semantic-Aware Neural Reconstruction from Sparse Views (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2508-18314.txt.
Introduction
Rebuilding a three-dimensional object from a handful of photos is plagued by ambiguity: sparse views leave the light and shape underdetermined. Bo Xu and co-authors propose SERES, which enriches the usual neural scene model with meaning. Alongside the signed distance and radiance fields, the system learns patch-level semantic labels, so matching patches across views must agree on what they depict, not just how they shine.
A further regulariser uses masks of simple geometric shapes to cut down shape ambiguity. On a standard benchmark the method cut reconstruction error sharply against two sparse-view rivals, by forty-four and twenty percent. Bolted onto dense-view systems as a plugin, it reduced their errors by roughly two thirds as well. Semantic sense, the results suggest, steadies geometry when views are few.
Catalog record
| YEAR | 2025 |
|---|---|
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 44 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2508-18314
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
SERES: Semantic-Aware Neural Reconstruction from Sparse Views Thanks: denotes equal contribution to this work. Thanks: denotes the corresponding authors. Bo Xu, Yuchao Wang and Xinyi Le are with the Department of Automation, Shanghai Jiao Tong University, Shanghai, China (email: [email protected]). Yuhu Guo and Charlie C.L. Wang are with Department of Mechanical and Aerospace Engineering, The University of Manchester, Manchester, UK (email: [email protected]). Wenting Wang and Yeung Yam are with Department of Mechanical and Automation Engineering, The Chinese University of Hong Kong, Hong Kong SAR. Abstract We propose a semantic-aware neural reconstruction method to generate 3D high-fidelity models from sparse images. To tackle the challenge of severe radiance ambiguity caused by mismatched features in sparse input, we enrich neural implicit representations by adding patch-based semantic logits that are optimized together with the signed distance field and the radiance field. A novel regularization based on the geometric primitive masks is introduced to mitigate shape ambiguity. The performance of our approach has been verified in experimental evaluation. The average chamfer distances of our reconstruction on the DTU dataset can be reduced by 44% for SparseNeuS and 20% for VolRecon. When working as a plugin for those dense reconstruction baselines such as NeuS and Neuralangelo, the average error on the DTU dataset can be reduced by 69% and 68% respectively. I Introduction Neural implicit representation-based 3D reconstruction methods [ 1 , 2 ] have gained significant popularity in recent years due to reconstruct geometric structures with remarkable fidelity. However, the construction of accurate and intricate geometry poses a challenge for these methods when the input comprises only a sparse set of images. The challenge is mainly caused by the limited coverage of views, which prevents the effective matching on many pixels in the input views. As a result, this leads to shape-ambiguity [ 3 , 4 , 5 ] in the radiance field to be learned. To tackle this challenge, scene priors are usually embedded into radiance field learning through a specially designed framework that integrates neural networks [ 6 ] [ 7 ] with the capability to acquire prior scene knowledge or utilize special 3D geometric data structures [ 8 ] . An alternative paradigm is based on incorporating external geometric cues [ 9 ] (e.g., surface normals [ 1 ] , depth maps [ 10 , 11 ] , and albedo information [ 12 ] ) as supplementary constraints to enhance the precision of scene geometry. However, both paradigms involve significant costs – i.e., introducing additional neural networks during training requires substantial computational overhead, and incorporating geometric cues necessitates acquiring extra data. An inherently pertinent question arises: How can we add a low-cost scene prior into neural implicit representations effectively? Fig. 1: We argue that the quality of 3D models from sparse views can be significantly improved by incorporating semantic awareness into the neural reconstruction pipeline. When reconstructing the statue of Vakhtang I Gorgasali from nine views (five are illustrated in the figure), our SERES approach can generate more geometric details than the state-of-the-art VolRecon [ 11 ] . We report novel view synthesis metrics for 3, 6, and 9 viewpoints input (center random selection), shown in the bottom left. Solid bars refer to the left y-axis, and dashed bars to the right y-axis. Our idea was inspired by the engineering drawing practice, where the engineers can rely on only three views to accurately reconstruct the 3D geometry of most objects. The key concept of this involves dividing the object into common semantic patches in various views, thereby acquiring a prior understanding of the scene. It can be inferred that incorporating a prior based on semantic patch matching would be a beneficial choice for 3D reconstruction from sparse views. Based on this insight, this paper proposes a novel SE mantic-aware framework for RE construction from S pare views (SERES). Specifically, we enrich the neural field representation by semantic logits, the initial values of which can be obtained by a training-free segmentation model and a vision transformer. Reliable feature matching and therefore high-fidelity reconstruction can be achieved by optimizing these semantic logits together with the signed distance field and the radiance field. During optimization, the geometric primitive masks are also employed as regularization, which offers additional constraints to alleviate shape-ambiguity. As shown in Fig. 1 , our SERES can successfully reconstruct intricate sculptures using only nine viewpoints, while preserving the correct geometric structure and capturing fine details. In summary, we make the following contributions: • We present a novel semantic field which embed patch-based semantic priors into neural implicit representations. This prior provides semantic-aware feature matching for improving the fidelity of sparse view reconstruction. • We propose a point prompt-guided regularization technique, mitigating the shape ambiguity of reconstructed shape and improving the completeness of geometric structures. • This technique has shown its generality to benefit different neural implicit representation frameworks, serving as an effective plugin with acceptable computational overhead. The performance of our SERES has been tested in a variety of academic benchmarks and real-world scenarios. II Related Work II-A Sparse Multi-view Stereo Modeling 3D surfaces from sparse views is a difficult task, as accurately aligning infrequent regions with 3D space presents a challenge [ 13 ] . Previous attempts [ 6 , 10 ] aim to reduce the number of required viewpoints, but they do not extract geometries. Instead, they focus mainly on novel view synthesis [ 14 , 8 ] or estimating unknown camera positions [ 15 ] . Recent studies such as [ 7 , 9 , 11 ] have made positive strides in sparse view reconstruction. However, previous research has not been successful in creating high-quality and easy-to-use sparse view reconstructions. SC-NeuS [ 16 ] requires multi-view constraints to refine camera poses. MonoSDF [ 9 ] necessitates precise depth and normal information, while SparseNeuS [ 7 ] often fails to reconstruct complex shapes, leaving them with holes in the surface. VolRecon [ 11 ] can produce satisfactory results, but tends to smooth textures in detail-rich regions and generate noise around them. II-B Neural Implicit Surface Reconstruction NeRF [ 17 ] has revolutionized the field of 3D world understanding by introducing novel view synthesis. This technique shares similarities with reconstruction, yet originates from a distinct conceptual foundation. Then, NeuS [ 2 ] highlights the difficulty of depicting high-quality surfaces in learned implicit representations, attributed to a lack of surface constraints. Directly estimating signed distance function (SDF) from multi-view images [ 18 ] in neural implicit representations can lead to geometric biases in surface reconstruction [ 19 , 9 ] . To obtain precise SDF representation, recent efforts aim to incorporate additional geometric constraints [ 20 , 21 ] or cues [ 9 ] , including object masks [ 22 , 23 ] , normal priors [ 1 , 24 ] , depth information [ 25 ] , viewpoint selection [ 26 ] and photometric consistency [ 27 ] . Notably, previous techniques have disregarded the significance of semantic coherence across multiple view, especially in cases of sparse views where semantics considerably assist in recuperating geometry. II-C SAM for 3D understanding The Segment Anything Model (SAM) [ 28 ] is highly proficient in carrying out detailed image segmentation. However, it lacks