Improving Novel view synthesis of 360 ∘ Scenes in Extremely Sparse Views by Jointly Training Hemisphere Sampled Synthetic Images

Guangan Chen, Anh Minh Truong, Hanhe Lin, Michiel Vlaminck, Wilfried Philips, Hiep Luong · 2025

Read Improving Novel view synthesis of 360 ∘ Scenes in Extremely Sparse Views by Jointly Training Hemisphere Sampled Synthetic Images by Guangan Chen, Anh Minh Truong, Hanhe Lin, Michiel Vlaminck, Wilfried Philips, Hiep Luong (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2505-19264.txt.

Introduction

Guangan Chen and co-authors tackle 360-degree view synthesis from very few photos in a 2025 paper. Creating new viewpoints of a full scene matters for virtual and augmented reality. With only a handful of inputs, standard pose estimation fails and models overfit the few views.

Their pipeline estimates poses with a modern reconstruction tool and builds a dense point cloud. It then samples many synthetic views from above the scene and trains on real and synthetic images together. A diffusion enhancer is retrained on their data to polish results. Wider coverage in training fights the sparse-view problem directly.

Catalog record

CREATORGuangan Chen, Anh Minh Truong, Hanhe Lin, Michiel Vlaminck, Wilfried Philips, Hiep Luong
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE23 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2505-19264

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Improving Novel view synthesis of 360 ∘ Scenes in Extremely Sparse Views by Jointly Training Hemisphere Sampled Synthetic Images Guangan Chen, Anh Minh Truong, Hanhe Lin, Michiel Vlaminck, Wilfried Philips, Hiep Luong Abstract Novel view synthesis in 360 ∘ scenes from extremely sparse input views is essential for applications like virtual reality and augmented reality. This paper presents a novel framework for novel view synthesis in extremely sparse-view cases. As typical structure-from-motion methods are unable to estimate camera poses in extremely sparse-view cases, we apply DUSt3R to estimate camera poses and generate a dense point cloud. Using the poses of estimated cameras, we densely sample additional views from the upper hemisphere space of the scenes, from which we render synthetic images together with the point cloud. Training 3D Gaussian Splatting model on a combination of reference images from sparse views and densely sampled synthetic images allows a larger scene coverage in 3D space, addressing the overfitting challenge due to the limited input in sparse-view cases. Retraining a diffusion-based image enhancement model on our created dataset, we further improve the quality of the point-cloud-rendered images by removing artifacts. We compare our framework with benchmark methods in cases of only four input views, demonstrating significant improvement in novel view synthesis under extremely sparse-view conditions for 360 ∘ scenes. The source code is available at https://github.com/angchen-dev/hemiSparseGS . † † address: 1 Image Processing and Interpretation (IPI), IMEC research group at Ghent University, Belgium 2 School of Science and Engineering, University of Dundee, United Kingdom Figure 1 : The framework of proposed method. Given a sparse set of views (reference images) with unknown camera poses of a scene, we first utilize an off-the-shelf multi-view stereo method to estimate the camera poses and generate a dense point cloud. Subsequently, we sample camera poses in the upper hemisphere space of the scene and render synthetic images using the sampled poses and the dense point cloud. Next, the rendered images are enhanced using a diffusion-based model. Finally, both the enhanced synthetic images and reference images are employed to train a 3DGS model. 1 Introduction Novel view synthesis (NVS) is a key challenge in computer vision, focusing on rendering images from previously unobserved viewpoints. Recent advancements, such as Neural Radiance Fields (NeRF) [ 12 ] and 3D Gaussian Splatting (3DGS) [ 6 ] , have demonstrated remarkable capabilities in generating novel views using dense captured images of a scene. However, acquiring hundreds to thousands of highly overlapping images of a scene is often time-consuming and impractical, especially for large-scale scenes that require reacquisition whenever changes occur [ 3 ] . Consequently, there is growing interest in developing efficient approaches to reconstruct 3D scenes from sparse views [ 4 , 5 , 22 , 24 , 21 , 25 , 13 ] . In the scenario of sparse views, NeRF and 3DGS often overfit due to limited input views, resulting in severe visual artifacts and a lack of coherent structures. To tackle this challenge, some existing methods proposed integrating diffusion-based approaches [ 15 ] into NVS methods because of their generative capabilities. These proposed approaches can be categorized into three main types. The first type involves using the knowledge of a pre-trained diffusion model and introduce a score distillation sampling (SDS) loss to guide training in the NVS pipeline, e.g., [ 14 , 24 ] . The performance of this type of methods, however, are still far from satisfactory in extremely sparse-view cases. The second type focuses on generating synthetic images of novel views by training 2D diffusion models on large-scale multi-view datasets. Those synthetic images and reference images are used for training 3D models jointly. For example, Wu et al. [ 21 ] utilized the Latent Diffusion Model (LDM) [ 15 ] trained on multiple large-scale datasets to generate images of unobserved views, serving as additional training data during the NeRF model training of scenes. Such methods are time-consuming and costly since they need to be trained on large-scale datasets from scratch. Additionally, the generative training images often contain content not originally present in the scenes due to the nature of generative models, resulting in unexpected elements in some synthesized novel views. The third type involves fine-tuning diffusion-based models to enhance the visual quality of NVS-rendered images, which are used as additional data for NVS model training. Paul et al. [ 13 ] created a dataset with 3DGS-rendered images and corresponding reference images to fine-tune the Instruct-Pix2Pix diffusion model [ 2 ] . During 3DGS training, the rendered images in the current iteration are enhanced by the fine-tuned model and used as training data for the next iteration. Yang et al. [ 25 ] adopted a diffusion-based Gaussian repair model to self-generate target images for unobserved views as additional training data for reconstructing extremely sparse-view 360^{\circ} object. Such approaches rely on accurate ground-truth camera poses, which are often difficult to estimate using typical structure-from-motion methods in extremely sparse-view scenarios. A few non-diffusion-based NVS methods are proposed to address the challenge of estimating camera poses in sparse-view scenarios. Fan et al. [ 4 ] proposed utilizing DUSt3R [ 19 ] to estimate camera poses for input images and integrate dense stereo priors with co-visibility relationships, enabling pixel-aligned, progressive expansion of scene geometry without redundancy. Jiang et al. [ 5 ] proposed a method for progressively constructing scenes by utilizing monocular depth to project pixels back into the 3D space, with camera registration and pose adjustments optimized within the pipeline. In this paper, we present a novel framework for reconstructing 360 ∘ scenes in extremely sparse views (four inputs only). We leverage 3DGS for its efficiency in training and rendering as our primary 3D representation method. To overcome the challenge of estimating camera poses in extremely sparse views, we utilize DUSt3R [ 19 ] to estimate camera poses and generate a dense point cloud for a scene. We expand more view coverage in 3D space by densely sampling views in the upper hemisphere, where additional training images are rendered using these sampled camera poses and the generated point cloud. Unlike existing methods [ 25 , 13 ] , where the camera poses are sampled along the trajectory of the input cameras, our approach provides more comprehensive information of appearance and geometry of scenes from 3D space to train a 3DGS model. Subsequently, we remove artifacts in the point-cloud-rendered images using a diffusion-based image enhancement model. Retraining the model on our created dataset, our method leverages the generative capabilities of diffusion models to enhance the quality of the point-cloud-rendered images without introducing new content into the images. The 3DGS model is trained on a combination of the reference images from sparse views and densely sampled synthetic images, enabling the use of more scene information of appearance and geometry for training, yielding the improvement of performance for extremely sparse-view cases. 2 Method 2.1 Preliminary 3DGS [ 6 ] is an explicit representation for modeling scenes with parameterized 3D Gaussians. A 3D Gaussian is parameterized by G=\{\mu,q,s,o,c,sh\} , where \mu is spatial mean, q is rotation quaternion, s is scaling vector, o is opacity, c is a view-dependent color, and sh is spherical harmonic (SH) coefficients. A 3D scene is represented by a

Read the complete text