MS-GS: Multi-Appearance Sparse-View 3D Gaussian Splatting in the Wild

Deming Li, Kaiwen Jiang, Yutao Tang, Ravi Ramamoorthi, Rama Chellappa, Cheng Peng · 2025

Read MS-GS: Multi-Appearance Sparse-View 3D Gaussian Splatting in the Wild by Deming Li, Kaiwen Jiang, Yutao Tang, Ravi Ramamoorthi, Rama Chellappa, Cheng Peng (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2509-15548.txt.

Introduction

Deming Li leads a team that rebuilds three dimensional scenes from few photos taken in the wild. Such photo sets vary in light, time of day, and season. Past methods tend to smooth detail or fit the few views too tightly. The team build on a method that splats soft blobs to form a scene.

Their system, MS-GS, starts from depth guesses made from single photos. A step anchored on known scene points pulls out local meaning regions for firm alignment. They then add checks at fake views, at both pixel and trait levels, to keep the scene true from many angles. They also release a new data set and test setup. Results look photo real across hard cases.

Catalog record

CREATORDeming Li, Kaiwen Jiang, Yutao Tang, Ravi Ramamoorthi, Rama Chellappa, Cheng Peng
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE46 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2509-15548

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

MS-GS: Multi-Appearance Sparse-View 3D Gaussian Splatting in the Wild Deming Li, Kaiwen Jiang, Yutao Tang, Ravi Ramamoorthi, Rama Chellappa, Cheng Peng Abstract In-the-wild photo collections often contain limited volumes of imagery and exhibit multiple appearances, e.g., taken at different times of day or seasons, posing significant challenges to scene reconstruction and novel view synthesis. Although recent adaptations of Neural Radiance Field (NeRF) and 3D Gaussian Splatting (3DGS) have improved in these areas, they tend to oversmooth and are prone to overfitting. In this paper, we present MS-GS, a novel framework designed with M ulti-appearance capabilities in S parse-view scenarios using 3D GS . To address the lack of support due to sparse initializations, our approach is built on the geometric priors elicited from monocular depth estimations. The key lies in extracting and utilizing local semantic regions with a Structure-from-Motion (SfM) points anchored algorithm for reliable alignment and geometry cues. Then, to introduce multi-view constraints, we propose a series of geometry-guided supervision at virtual views in a fine-grained and coarse scheme to encourage 3D consistency and reduce overfitting. We also introduce a dataset and an in-the-wild experiment setting to set up more realistic benchmarks. We demonstrate that MS-GS achieves photorealistic renderings under various challenging sparse-view and multi-appearance conditions and outperforms existing approaches significantly across different datasets. 1 Introduction High-quality scene reconstruction and novel view synthesis from images is a long-standing research problem with wide-ranging applications in AR/VR, 3D site modeling, autonomous driving, robotics, etc. Remarkably, neural radiance field (NeRF) [ 1 ] and 3D Gaussian Splatting (3DGS) [ 2 ] achieve photorealistic novel view synthesis with differentiable rendering pipelines and different scene parameterizations. Both approaches build on the fundamental constraint that a voxel in space projects to similar photometric values across views, a constraint that requires a dense scene coverage and multi-view consistency. In practice, such coverage and consistency are often not guaranteed, largely affecting their performance in unconstrained settings. With sparse image sets, overfitting the photometric objectives to an incorrect geometry is a common issue in novel view synthesis. To counter this, semantic constraints [ 3 ] , depth and novel-view regularization [ 4 , 5 , 6 ] , frequency regularization [ 7 ] , and ray-entropy minimization [ 8 ] , have been introduced. While effective, these NeRF-based methods incur a heavy computational cost because volumetric rendering requires densely sampling points along camera rays. More recently, 3DGS adaptations have pushed sparse-view synthesis further by exploiting their explicit representation and fast rasterization. Depth regularization [ 9 ] , floater pruning [ 10 ] , and proximity-based Gaussian densification [ 11 ] regularize reconstruction and suppress artifacts during training. Despite improved metrics, these methods remain limited by the standard initialization of sparse Structure-from-Motion (SfM) point cloud when few features are triangulated correctly. Furthermore, applying monocular depth constraints globally to 3DGS is often inaccurate, leading to noisy gradients that prevent proper densification in sparse regions. As shown in Fig. 1(h) , they synthesize overly smooth regions, while our method recovers fine surface details. Figure 1 : With 20 input views , DNGS and FSGS produce overly smooth rendering in regions lacking support from sparse point cloud initialization. For scenes with multiple appearances and sparse inputs, methods like GS-W and Wild-GS experience large artifacts at novel views. In contrast, our method in Fig. 1(d) and 1(h) renders details and provides a coherent reconstruction. (a) Ground truth (b) DNGS [ 9 ] (c) FSGS [ 11 ] (d) Ours (e) Ground truth (f) GS-W [ 12 ] (g) Wild-GS [ 13 ] (h) Ours Beyond limited viewpoints, in-the-wild image sets often exhibit photometric inconsistencies across views. These range from subtle exposure shifts to pronounced appearance changes when the same scene is captured at different times of day or in different weather. NeRF-in-the-Wild [ 14 ] first demonstrated the ability to model a canonical 3D structure from multi-appearance imagery; since then, various works have followed up to improve synthesis quality [ 15 , 16 ] and incorporate this capability to 3DGS-based approaches [ 12 , 13 , 17 ] . These methods require more data to disambiguate image-specific radiance compared to appearance-consistent scenarios. As shown in Fig. 1(h) , a moderate number of views still leads to noisy rendering, greatly limiting the application of these methods. In this paper, we present MS-GS, which improves the robustness of 3DGS in dealing with unconstrained images when limited viewpoints and varying appearances exist, which is underexplored. We find that the performance of 3DGS relies heavily on the initial point cloud: these explicit structures steer the adaptive control of Gaussians and subsequent optimization. To overcome the limitation of the sparse SfM point cloud with limited views, we draw knowledge from the monocular depth estimators [ 18 , 19 , 20 ] that have rapidly progressed. MS-GS aligns the depth prediction with SfM depth, then back-projects pixels into the scene space for a dense point cloud. A key challenge is that monocular depth estimation is often incorrect at relative depth between objects due to single-view ambiguity. We address this with a Semantic Depth Alignment approach. A point-prompted segmentation model [ 21 ] is leveraged to extract semantically consistent regions using projected SfM points. We design an iterative refinement algorithm to identify each region–expanding or discarding according to the number of enclosed SfM points–and perform alignment inside it before back-projection. The resulting point cloud is denser and better structured than the original sparse SfM output, helping regularize 3DGS structures and promote Gaussian densification. To enable sparse-view multi-appearance scene modeling, MS-GS decomposes appearance into an image-specific and Gaussian-specific component. The per-image appearance embedding captures the global appearance variations, while each Gaussian’s feature embedding encodes the canonical scene appearance. Under sparse-view conditions, it becomes challenging to disambiguate the image-specific radiance from a consistent scene appearance, resulting in overfitting to training image appearance. Building upon the sufficiently accurate geometry that 3DGS optimizes with our dense initialization, we exploit the use of virtual views for multi-view constraints, where a series of Geometry-guided Supervisions based on 3D warping is proposed. Specifically, we back-project training images to 3D and then project to virtual views created between training cameras to establish correspondences. Appearance consistency is enforced at a fine-grained pixel level and a coarse semantic feature level for precise supervision and handling occluded areas. This approach aims to transfer the well-rendered appearance of training images to multiple views. Coupled with our densified point cloud, this design markedly improves geometric coherence and the rendering quality given sparse and multi-appearance imagery. In addition, the benchmark dataset Phototoursim [ 22 ] is collected through the internet such that each image has a unique appearance. Therefore, methods evaluated on this dataset require access to the test view image to obtain appearance, which is not ideal for train-test separation. To this end, we introduce an unbounded drone dataset that features multi-view appearance. With only camera metadata, not the target images themselves, testing views are rendered from the appearance of training images during evaluation. Similarly, the experimental setting of novel view synthesis in the wild is non-trivial. While methods often assume camera poses are exact, sparse-view and multi-appearance registration are themselves prone to error. Accurate reflection of performance in the wild needs to

Read the complete text