Generalizable Sparse-View 3D Reconstruction from Unconstrained Images
Vinayak Gupta, Chih-Hao Lin, Shenlong Wang, Anand Bhattad, Jia-Bin Huang, [0.45em] University of Maryland, College Park · 2026
Read Generalizable Sparse-View 3D Reconstruction from Unconstrained Images by Vinayak Gupta, Chih-Hao Lin, Shenlong Wang, Anand Bhattad, Jia-Bin Huang, [0.45em] University of Maryland, College Park (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2604-28193.txt.
Introduction
Gupta and colleagues present GenWildSplat, a feed-forward system that rebuilds outdoor 3D scenes from sparse, unposed internet photos. From learned geometric priors it estimates depth, cameras and 3D Gaussians in a common space, adapts appearance to target lighting, and segments transient objects away. Trained by curriculum on synthetic and real data, it generalises across lighting and occlusion patterns. It renders in real time with state-of-the-art feed-forward quality on two benchmarks.
Catalog record
| CREATOR | Vinayak Gupta, Chih-Hao Lin, Shenlong Wang, Anand Bhattad, Jia-Bin Huang, [0.45em] University of Maryland, College Park |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 36562 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2604.28193
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Generalizable Sparse-View 3D Reconstruction from Unconstrained Images Vinayak Gupta, Chih-Hao Lin, Shenlong Wang, Anand Bhattad, Jia-Bin Huang, [0.45em] University of Maryland, College Park Abstract Reconstructing 3D scenes from sparse, unposed images remains challenging under real-world conditions with varying illumination and transient occlusions. Existing methods rely on scene-specific optimization using appearance embeddings or dynamic masks, which requires extensive per-scene training and fails under sparse views. Moreover, evaluations on limited scenes raise questions about generalization. We present GenWildSplat , a feed-forward framework for sparse-view outdoor reconstruction that requires no per-scene optimization. Given unposed internet images, GenWildSplat predicts depth, camera parameters, and 3D Gaussians in a canonical space using learned geometric priors. An appearance adapter modulates appearance for target lighting conditions, while semantic segmentation handles transient objects. Through curriculum learning on synthetic and real data, GenWildSplat generalizes across diverse illumination and occlusion patterns. Evaluations on PhotoTourism and MegaScenes benchmark demonstrate state-of-the-art feed-forward rendering quality, achieving real-time inference without test-time optimization. Figure 1 : GenWildSplat reconstructs 3D scenes from sparse, unposed images with varying illumination and transient objects in a single 3-second feed-forward pass, and no per-scene optimization is required. Given 2–6 input views, our method predicts novel views under target lighting conditions while handling occlusions. Top: Novel-view synthesis under different lighting from the same sparse inputs, demonstrating appearance control. Bottom: Reconstruction quality across varying input sparsity (2–6 views), showing view-consistent rendering even with minimal observations. In each block, the top-left image (inset) is an input view; the remaining images are novel-view predictions under novel lighting. All scenes are unseen during training, demonstrating strong generalization to real-world environments. 1 Introduction Reconstructing 3D scenes from 2D images is crucial for applications such as AR/VR and navigation 26 ; 57 . Extending these techniques to in-the-wild imagery remains challenging due to three factors: (1) Internet photos show wide lighting variations across time and seasons, (2) handheld captures contain transient occluders like tourists or vehicles that must be excluded, and (3) real-world scenes often provide sparse viewpoints, unlike curated multi-view datasets. Effective reconstruction requires disentangling static scene content from dynamic lighting and transient objects. Prior NeRF 4 ; 32 ; 35 and Gaussian Splatting 16 ; 49 ; 8 ; 45 ; 14 ; 44 methods rely on per-scene optimization and dense views, while sparse-view in-the-wild approaches are time-intensive. Feed-forward models 40 ; 11 ; 48 enable real-time reconstruction but are limited to fixed lighting and fail under dynamic conditions. In Tab. 1 , we show the key characteristic differences between the previous methods. While these methods perform well on benchmarks like PhotoTourism 34 , they fail on more challenging datasets such as MegaScenes 38 (Fig. 2 ), which feature sparse views, diverse lighting, and heavy occlusions. To address these limitations, we introduce GenWildSplat , a generalizable model for fast feed-forward 3D scene reconstruction from sparse in-the-wild scenes without requiring per-scene optimization. To our knowledge, this is the first approach to integrate both appearance and occlusion modeling within a feed-forward 3D reconstruction paradigm. The key insight is that combining large-scale synthetic and real-world data enables the model to learn robust associations across diverse illumination conditions. Our framework uses VGGT’s transformer to process sparse, unposed multi-view images into rich feature maps, which specialized heads decode into depth, camera parameters, and per-pixel Gaussians. The resulting set of attributes defines a canonical representation that captures a unified scene geometry disentangled from illumination. However, directly decoding these canonical Gaussians into a novel view via a differentiable rasterizer leads to multi-view inconsistencies. To effectively map the canonical space to a desired target lighting, we introduce an appearance adapter that conditions on the target lighting and transforms the canonical Gaussian colors to the corresponding target lighting space. We parameterize lighting information using a light code estimated by a light encoder, which represents it in a compact latent space. To handle transient objects, we leverage a pre-trained segmentation network that identifies dynamic elements, such as people or cars. It produces explicit occlusion masks that guide our model to ignore transient regions during supervision, ensuring a clean and stable 3D scene. Ideally, one would train the model using multi-view images under varying illumination to render novel views under new lighting conditions. However, the absence of such multi-view, multi-lighting datasets makes direct supervised learning infeasible. We train GenWildSplat without paired multi-view multi-illumination data, using unordered image collections. Each input image is mapped to a compact light code, and the appearance adapter conditions on this code and the canonical Gaussian colors to generate transformed colors, which are supervised via image reconstruction. Direct training on large-scale real-world data is unstable, as jointly learning geometry and illumination from sparse views is a highly ill-posed problem. To avoid collapse, we adopt a curriculum: first, learn appearance on synthetic data; and finally, add synthetic occlusions. This staged strategy enables stable optimization and strong generalization. Table 1 : Comparison of key characteristics across existing methods and our approach. Unlike optimization-based or feed-forward baselines, our method is fast, capable of sparse views, view-consistent, and generalizes to novel lighting conditions. Method In-the-Wild Fast View-Consis Few-Views Optimization-Based 36 ; 16 ✓ ✗ ✓ ✗ Feed-Forward Based 11 ✗ ✓ ✗ ✓ Ours ✓ ✓ ✓ ✓ Figure 2 : Limitations of Prior Work. Prior methods 16 ; 36 fail under sparse-view conditions. (a) Overfitting: Scene-specific optimization produces artifacts and geometric spikes with small camera perturbations. (b) Camera dependency: Methods rely on COLMAP for pose estimation, which fails under sparsity. Even with higher-quality transformer-based poses (e.g., VGGT), reconstructions exhibit severe artifacts and blurring. (c) Limited appearance adaptation: Test-time optimization cannot adapt to novel lighting, causing color bleeding and geometric distortions when target illumination differs from training conditions. Figure 3 : Overview of GenWildSplat. Given sparse, unposed images \{I_{i}\}_{i=1}^{V} with appearance variations and transient objects, a geometry transformer extracts multi-view features \mathbf{F}_{i} encoding semantic and geometric information. Specialized prediction heads process these features to output per-pixel depth \mathbf{D}_{i} , camera parameters (\mathbf{K}_{i},\mathbf{E}_{i}) , and Gaussian attributes, which are unprojected into canonical 3D Gaussians \mathbf{G}_{c} . A light encoder \mathcal{E}_{Light} extracts per-image lighting codes \mathbf{L}_{i}=\mathcal{E}_{Light}(I_{i}) . An MLP F_{\text{light}} modulates the canonical Gaussian colors using these codes: \mathbf{G}_{\ell_{i}}=F_{\text{light}}(\mathbf{G}_{c},\mathbf{L}_{i}) . Each set of transformed Gaussians \mathbf{G}_{\ell_{i}} is rasterized to reconstruct \hat{I}_{i} . A pre-trained segmentation network provides occlusion masks M_{i} to identify transient objects. Masked reconstruction loss focuses supervision on static content, enabling photorealistic, view-consistent reconstruction from sparse in-the-wild imagery. To evaluate generalization on unseen scenes, we benchmark GenWildSplat on both the Phototourism and the more challenging MegaScenes datasets. Our method consistently outperforms existing approaches 54 ; 16 ; 36 in reconstructing accurate scene geometry, modeling appearance under varying illumination, and effectively handling occlusions. Interestingly, GenWildSplat surpasses scene-specific methods 16 ; 36 that rely on per-image appearance optimization and test-time fine-tuning, demonstrating the strength of leveraging large-scale pre-trained priors for robust 3D understanding. Overall, GenWildSplat represents a step toward generalizable 3D reconstruction, offering a feed-forward, illumination- and occlusion-aware framework that scales to diverse, real-world environments for real-time 3D scene understanding. 2 Related Works Optimization‐based Novel View Synthesis (NVS) reconstructs 3D scenes from 2D images for novel viewpoint generation. Gaussian Splatting (3DGS) 15 represents scenes with explicit 3D Gaussian primitives, enabling real‐time rasterization via a CUDA pipeline. Extensions improve depth