SRUG : S hadow-Guided R elightable U rban Scene with G eneration Model
Yonghao Zhao, Zexin Yin, Jian Yang, Beibei Wang, Jin Xie † · 2026
Read SRUG : S hadow-Guided R elightable U rban Scene with G eneration Model by Yonghao Zhao, Zexin Yin, Jian Yang, Beibei Wang, Jin Xie † (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2605-24700.txt.
Introduction
Yonghao Zhao and others tackle relightable urban scenes, a task made hard by unbounded streets and unseen regions that still cast shadows. Sparse views and tricky lighting further muddy surface recovery. Their SRUG framework uses shadows to guide a 3D fill-in model rebuilding hidden shape, yielding sensible shadows. A step-by-step scheme then applies a large material model to split surface traits. A physics-based lighting model captures complex urban lighting. Number tests and visual checks beat existing methods on both novel views and new lighting.
Catalog record
| CREATOR | Yonghao Zhao, Zexin Yin, Jian Yang, Beibei Wang, Jin Xie † |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 62365 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2605.24700
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
SRUG : S hadow-Guided R elightable U rban Scene with G eneration Model Thanks: † Corresponding author. Yonghao Zhao, Zexin Yin, Jian Yang, Beibei Wang, Jin Xie † Abstract . Creating relightable urban scenes from images or videos is widely useful but highly ill-posed. Urban environments are typically unbounded and extend beyond the visible regions. As a result, many portions of the scene remain unobserved, yet these invisible regions can cast shadows onto visible areas. Reasonably modeling shadows cast by such invisible regions is challenging and poses a significant obstacle to creating relightable urban scenes. At the same time, sparse input views and complex illumination conditions further complicate relighting, as they introduce severe ambiguities in material decomposition. In this paper, we propose S hadow-guided R elightable U rban Scene with G eneration model (SRUG), a novel framework designed to address relighting challenges in urban scenes. SRUG leverages shadows to guide a 3D completion model for recovering the geometry of invisible regions, promoting the synthesis of physically reasonable shadows. In addition, SRUG employs an iterative material decomposition scheme that applies the large material model (LMM) to provide material supervision and iteratively decompose the scene’s material properties, enabling robust material decomposition. Building upon these components, we introduce a physically-based lighting model that captures the complex illumination of urban scenes and supports reliable relighting. Extensive quantitative evaluations and visual comparisons demonstrate that our method outperforms existing approaches in both novel view synthesis and relighting tasks. Figure 1 . We propose SRUG, a novel framework for constructing relightable urban scenes from multi-view images or videos. SRUG reconstructs 3D scene representations, enables robust material–lighting decomposition, and supports physically reasonable relighting. Compared with the existing urban scene relighting method UrbanIR ( 30 ) , SRUG more effectively disentangles lighting from material properties, avoiding lighting bake-in in the estimated albedo. During relighting, SRUG produces physically consistent shadows under changes in sun direction and avoids floating artifacts. Moreover, under environment map–based relighting, SRUG yields more reliable results due to its robust scene decomposition. 1. Introduction Reconstructing relightable 3D urban scenes from multiple images or videos has numerous applications, including building world models, aiding autonomous driving, and creating digital twins. However, this task is inherently difficult due to the ill-posed nature of separating lighting and materials from images or videos. The challenges are further intensified by the characteristics of urban environments. First, shadow modeling is essential for urban scene relighting. Yet, the unbounded nature of urban scenes leaves many regions unobserved. These invisible areas can still cast shadows onto visible regions, making shadow modeling particularly difficult. Moreover, urban environments exhibit complex illumination composed of sunlight, skylight, and indirect light, which complicates material–lighting decomposition and often leads to severe lighting bake-in artifacts. These challenges are further amplified under sparse-view settings. Several studies ( 41 ; 50 ; 30 ) have been developed to create relightable urban scenes by incorporating specially designed regularization priors into the neural radiance field (NeRF) ( 34 ) framework. However, these methods face challenges related to their low expressive power and high computational demands, which significantly limit both the effectiveness and efficiency of relighting. Furthermore, while these regularization priors assist in the separation of materials and lighting, they are restricted to diffuse colors, which leads to noticeable lighting bake-in issues. Recent advancements in 3D Gaussian Splatting (3DGS) ( 23 ) show enhanced representation capabilities. Although several approaches ( 28 ; 13 ; 21 ; 15 ; 12 ) address object- or small-scene relighting, they neglect invisible regions and struggle with material decomposition under complex urban illumination, limiting their effectiveness in urban scenes. GS-ID ( 12 ) leverages a large material model (LMM) for material decomposition and conventional shadow mapping for shadow modeling. However, it relies solely on the LMM, which limits the robustness of the decomposition, and it does not consider shadow modeling of invisible areas. In this paper, we propose S hadow-guided R elightable U rban Scene with G eneration model (SRUG), a novel framework for urban scene relighting. The core challenge of urban scene relighting lies in completing invisible regions, ensuring physically plausible shadow effects, and robustly decomposing scene materials under complex illumination and sparse-view conditions. To address the challenges posed by invisible regions, we propose a shadow-guided invisible geometry completion module. This module leverages shadows to guide a 3D completion model in recovering the geometry of invisible areas, thereby enabling physically plausible shadow effects. To facilitate shadow-based guidance, we introduce a differentiable Gaussian shadow mapping (DGSM), which models shadows in a differentiable manner and supports effective shadow-guided optimization. To achieve robust scene decomposition, we further employ a pretrained large material model to provide material supervision for scene material decomposition. However, directly applying LMMs to real-world scenes often produces unrealistic results due to the domain gap between the synthetic data used for LMM training and real-world urban application scenarios. To address this limitation, we propose an iterative material decomposition scheme that leverages the scene’s materials as additional conditioning to bridge the LMM domain gap, progressively refines material properties, and achieves robust decomposition. We then apply a physically based lighting model for urban scenes that explicitly accounts for sunlight, skylight, and indirect illumination, enabling reliable relighting. We evaluate our method on the real-world and synthetic datasets across novel view synthesis (NVS), material decomposition, and relighting tasks. Our framework consistently achieves state-of-the-art performance compared with existing relighting methods. To summarize, our main contributions are as follows: • a novel framework, SRUG, for constructing relightable urban scenes from multi-view images or videos, • a shadow-guided invisible geometry completion module that leverages shadow cues to guide a 3D completion model in recovering invisible regions and synthesizing physically plausible shadows, and • an iterative material decomposition scheme that uses LMM to refine material properties for robust decomposition. 2. Related Work Novel view synthesis generates images from arbitrary viewpoints using image sequences or videos. Neural radiance fields ( 34 ) pioneered scene representation through volume rendering with neural networks, achieving photorealistic synthesis. Subsequent works ( 35 ; 3 ; 46 ; 48 ; 39 ; 38 ; 22 ) have expanded NeRF’s applications across diverse domains. To enhance efficiency and quality, 3D Gaussian splatting ( 23 ) introduced a radiance field representation using 3D Gaussians, achieving state-of-the-art results through efficient rasterization. This advancement has inspired numerous extensions, including improvements to synthetic quality ( 32 ; 57 ; 8 ) , geometry reconstruction ( 19 ; 58 ; 60 ) , autonomous driving scenes ( 54 ; 63 ) , dynamic scenes ( 52 ; 20 ) , and inverse rendering ( 28 ; 13 ; 15 ; 12 ; 6 ) . Object-scale inverse rendering is a fundamental task in computer graphics and vision, aiming to estimate scene properties—geometry, materials, and lighting—from images or videos. Recent advances in neural rendering have enhanced inverse rendering by enabling joint estimation of these properties through differentiable rendering frameworks. Such methods improve the accuracy of property decomposition while maintaining high-quality rendering. Among these approaches, NeRF-based methods ( 22 ; 26 ; 31 ; 4 ; 5 ; 56 ; 61 ; 44 ) leverage Neural Radiance Fields to achieve inverse rendering, demonstrating the ability to recover scene properties while preserving view-consistent rendering. Other methods