LightSwitch: Multi-view Relighting with Material-guided Diffusion
Yehonathan Litman, Fernando De la Torre, Shubham Tulsiani, Carnegie Mellon University, https://yehonathanlitman.github.io/light_switch · 2025
Read LightSwitch: Multi-view Relighting with Material-guided Diffusion by Yehonathan Litman, Fernando De la Torre, Shubham Tulsiani, Carnegie Mellon University, https://yehonathanlitman.github.io/light_switch (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2508-06494.txt.
Introduction
Yehonathan Litman, Fernando De la Torre, and Shubham Tulsiani present LightSwitch, a tool to relight photos of an object. Their twenty twenty-five paper deals with three-dimensional scenes shot from many angles. The goal is to change the light on the object while keeping its true shape and look.
Past tools relight one image at a time and ignore what the object is made of. LightSwitch infers the material traits of the subject and feeds them into a tuned diffusion model. A design that scales lets it relight many views at once, in as little as two minutes. Tests on fake and real objects match or beat top past methods.
Catalog record
| CREATOR | Yehonathan Litman, Fernando De la Torre, Shubham Tulsiani, Carnegie Mellon University, https://yehonathanlitman.github.io/light_switch |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 30 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2508-06494
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
LightSwitch: Multi-view Relighting with Material-guided Diffusion Yehonathan Litman, Fernando De la Torre, Shubham Tulsiani, Carnegie Mellon University, https://yehonathanlitman.github.io/light_switch Abstract Recent approaches for 3D relighting have shown promise in integrating 2D image relighting generative priors to alter the appearance of a 3D representation while preserving the underlying structure. Nevertheless, generative priors used for 2D relighting that directly relight from an input image do not take advantage of intrinsic properties of the subject that can be inferred or cannot consider multi-view data at scale, leading to subpar relighting. In this paper, we propose Lightswitch, a novel finetuned material-relighting diffusion framework that efficiently relights an arbitrary number of input images to a target lighting condition while incorporating cues from inferred intrinsic properties. By using multi-view and material information cues together with a scalable denoising scheme, our method consistently and efficiently relights dense multi-view data of objects with diverse material compositions. We show that our 2D relighting prediction quality exceeds previous state-of-the-art relighting priors that directly relight from images. We further demonstrate that LightSwitch matches or outperforms state-of-the-art diffusion inverse rendering methods in relighting synthetic and real objects in as little as 2 minutes. Figure 1 : Consistent Multi-view via Material-Guided Relighting Diffusion. We present LightSwitch, a framework for multi-view relighting. Given any number of input images under an unknown illumination, LightSwitch leverages multi-view attention and inferred material properties to predict consistent relighting, enabling applications for 2D and 3D relighting. 1 Introduction We have witnessed impressive advances in the task of recovering 3D representations from multi-view captures, with methods like NeRF [ 27 ] and Gaussian Splatting [ 18 ] allowing one to reconstruct generic objects or scenes easily. While these representations excel at modeling detailed geometry and appearance, they only seek to model the appearance within the capture environment, thus baking in lighting effects into the obtained representation. This prevents reconstructions from being imported and relit in novel environments for applications such as virtual reality or synthesizing visual effects. In this work, we aim to enable such relightable rendering of 3D representations under generic illumination and present an approach that given posed multi-view images, enables synthesizing novel relit views. Current methods that tackle this relighting task can be categorized as either leveraging inverse rendering or learned relighting. In particular, the former class of methods [ 12 , 23 , 22 , 46 , 16 , 48 , 10 ] seek to infer a 3D representation disentangling geometry, appearance, and material properties, allowing relighting via physics-based rendering. However, these optimization-based approaches tend to be slow and their usage of (simple) differentiable renderers can limit their ability to model complex lighting effects. In contrast, ‘direct relighting’ methods learn to directly generate a relit image a given (captured/rendered) source image and a target illumination. By adapting image diffusion priors, these methods can efficiently synthesize photo-realistic quality output in a feed-forward manner. However, these approaches operate on a single view, leading to inconsistencies in relighting across viewpoints. Our work also adopts a learning-based approach for direct relighting, with the key insight that instead of formulating this as a single-view relighting task, we should formulate it as one of consistently relighting multiple input views. This can make the predictions consistent across views (making such a system better suited for 3D relighting), while also improving relighting performance as the cues observed in one view ( e.g . sharpness of a specularity) can inform the relighting in another. In addition to incorporating multi-view cues, we also draw inspiration from inverse rendering methods that benefit from understanding material properties and seek to leverage (predicted) material properties (intrinsics and albedo) as additional input. We build on these insights and propose ‘LightSwitch’, a relighting diffusion framework that produces multi-view consistent relighting informed by inferred intrinsic properties. We validate LightSwitch on both synthetic and real-world data and find that leveraging multi-view and predict material cues yield significantly improved relighting compared to prior learning-based relighting methods. We also tackle the ‘3D relighting’ task requiring synthesizing and consistently relighting a large set of query views, and design a distributed inference scheme for efficient inference. We show that when compared to state-of-the-art inverse rendering methods, LightSwitch allows improved/comparable relighting, while being significantly faster. 2 Related Work Image-based 3D Reconstruction. We have witnessed impressive recent progress in the task of recovering 3D from images. In particular representations like NeRF [ 27 ] , Gaussian Splatting [ 18 ] , and their variants [ 3 , 2 , 35 , 28 , 13 , 26 , 41 ] allow representing detailed geometry and appearance, and can be inferred from multi-view images of generic objects and scenes. More recent approaches, often leveraging generative priors, even allow inferring such detailed outputs from sparse or single-view input [ 49 , 24 , 36 , 39 , 25 , 7 ] , allowing end users to easily capture 3D. While these obtained reconstructs can capture the rich details of underlying 3D scenes, they are only capable of modeling the static environment observed in the images and cannot be imported to novel environments where their appearance would change under a different environmental illumination. Relighting via Inverse Rendering. To enable relighting under novel environments, some works utilize inverse rendering to recover a 3D relightable asset. These approaches model intrinsic properties [ 29 , 46 , 22 , 12 , 45 , 16 , 10 , 23 ] or light transport effects [ 33 , 48 , 5 ] to infer a factorized 3D representation that explains the observed images, allowing for relighting under novel illuminations. However, recovering intrinsics from a set of multi-view images is a non-trivial task, given that many different combinations of the intrinsic properties can be composed together to get the appearance in the source images. Some approaches use data-driven prediction of intrinsics [ 40 , 19 ] to aid this, for example MaterialFusion [ 23 ] , whose material model we adapt that uses a material prior model to aid with relighting. Nevertheless, inverse rendering pipelines are forced to rely on simple material models together with simpler lightweight differentiable renderers due to the computational constraints of physically-based renderers. Even so, optimization is still slow as real-time rendering requires considerable computation, making inverse rendering approaches time-consuming. Figure 2 : LightSwitch Material-Relighting Diffusion Framework. LightSwitch relights multi-view posed input images to a given target illumination. It infers and encodes multi-view consistent material image maps ( \mathbf{I}_{\text{d}},\mathbf{I}_{\text{orm}} ) using a material diffusion model (StableMaterialMV [ 23 ] ) and concatenates them to the Plücker ray maps ( \mathbf{P} ), encoded input images ( \mathbf{x}_{\text{src}} ), and noisy latents ( \mathbf{z}_{t} ) in the channel dimension. The multi-view relighting UNet denoises the noisy latents and cross-attends to the lighting latents concatenated with the latent lighting directions ( \mathbf{E}_{\text{dir}} ). The lighting latents are encoded from the processed target environment map images (\mathbf{E}^{H}_{\text{tgt}},\mathbf{E}^{L}_{\text{tgt}}) . The Stable Diffusion encoders and decoder are kept frozen. Learning Direct Relighting. An alternate relighting approach that allows for high fidelity relighting is to learn a model that directly predicts relit images, in particular by adapting generative diffusion priors for high quality generation. These direct feed-forward approaches allow for predicting accurate relighting in little time for high resolution images while generalizing to