UniRelight: Learning Joint Decomposition and Synthesis for Video Relighting

Kai He, Ruofan Liang, Jacob Munkberg, Jon Hasselgren, Nandita Vijaykumar, Alexander Keller, Sanja Fidler, Igor Gilitschenski, Zan Gojcic, Zian Wang · 2025

Read UniRelight: Learning Joint Decomposition and Synthesis for Video Relighting by Kai He, Ruofan Liang, Jacob Munkberg, Jon Hasselgren, Nandita Vijaykumar, Alexander Keller, Sanja Fidler, Igor Gilitschenski, Zan Gojcic, Zian Wang (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2506-15673.txt.

Introduction

Kai He and co-authors presented UniRelight in 2025, a model that relights a photo or video in one pass. Relighting needs a deep grasp of scene parts plus clean synthesis of new light. Models trained end to end lack paired examples under many lights, while two-stage methods pile up errors.

Their model estimates surface color and makes the relit output together, using the generative power of video diffusion models. Joint work sharpens scene understanding and yields rich effects such as shadows, reflections, and see-through materials. Training mixed made-up multi-light data with auto-labeled real videos. Tests beat past methods on image quality and steadiness over time, across many scene types.

Catalog record

CREATORKai He, Ruofan Liang, Jacob Munkberg, Jon Hasselgren, Nandita Vijaykumar, Alexander Keller, Sanja Fidler, Igor Gilitschenski, Zan Gojcic, Zian Wang
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE49 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2506-15673

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

UniRelight: Learning Joint Decomposition and Synthesis for Video Relighting Kai He, Ruofan Liang, Jacob Munkberg, Jon Hasselgren, Nandita Vijaykumar, Alexander Keller, Sanja Fidler, Igor Gilitschenski, Zan Gojcic, Zian Wang Abstract We address the challenge of relighting a single image or video, a task that demands precise scene intrinsic understanding and high-quality light transport synthesis. Existing end-to-end relighting models are often limited by the scarcity of paired multi-illumination data, restricting their ability to generalize across diverse scenes. Conversely, two-stage pipelines that combine inverse and forward rendering can mitigate data requirements but are susceptible to error accumulation and often fail to produce realistic outputs under complex lighting conditions or with sophisticated materials. In this work, we introduce a general-purpose approach that jointly estimates albedo and synthesizes relit outputs in a single pass, harnessing the generative capabilities of video diffusion models. This joint formulation enhances implicit scene comprehension and facilitates the creation of realistic lighting effects and intricate material interactions, such as shadows, reflections, and transparency. Trained on synthetic multi-illumination data and extensive automatically labeled real-world videos, our model demonstrates strong generalization across diverse domains and surpasses previous methods in both visual fidelity and temporal consistency. † † footnotetext: † Joint Advising 1 Introduction Lighting is crucial in defining a scene’s visual appearance, influencing both aesthetics and the perception of objects and materials. Modifying illumination while preserving scene content enables a wide range of applications, including creative editing, simulation, and robust vision systems. Realistic relighting necessitates an accurate understanding of scene geometry and materials, as well as faithful light transport modeling. Recent methods aim to learn these complex relationships from data, but a significant challenge is the scarcity of multi-illumination datasets capturing the same scene under varied lighting conditions. Acquiring such data in the real world requires controlled environments, calibrated equipment, and repeated captures, making it costly and impractical. As a result, most existing approaches rely on synthetic data or focus on narrow domains like portraiture, limiting their ability to generalize to real-world scenes. To address the lack of multi-illumination data, recent works zeng2024rgb ; ruofan2025diffusion have decomposed the relighting task into two stages: an inverse rendering step that estimates scene attributes such as albedo, normal, and depth (i.e., G-buffers), followed by a forward rendering step that synthesizes relit images conditioned on these estimates. This pipeline avoids the need for multi-illumination supervision, allowing each stage to be trained separately using single-illumination data. However, inverse and forward rendering are inherently coupled, and modeling them separately with discrete intermediate representations introduces significant limitations. The forward renderer is highly sensitive to errors in the estimated G-buffers and struggles to capture complex material properties not encoded by them, such as transparency and subsurface scattering. Learning an end-to-end relighting model in low data regime is challenging. While prior methods for intrinsic estimation (e.g., geometry ke2023repurposing ; fu2024geowizard and albedo kocsis2023iid ; ruofan2025diffusion ) trained on synthetic data often generalize well to real-world scenes, we observe that directly training relighting models as video-to-video translation leads to poor generalization to unseen domains. We argue that robust relighting requires an explicit understanding of illumination and scene properties. Hence, we propose a relighting framework that jointly models the distribution of scene intrinsics and illumination. Inspired by VideoJAM hila2025videojam , we train a video generative model that jointly denoises latent space for relighting and albedo demodulation in a single pass. Practically, we concatenate both latent representations and treat them as a single video clip. This design is motivated by the hypothesis that demodulation provides a strong prior for the relighting task, such as removing shadows. This joint formulation encourages the model to learn an internal representation of scene structure, leading to improved generalization across diverse and unseen domains. Our approach contrasts with two-stage pipelines that rely on explicit G-buffer estimation. Instead, we implicitly reason about intrinsic scene representation, enabling better representation learning, reducing error accumulation, and modeling of complex visual effects such as specular highlights, transparency, and subsurface scattering. Our formulation also offers flexibility to learn from diverse sources of supervision. We leverage high-quality synthetic data for full supervision and complement it with real-world single illumination videos that can be auto-labeled at scale. This hybrid training strategy enables the model to handle complex lighting effects while significantly improving generalization to unseen domains, see Figure 1 . UniRelight enables high-quality relighting and intrinsic decomposition from a single input image or video, producing temporally consistent shadows, reflections, and transparency, and outperforms state-of-the-art methods. 2 Related Work Figure 1: Given an input image (top left) or video, our method jointly estimates albedo (bottom left) and synthesizes relit videos with novel lighting conditions using provided HDR probes. Notably, our estimated albedo maps effectively demodulate shadows and specular highlights, while the relit images exhibit plausible shadows and specular highlights. Inverse rendering estimates intrinsic scene properties such as geometry, materials, and lighting from input images. Traditional approaches use hand-crafted priors within an optimization framework land1971lightness1 ; barrow1978recovering ; grosse2009ground ; bousseau2009user ; zhao2012closed ; barron2014shape to handle low-order effects like diffuse shading and reflectance. Recently, the field has been revitalized by applying supervised and self-supervised learning bell2014intrinsic ; kovacs2017shading ; li2018cgintrinsics ; neuralSengupta19 ; yu19inverserendernet ; li2020inverse ; li2020openrooms ; Boss2020-TwoShotShapeAndBrdf ; wang2021learning ; wang2022neural ; wimbauer2022rendering ; bhattad2023styleganknowsnormaldepth to inverse rendering tasks. These methods typically require large, domain-specific datasets, and struggle to generalize outside the training domain. Most related to our approach are methods that leverage large generative image and video models for inverse rendering intrinsiclora ; kocsis2023iid ; Phongthawee2023DiffusionLight ; liang2024photorealistic ; kocsis2025intrinsix ; chen2025unirenderer ; munkberg2025videomat . Notably, RGB \leftrightarrow X zeng2024rgb demonstrates both intrinsic decomposition and neural rendering from intrinsics using finetuned image diffusion models. The DiffusionRenderer ruofan2025diffusion extends RGB \leftrightarrow X to video and also supports relighting. Relighting is the task of modifying the lighting conditions in images or videos. Many methods first reconstruct 3D scenes from multi-view images munkberg2021nvdiffrec ; hasselgren2022nvdiffrecmc ; chen2021dibrpp ; boss2021nerd ; physg2021 ; zhang2021nerfactor ; iron-2022 ; rudnev2022nerfosr ; wang2023fegr ; liang2023envidr ; chen2024intrinsicanything ; liang2023gs ; shi2023gir ; jiang2024gaussianshader . Previously, surface reflectance has been captured using calibrated light stages debevec2000acquiring . Now, relighting is supported by material properties recovered from inverse rendering. These methods typically require per-scene optimization and are limited to static, object-centric scenes with single illumination. Training across multiple scenes has led to the exploration of latent feature learning Zhou_2019_ICCV ; liu2020factorize ; StyLitGAN , often incorporating neural rendering modules with intrinsic buffers as priors philip2019multi ; griffiths2022outcast ; TotalRelighting ; kocsis2024lightit ; xing2024luminet . Recent approaches diffusion_relighting ; kocsis2024lightit ; zeng2024dilightnet ; jin2024neural_gaffer ; ruofan2025diffusion ; bharadwaj2024genlit ; zhang2025relitlrm leverage diffusion models for relighting tasks. These methods are often domain-specific, such as portraits, single objects, and outdoor scenes. Despite promising results, a significant challenge remains the need for multi-illumination datasets murmann19 , which are difficult to capture at scale. Joint generative modeling approaches enable diffusion models to predict multiple modalities. Matrix3D lu2025matrix3d predicts pose estimation, depth, and novel view synthesis using a single DiT peebles2022dit model. VideoJAM hila2025videojam extends this by predicting both generated pixels and their corresponding motion from a single DiT. We leverage this approach to jointly predict a relit image and albedo. Figure 2: Method overview. Given an input video

Read the complete text