Relit-LiVE : Reli gh t Video by Jointly L earning E nvironment Vi deo Conference: Special Interest Group on Computer Graphics and Interactive Techniques Conference Conference Papers; July 19–23, 2026; Los Angeles, CA, USA Special Interest Group on Computer Graphics and Interactive Techniques Conference Conference Papers (SIGGRAPH Conference Papers ’26), July 19–23, 2026, Los Angeles, CA, USA DOI: 10.1145/3799902.3811200 ISBN: 979-8-4007-2554-8/2026/07 CCS: Computing methodologies Rendering CCS: Computing methodologies Computer vision

Weiqing Xiao, Hong Li, Xiuyu Yang, Houyuan Chen, Wenyi Li, Tianqi Liu, Shaocong Xu, Chongjie Ye, Hao Zhao, Beibei Wang · 2026

Read Relit-LiVE : Reli gh t Video by Jointly L earning E nvironment Vi deo Conference: Special Interest Group on Computer Graphics and Interactive Techniques Conference Conference Papers; July 19–23, 2026; Los Angeles, CA, USA Special Interest Group on Computer Graphics and Interactive Techniques Conference Conference Papers (SIGGRAPH Conference Papers ’26), July 19–23, 2026, Los Angeles, CA, USA DOI: 10.1145/3799902.3811200 ISBN: 979-8-4007-2554-8/2026/07 CCS: Computing methodologies Rendering CCS: Computing methodologies Computer vision by Weiqing Xiao, Hong Li, Xiuyu Yang, Houyuan Chen, Wenyi Li, Tianqi Liu, Shaocong Xu, Chongjie Ye, Hao Zhao, Beibei Wang (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2605-06658.txt.

Introduction

Weiqing Xiao and nine co-authors present Relit-LiVE, a framework that relights videos under new illumination without needing camera poses. Neural relighting usually decomposes a video into intrinsic scene parts first, but that decomposition is unreliable on real footage and errors pile up as distorted materials and flicker. Their key move is feeding raw reference images directly into the rendering process.

A single diffusion process jointly generates the relit video and per-frame environment maps aligned to each viewpoint, enforcing consistency between geometry and lighting. That supports moving lights and cameras naturally. Experiments beat leading relighting and neural rendering methods on synthetic and real benchmarks, with uses in material editing and object insertion.

Catalog record

CREATORWeiqing Xiao, Hong Li, Xiuyu Yang, Houyuan Chen, Wenyi Li, Tianqi Liu, Shaocong Xu, Chongjie Ye, Hao Zhao, Beibei Wang
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE64 KB
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2605.06658

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Relit-LiVE : Reli gh t Video by Jointly L earning E nvironment Vi deo Conference: Special Interest Group on Computer Graphics and Interactive Techniques Conference Conference Papers; July 19–23, 2026; Los Angeles, CA, USA Special Interest Group on Computer Graphics and Interactive Techniques Conference Conference Papers (SIGGRAPH Conference Papers ’26), July 19–23, 2026, Los Angeles, CA, USA DOI: 10.1145/3799902.3811200 ISBN: 979-8-4007-2554-8/2026/07 CCS: Computing methodologies Rendering CCS: Computing methodologies Computer vision Weiqing Xiao, Hong Li, Xiuyu Yang, Houyuan Chen, Wenyi Li, Tianqi Liu, Shaocong Xu, Chongjie Ye, Hao Zhao, Beibei Wang Abstract . Recent advances have shown that large-scale video diffusion models can be repurposed as neural renderers by first decomposing videos into intrinsic scene representations and then performing forward rendering under novel illumination. While promising, this paradigm fundamentally relies on accurate intrinsic decomposition, which remains highly unreliable for real-world videos and often leads to distorted appearances, broken materials, and accumulated temporal artifacts during relighting. In this work, we present Relit-LiVE , a novel video relighting framework that produces physically consistent, temporally stable results without requiring prior knowledge of camera pose. Our key insight is to explicitly introduce raw reference images into the rendering process, enabling the model to recover critical scene cues that are inevitably lost or corrupted in intrinsic representations. Furthermore, we propose a novel environment video prediction formulation that simultaneously generates relit videos and per-frame environment maps aligned with each camera viewpoint in a single diffusion process. This joint prediction enforces strong geometric–illumination alignment and naturally supports dynamic lighting and camera motion, significantly improving physical consistency in video relighting while easing the requirement of known per-frame camera pose. To further enhance generalization, we introduce two complementary training strategies: (i) latent-space interpolation between relighting and rendering outputs to synthesize diverse, photorealistic multi-illumination data, and (ii) a cycle-consistent self-supervised illumination learning scheme that enforces temporal lighting coherence without additional annotations. Extensive experiments demonstrate that Relit-LiVE consistently outperforms state-of-the-art video relighting and neural rendering methods across synthetic and real-world benchmarks. Beyond relighting, our framework naturally supports a wide range of downstream applications, including scene-level rendering, material editing, object insertion, and streaming video relighting. The Project is available at https://github.com/zhuxing0/Relit-LiVE . Figure 1. We present Relit-LiVE , a novel video relighting framework that produces physically consistent and temporally stable results without needing prior knowledge of camera pose. This is achieved by jointly generating relighting videos and environment videos. Additionally, by integrating real-world lighting effects with intrinsic constraints, the relighting videos demonstrate remarkable physical plausibility, showcasing realistic reflections and shadows. † † cc-license: by 1. Introduction Video relighting aims to modify a video’s illumination while preserving the scene’s intrinsic properties. It has various applications, including content creation, creative editing, and robust vision systems. However, it remains a long-standing challenge to achieve physically consistent and temporally accurate lighting effects, such as realistic reflections or stable, time-coherent shadows. Addressing this requires not only accounting for different material properties but also precise, controllable modeling of lighting conditions. Building upon powerful pre-trained diffusion models, several studies ( 47 ; 22 ) directly generate relit videos using text prompts or background images as lighting conditions. While achieving breakthroughs in visual quality, these methods typically lack precise lighting control and often retain artifacts from the original illumination. In contrast to direct generation, another line of research ( 17 ; 11 ) explores a two-stage architecture that incorporates an intermediate step of intrinsic decomposition. This approach first separates scene intrinsics from illumination, then performs relighting synthesis based on these components, using environment maps for conditioning. This explicit separation enables a clearer decoupling between scene properties and lighting, facilitating higher visual quality and more precise control. However, this paradigm is heavily dependent on the fidelity of the intermediate intrinsic representation. In challenging scenarios, such as transparent objects with complex light transport or subsurface scattering, neural intrinsic rendering might yield flawed or implausible outputs. A recent work by He et al. ( 13 ) unifies albedo estimation with direct relighting, synthesizing scene albedo and relighting video in parallel to effectively decouple and reshape scene illumination. However, constrained by the inherent challenges of training parallel inference paradigms, their approach struggles to extend to more intrinsic properties, limiting its capabilities. Furthermore, these methods require precise prior knowledge of the video camera’s pose to position the environment map in the viewport, which constrains their flexibility. In this paper, we propose Relit-LiVE , a novel video relighting framework that produces physically consistent, temporally stable results without requiring prior knowledge of camera pose. To this end, we address two core challenges: (1) preserving scene content integrity under complex light transport, and (2) flexibly injecting novel lighting conditions without known camera pose. We present two key insights to address these challenges. First, while decomposed intrinsic attributes often struggle to capture complex global illumination effects, these effects are directly observable in the original RGB video sequence. Therefore, we propose an RGB-intrinsic fusion renderer that utilizes the input RGB frames—also known as raw reference images—to guide and correct the rendering process, providing both visual and semantic-level cues. This design fuses the RGB space with the intrinsic space, enabling the model to incorporate real-world lighting effects alongside estimated physical constraints, resulting in realistic relighting results. Second, to facilitate arbitrary relighting without requiring per-frame camera poses, we reformulate relit video learning as the simultaneous learning of a per-frame warping of the environment map in combination with relit video synthesis. This approach allows our model to generate both relit videos and per-frame warped environment maps (referred to as environment video) during a single inference pass. By inferring the lighting transformation implicitly, our approach eliminates the need for explicit pose estimation, enhancing practical flexibility. Furthermore, we improve the robustness of our model to handle complex scenarios by enhancing the training data in two ways. First, we perform latent-space interpolation between relighting and rendering outputs using the initially trained model. This allows us to synthesize diverse, photorealistic multi-illumination data. Second, we employ a cycle-consistent self-supervised illumination learning scheme that ensures temporal lighting coherence without the need for additional annotations. Extensive experiments demonstrate that Relit-LiVE outperforms existing state-of-the-art methods, achieving realistic material reflection effects and effectively modeling viewpoint changes in videos. This enables us to perform physically plausible and spatio-temporally accurate relighting of videos without requiring camera pose priors. Relit-LiVE also offers flexibility for task extension, enabling scene-level rendering, editing, and streaming video relighting through modifying generation conditions and intermediate outputs. In summary, our contributions are as follows: • a novel video relighting framework, Relit-LiVE , that produces physically consistent, temporally stable results without requiring prior knowledge of camera pose, • an RGB-intrinsic fusion renderer, that effectively integrates real-world lighting effects from the RGB space with physical constraints from the intrinsic space, enabling the generation of physically consistent video lighting effects, and • jointly generation of relit video and environment video, enabling geometry-illumination aligned video relighting without requiring per-frame camera poses. 2. Related work Figure 2. Overview of our Relit-LiVE . Given an input video and environment maps of the initial viewpoint, our method jointly predicts relit videos and frame-specific environment maps (i.e., environment video). The input video is converted into intrinsic properties by a pre-trained inverse rendering model, then mapped into latent space alongside environment

Read the complete text