LumiMotion: Improving Gaussian Relighting with Scene Dynamics

Joanna Kaleta, Piotr Wójcik, Kacper Marzol, Tomasz Trzciński, Kacper Kania, Marek Kowalski · 2026

Read LumiMotion: Improving Gaussian Relighting with Scene Dynamics by Joanna Kaleta, Piotr Wójcik, Kacper Marzol, Tomasz Trzciński, Kacper Kania, Marek Kowalski (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2604-10994.txt.

Introduction

Kaleta and five colleagues present LumiMotion, the first Gaussian-based method using scene motion to guide inverse rendering. Inverse rendering recovers lighting and material properties, but existing Gaussian splatting methods target static scenes and assume simple lighting. Motion reveals the same surfaces under changing light, giving stronger cues to separate material from light. LumiMotion learns a dynamic splatting model that lets moving regions deform while static regions stay stable. Tests show gains of 23 percent on albedo and 15 percent on relighting over the next-best baseline.

Catalog record

CREATORJoanna Kaleta, Piotr Wójcik, Kacper Marzol, Tomasz Trzciński, Kacper Kania, Marek Kowalski
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE51606
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2604.10994

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

LumiMotion: Improving Gaussian Relighting with Scene Dynamics Joanna Kaleta, Piotr Wójcik, Kacper Marzol, Tomasz Trzciński, Kacper Kania, Marek Kowalski Abstract In 3D reconstruction, the problem of inverse rendering, namely recovering the illumination of the scene and the material properties, is fundamental. Existing Gaussian Splatting-based methods primarily target static scenes and often assume simplified or moderate lighting to avoid entangling shadows with surface appearance. This limits their ability to accurately separate lighting effects from material properties, particularly in real-world conditions. We address this limitation by leveraging dynamic elements—regions of the scene that undergo motion—as a supervisory signal for inverse rendering. Motion reveals the same surfaces under varying lighting conditions, providing stronger cues for disentangling material and illumination. This thesis is supported by our experimental results which show we improve LPIPS by 23% for albedo estimation and by 15% for scene relighting relative to next-best baseline. To this end, we introduce LumiMotion , the first Gaussian-based approach that leverages dynamics for inverse rendering and operates in arbitrary dynamic scenes. Our method learns a dynamic 2D Gaussian Splatting representation that employs a set of novel constraints which encourage the dynamic regions of the scene to deform, while keeping static regions stable. As we demonstrate, this separation is crucial for correct optimization of the albedo. Finally, we release a new synthetic benchmark comprising five scenes under four lighting conditions, each in both static and dynamic variants, for the first time enabling systematic evaluation of inverse rendering methods in dynamic environments and challenging lighting. Link to project page in footnote 1 1 1 https://joaxkal.github.io/LumiMotion/ . 1 Introduction Figure 1 : Qualitative results for LumiMotion and IRGS 13 on a real-world dataset. Note that thanks to modeling the scene dynamics LumiMotion can successfully remove shadows from the albedo, leading to improved relighting. IRGS struggles to correctly remove shadows from the albedo, and the specular component is not properly optimized. To allow for comparison of LumiMotion and IRGS in absence of the latter’s specularity estimation errors we show some of IRGS results with diffuse component only. Inverse rendering - the task of recovering geometry, material properties, and illumination from a set of images of a scene - is a fundamental challenge in computer vision and graphics 1 . Although methods like Neural Radiance Fields 33 and Gaussian Splatting 23 allow for accurately representing scene geometry, they represent the color of the reconstructed scene as it was observed, including shadows and other illumination-dependent effects. This conflation of lighting and material means the scene cannot be rendered under a different illumination (it cannot be relit). This limits the applicability of those reconstructions in fields such as gaming or film making, which require control over illumination. Recent works have introduced physically-motivated decomposition methods to address this limitation 13 ; 7 ; 11 , though, as shown in Fig. 1 , such approaches may struggle in real-world scenes with significant direct illumination. We hypothesize that the reason existing methods struggle is that they operate on static scenes, where the content is not moving. This lack of motion means that it is difficult to discern between the intrinsic color of objects and the observed color, which is a function of the incident light. For example, when viewing images of static scenes, it may be difficult to determine whether a certain part of the scene is darker because a shadow is cast on it or because the material itself is dark. Thus, we propose LumiMotion , the first Gaussian-based method to perform inverse rendering of arbitrary dynamic scenes. Based solely on a video sequence of a scene with dynamic elements, our approach reconstructs the scene’s geometry, as well as its material properties and illumination. The lighting conditions are assumed to be static, not known a priori and are inferred as part of the training. We leverage the scene dynamics to improve the quality of the estimated illumination and albedo, thereby enhancing the ability to render those scenes under novel illumination conditions. LumiMotion operates in two stages. In the first stage, we jointly learn the static scene geometry and a neural network that models the dynamics of the scene including changes to shape and color of objects in the scene. The geometry is modeled with 2D Gaussian Splats 16 that represent the surfaces in the scene with a collection of flat disks. A key benefit of this approach over 3D Gaussian Splatting or Neural Radiance Fields is that the 2D Gaussians have normal vectors associated with them and those are necessary for our second stage. In the second stage we freeze both the geometry and the neural network and proceed to inverse rendering. At this point we move away from rendering the Gaussian color directly and instead we compute the color of each Gaussian as a function of its material and the incident light that is computed via ray tracing. This allows for jointly optimizing the material (described jointly by albedo and roughness) and the illumination. Once we have those parameters, we can relight the scene with novel illumination or use the estimated illumination to relight other scenes. We demonstrate those abilities qualitatively on real data and quantitatively on a new synthetic dataset we created, where the ground truth illumination and material are known. Please see Fig. 1 for a teaser of the results. Our contributions can be summarized as follows: • First Gaussian-based approach for inverse rendering in arbitrary dynamic scenes, achieving better separation of material and illumination by leveraging scene dynamics as a supervisory signal. • A novel set of constraints on the deformation network that allows for better separation of static and dynamic parts of the scene and for improved modeling of scene geometry in time. • A new synthetic dataset allowing for comparing the performance of inverse rendering approaches for static and dynamic scenes. Table 1 : Overview of the representative relightable methods and datasets. Existing methods often target static scenes or focus on human avatars, assume known lights or require unnatural lighting setup. In contrast, LumiMotion uniquely performs inverse rendering of generic dynamic scenes under unknown natural lighting. We also show that none of the available datasets with known GT lighting meet the requirements for the tackled setup, which we address with our newly released dataset. Relightable methods Method Name Supports Dynamics Object-Agnostic Unknown Train Light Natural Train Light GS 3 2 ✗ ✓ ✗ ✗ R-3DGS 11 , GI-GS 7 , IR-GS 13 ✗ ✓ ✓ ✓ TensorIR 20 , NeRFactor 52 ✗ ✓ ✓ ✓ Relightable Neural Actor 30 ✓ ✗ ✗ ✓ Gaussian Codec Avatars 36 & 39 ✓ ✗ ✗ ✗ IntrinsicAvatar 38 ✓ ✗ ✓ ✓ Relightable […] Neural Avatars 49 ✓ ✗ ✓ ✓ LumiMotion ✓ ✓ ✓ ✓ Relightable datasets Dataset Name Domain Scene Type Train Light Type Static-Dynamic Reference OLAT 28 generic static OLAT ✗ TensoIR 20 generic static natural ✗ Synthetic4Relight 53 generic static natural ✗ Stanford-ORB 24 generic static natural ✗ RANA 18 avatars dynamic natural ✗ Codec Avatar Studio 32 avatars both multi-light ✗ LumiMotion generic both natural ✓ 2 Related Work Novel View Synthesis. Novel view synthesis aims to generate images of a scene from novel viewpoints using a limited set of input observations. Neural Radiance Fields

Read the complete text