LightMover: Generative Light Movement with Color and Intensity Controls

Gengze Zhou, Tianyu Wang, Soo Ye Kim, Zhixin Shu, Xin Yu, Yannick Hold-Geoffroy, Sumit Chaturvedi, Zhe Lin, Scott Cohen · 2026

Read LightMover: Generative Light Movement with Color and Intensity Controls by Gengze Zhou, Tianyu Wang, Soo Ye Kim, Zhixin Shu, Xin Yu, Yannick Hold-Geoffroy, Sumit Chaturvedi, Zhe Lin, Scott Cohen (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2603-27209.txt.

Introduction

Zhou and co-authors present LightMover, a generative tool for moving and recoloring light in a single photo. The method leans on video diffusion priors to keep illumination changes physically plausible. Light editing becomes a token sequence prediction task over the image. Control tokens set the light's position, color, and intensity together. Reflections, shadows, and falloff adjust along with the light itself. An adaptive pruning scheme cuts the control sequence length by 41 percent. A rendering pipeline generated consistent training pairs across many light settings.

Catalog record

CREATORGengze Zhou, Tianyu Wang, Soo Ye Kim, Zhixin Shu, Xin Yu, Yannick Hold-Geoffroy, Sumit Chaturvedi, Zhe Lin, Scott Cohen
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE49746
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2603.27209

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

LightMover: Generative Light Movement with Color and Intensity Controls Gengze Zhou, Tianyu Wang, Soo Ye Kim, Zhixin Shu, Xin Yu, Yannick Hold-Geoffroy, Sumit Chaturvedi, Zhe Lin, Scott Cohen Abstract We present LightMover, a framework for controllable light manipulation in single images that leverages video diffusion priors to produce physically plausible illumination changes without re-rendering the scene. We formulate light editing as a sequence-to-sequence prediction problem in visual token space: given an image and light-control tokens, the model adjusts light position, color, and intensity together with resulting reflections, shadows, and falloff from a single view. This unified treatment of spatial (movement) and appearance (color, intensity) controls improves both manipulation and illumination understanding. We further introduce an adaptive token-pruning mechanism that preserves spatially informative tokens while compactly encoding non-spatial attributes, reducing control sequence length by 41 % 41\% while maintaining editing fidelity. To train our framework, we construct a scalable rendering pipeline that generates large numbers of image pairs across varied light positions, colors, and intensities while keeping the scene content consistent with the original image. LightMover enables precise, independent control over light position, color, and intensity, and achieves high PSNR and strong semantic consistency (DINO, CLIP) across different tasks. Figure 1 : Results of light source movement. LightMover demonstrates robust control over light movement, color, and intensity in diverse illumination scenarios. It maintains global radiometric balance, reconstructs occluded illumination, and infers plausible light–object interactions under varying geometry and materials. Beyond single-attribute control, LightMover supports compositional manipulation that simultaneously adjusts light position, color, and intensity. These results highlight the potential of our 2.5D learning paradigm to achieve physically coherent lighting reasoning within a purely 2D generative framework. † † * Work done during an internship at Adobe. † † \dagger Corresponding author. 1 Introduction Imagine shopping online for a new floor lamp. Product photos reveal its appearance, but not what truly matters: how the lamp would change the mood and illumination of your own living room. With only a single image of a living room, a user naturally wants to insert a lamp, drag it across the room, warm its color, dim its brightness, and observe how these adjustments reshape shadows, reflections, and ambience. Achieving such light-centric editing in a single real image is extremely challenging as illumination interacts globally with geometry, materials, and occlusion, while the user expects precise, causal control. Existing methods fall into two categories. Inverse-rendering approaches Li et al. 2020 ; Sengupta et al. 2019 ; Li et al. 2022 ; Wang et al. 2025 ; Wu et al. 2025b reconstruct dense geometry, materials, and lighting before re-rendering the scene. While physically grounded, these pipelines are highly ill-posed from a single image, computationally expensive, and do not offer intuitive controls, such as moving or recoloring individual lights interactively. Diffusion-based editors (e.g., LightLab Magar et al. 2025 ) can manipulate tone and ambient illumination but do not model spatial light movement, while object-centric diffusion models (e.g., ObjectMover Yu et al. 2025 ) rely on video priors to move objects but treat illumination only implicitly, and thus fail to propagate the correct shadows, colors, and falloff when moving a light source. More recently, general generative image editing models (SDEdit Meng et al. 2022 , DiffEdit Couairon et al. 2023 , InstructPix2Pix Brooks et al. 2023 , Nano-Banana Google DeepMind 2025 , Qwen-Image(-Edit) Wu et al. 2025a ; Qwen Team 2025 ) support powerful text- or mask-driven semantic edits; however, they lack an explicit light-centric parameterization and thus struggle to provide precise, physically plausible control over individual lights. In this paper, we introduce LightMover, a video-diffusion framework for explicit, parametric illumination control from a single image. Our key idea is to extend the ObjectMover Yu et al. 2025 token formulation with illumination-specific control tokens for color and intensity, embedded using a multi-signal positional encoding (spatial, temporal, condition-type, and role). This design yields a unified model that supports both object edits (movement, insertion, removal) and illumination edits within the same backbone, avoiding task-specific adapters. A naive extension of this formulation is to encode color and intensity as extra “frames”, but this dramatically increases the token count and thereby limits the model’s efficiency and resolution. To avoid control tokens inflating the sequence length, we introduce an adaptive token-pruning strategy: spatial controls (e.g., movement maps) retain fine-grained tokens in small, localized regions and are downsampled elsewhere, while non-spatial controls (color, intensity) are represented with learnable compression ratios. This reduces the length of the control token by 41\% while maintaining accurate spatial and illumination control. To support training, we build a scalable physically-based rendering pipeline that generates matched image pairs varying in light location, spectrum, and intensity. These pairs enable the model to learn the causal effects of illumination: how shadows shift, how reflections brighten, and how indirect lighting propagates, to unlock capabilities that video priors alone cannot provide. Moreover, we find that training these illumination controls jointly in a multi-task manner produces strong mutual reinforcement: positional, color, and intensity signals regularize one another, improving accuracy and stability across all light-related tasks. In summary, our contributions are threefold: • We introduce a unified diffusion-based framework that formulates light control as a sequence-to-sequence task, enabling fine-grained manipulation of illumination. • We propose an adaptive token-pruning mechanism that improves efficiency by compactly encoding non-spatial attributes while preserving spatial precision. • We develop a scalable data generation pipeline with physically accurate rendering and a multi-task training strategy for light insertion, removal, and movement, resulting in state-of-the-art photorealistic and controllable lighting manipulation. 2 Related Works Object Movement. Diffusion-based inpainting methods ( Avrahami et al. 2022 ; Suvorov et al. 2021 ; Lugmayr et al. 2022 ; Meng et al. 2022 ; Saharia et al. 2022 ) represent the mainstream approach for generative removal, while insertion frameworks ( Lu et al. 2023 ; Zhang et al. 2023 ; Chen et al. 2024 ; Yang et al. 2023 ; Song et al. 2023 ; Song et al. 2024 ) extend them to conditionally synthesize new objects under identity constraints. However, inpainting inherently limits the handling of global illumination cues, as expanding masks often destroy scene consistency and fail to reconstruct shadows or reflections. Shadow-specific methods ( Guo et al. 2023 ; Wang et al. 2020 ) partially alleviate this but rely on precise masks and cannot generalize to broader lighting effects. Mask-free approaches such as ObjectDrop ( Winter et al. 2024 ) decouple removal and insertion into two steps, often leading to misaligned perspective or inconsistent lighting, while scene-level insertion ( Tarrés et al. 2024 ) allows broader context editing but lacks true movement capability. Recent progress in generative movement seeks to reposition objects directly through single-pass diffusion. Control paradigms include drag-based manipulation ( Mou et al. 2024a ; Shi et al. 2024 ; Avrahami et al. 2024 ; Mou et al. 2024b ) , motion-field guidance ( Geng and Owens 2024 ) , 3D-aware conditioning ( Michel et al. 2024 ; Pandey et al. 2024 ; Yenphraphai et al. 2024 ; Bhat et al. 2024 ) , and video diffusion-based sequence modeling ( Yu et al. 2025 ) . Relighting and Light Control. Image-based light editing has been widely explored in flash photography, addressing white balancing ( Hui et al.

Read the complete text