GeoRelight: Learning Joint Geometrical Relighting and Reconstruction with Flexible Multi-Modal Diffusion Transformers
Yuxuan Xue, Ruofan Liang, Egor Zakharov, Timur Bagautdinov, Chen Cao, [0.25em] Giljoo Nam, Shunsuke Saito, Gerard Pons-Moll, Javier Romero, [0.5em] Codec Avatars Lab, Meta, [0.25em] Max Planck Institute for Informatics, Saarland Informatics Campus · 2026
Read GeoRelight: Learning Joint Geometrical Relighting and Reconstruction with Flexible Multi-Modal Diffusion Transformers by Yuxuan Xue, Ruofan Liang, Egor Zakharov, Timur Bagautdinov, Chen Cao, [0.25em] Giljoo Nam, Shunsuke Saito, Gerard Pons-Moll, Javier Romero, [0.5em] Codec Avatars Lab, Meta, [0.25em] Max Planck Institute for Informatics, Saarland Informatics Campus (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2604-20715.txt.
Introduction
Xue and eight colleagues propose GeoRelight, a Multi-Modal Diffusion Transformer that solves relighting and 3D shape estimation from a single photo. Relighting from one photo is ill-posed because a 2D image mixes shape, appearance, and light. Current methods use pipelines that pile up errors or ignore shape during relighting. GeoRelight works through isotropic NDC-Orthographic Depth, a distortion-free 3D model for diffusion models, and mixed synthetic and auto-labeled real data training. Solving shape and relighting jointly, GeoRelight beats step-by-step models and previous systems.
Catalog record
| CREATOR | Yuxuan Xue, Ruofan Liang, Egor Zakharov, Timur Bagautdinov, Chen Cao, [0.25em] Giljoo Nam, Shunsuke Saito, Gerard Pons-Moll, Javier Romero, [0.5em] Codec Avatars Lab, Meta, [0.25em] Max Planck Institute for Informatics, Saarland Informatics Campus |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 63215 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2604.20715
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
GeoRelight: Learning Joint Geometrical Relighting and Reconstruction with Flexible Multi-Modal Diffusion Transformers Yuxuan Xue, Ruofan Liang, Egor Zakharov, Timur Bagautdinov, Chen Cao, [0.25em] Giljoo Nam, Shunsuke Saito, Gerard Pons-Moll, Javier Romero, [0.5em] Codec Avatars Lab, Meta, [0.25em] Max Planck Institute for Informatics, Saarland Informatics Campus Abstract Relighting a person from a single photo is an attractive but ill-posed task, as a 2D image ambiguously entangles 3D geometry, intrinsic appearance, and illumination. Current methods either use sequential pipelines that suffer from error accumulation, or they do not explicitly leverage 3D geometry during relighting, which limits physical consistency. Since relighting and estimation of 3D geometry are mutually beneficial tasks, we propose a unified Multi-Modal Diffusion Transformer (DiT) that jointly solves for both: GeoRelight . We make this possible through two key technical contributions: isotropic NDC-Orthographic Depth (iNOD), a distortion-free 3D representation compatible with latent diffusion models; and a strategic mixed-data training method that combines synthetic and auto-labeled real data. By solving geometry and relighting jointly, GeoRelight achieves state-of-the-art results in photorealistic relighting, as well as high-fidelity 3D reconstruction and intrinsic estimation from a single image. Project page: https://yuxuan-xue.com/georelight 1 Introduction Altering illumination on a person from a single image is a core problem for computer graphics and vision Pandey et al. 2021 ; Kim et al. 2024 ; Chaturvedi et al. 2025 ; Mei et al. 2025 . It enables enormous applications, from creative editing and computational photography to virtual reality and film production. However, the task is profoundly ill-posed. A single 2D image is an ambiguous projection of three distinct and entangled factors: the 3D geometry of the subject, their intrinsic appearance like base color, and illumination of the scene. To realistically change the light, a model must, explicitly or implicitly, disentangle these components Liang et al. 2025 ; Zeng et al. 2024b . Recent learning-based approaches often fall into two categories, each with a specific drawback. First, end-to-end "translator" models Chaturvedi et al. 2025 ; Mei et al. 2025 ; He et al. 2025 ; Bharadwaj et al. 2025 learn a direct pixel-to-pixel mapping. This approach is often limited in physical plausibility due to the absence of geometric modeling, failing to produce realistic shadows or highlights that are consistent with the subject’s 3D shape. Second, and more common, are sequential pipelines Pandey et al. 2021 ; Kim et al. 2024 ; Liang et al. 2025 ; Zeng et al. 2024b . These methods first predict a set of intermediate buffers like albedo and surface normals which are passed to a separate neural rendering module to synthesize the final image. The critical flaw in this sequential design is error accumulation He et al. 2025 : any inaccuracies in the initial geometry estimation are baked into the buffers and cannot be corrected by the renderer. In this paper, we argue that relighting and geometry reconstruction are not separate sequential tasks, but are mutually beneficial and must be solved jointly . This synergy is two-fold. First, accurate 3D geometry is essential for the relighting model to render physically-plausible local shading and, crucially, to form correct cast shadows. Conversely, the appearance of shading in the image provides powerful shape-from-shading cues that can refine the underlying geometry estimation. We introduce GeoRelight , a unified generative framework that first embodies this joint approach. At its core is a Multi-Modal Diffusion Transformer (DiT) Peebles and Xie 2022 that simultaneously denoises both the relit image and the 3D geometry, allowing information to flow between both tasks throughout the generation process This joint generative approach, however, introduces two significant technical challenges. First, 3D geometry must be made compatible with a latent diffusion model that uses a pretrained 2D VAE to enable high-resolution generation. Standard representations like point maps become extremely noisy after lossy compression into VAE latent space, while traditional depth maps require normalization that anisotropically distorts the 3D shape. To solve this, we propose isotropic NDC-Orthographic Depth (iNOD) (Sec. 3.3 ), a novel representation that isotropically scales the 3D shape before projection, preserving the geometry in a “VAE-friendly” image format without distortion. Second, this model requires data that is both photorealistic and has accurate 3D geometry labels, which is difficult to acquire. We observe that synthetic trained model is robust to produce good intrinsics and geometry. Hence use our own model to annotate pseudo labels for photorealistic images from light stage and in-the-wild data. Then, we use the strategic mixed-data training procedure (Sec. 3.4 ) where we leverage our model’s flexibility to train on a mix of synthetic data (for perfect labels) and real-world data (for photorealism). Our unified multi-modal DiT model achieves state-of-the-art results in both photorealistic relighting and high-fidelity 3D reconstruction from a single in-the-wild image. In summary, our contributions are: • A unified Multi-Modal Diffusion Transformer framework that jointly generates relit images, intrinsic color, surface normal details, and 3D geometry. • iNOD, a novel and VAE-friendly depth representation that isotropically preserves 3D geometry in latent space. • A strategic mixed-data training methodology that combines synthetic data with auto-labeled real-world images to achieve both geometric accuracy and photorealism. 2 Related Works Image relighting and its related problems (e.g. intrinsic decomposition) have a long tradition in computer vision, going back to the work on intrinsic images by Barrow Barrow et al. 1978 and the Retinex algorithm by Land Land and McCann 1971 . We encourage the reader to read recent surveys Garces et al. 2022 ; Zhu et al. 2025 for a wide viewpoint on intrinsic decomposition and human relighting, while we focus on the latest work on relighting in the remainder. 2.1 Neural Rendering and Relighting Recent generative models have approached monocular image relighting in several ways. NeuralGaffer Jin et al. 2024 treats relighting as an end-to-end 2D diffusion process, finetuning a model on a large synthetic dataset to relight any object without explicit intrinsic decomposition. IC-Light Zhang et al. 2025 focuses on illumination harmonization, using diffusion to relight a foreground object to match a new background. LightLab Magar et al. 2025 enables direct manipulation of existing light sources in an image through diffusion models. To overcome limitations from above methods, two-stage pipelines like Total Relighting Pandey et al. 2021 first estimates intrinsics (e.g., albedo, normals) and then render the image with a neural shading module. This concept has been adapted in RGB-X Zeng et al. 2024b and DiffusionRenderer Liang et al. 2025 , by using separate diffusion models for forward and inverse rendering. Other methods like DiLightNet Zeng et al. 2024a use coarse geometry estimates to create "radiance hints" that guide a ControlNet-based diffusion model. Careaga and Aksoy Careaga and Aksoy 2025 leverage physical intrinsic decomposition for controllable relighting of photographs. For video, Lux Post Facto Mei et al. 2025 also uses a two-stage pipeline, addressing temporal consistency by finetuning a video diffusion model on a hybrid of static OLAT (One-Light-At-a-Time) data and in-the-wild videos. A primary drawback of sequential pipelines is their susceptibility to error accumulation, where inaccuracies from the intrinsic estimation stage are amplified by the renderer. The advantage of estimating geometry and intrinsics was already raised by the seminal work by Barron et al. Barron and Malik 2014 in the context of intrinsic image decomposition, without considering relighting yet. More recently, UniRelight