PIXLRelight: Controllable Relighting via Intrinsic Conditioning

Miguel Farinha, Ronald Clark · 2026

Read PIXLRelight: Controllable Relighting via Intrinsic Conditioning by Miguel Farinha, Ronald Clark (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2605-18735.txt.

Introduction

Miguel Farinha and Ronald Clark present PIXLRelight, a 2026 method for controllable single-image relighting. Older methods give limited lighting control or need costly per-image optimization. Their idea bridges physically based rendering and learned synthesis through shared intrinsic conditioning. Training photos split into albedo, diffuse shading, and residuals that condition the model. At run time, the same conditioning comes from a path-traced render of a coarse 3D rebuild. A transformer renderer applies the target light, keeping fine detail via per-pixel affine modulation.

Catalog record

CREATORMiguel Farinha, Ronald Clark
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE44313
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2605.18735

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

PIXLRelight: Controllable Relighting via Intrinsic Conditioning Miguel Farinha, Ronald Clark Abstract We present PIXLRelight , a feed-forward approach for physically controllable single-image relighting. Existing methods either provide limited lighting control (e.g. through text or environment maps), accumulate errors when chaining inverse and forward rendering, or require costly per-image optimization. Our key idea is to bridge physically based rendering (PBR) and learned image synthesis through a shared intrinsic conditioning that can be obtained from either real photographs or PBR renders. At training time, paired multi-illumination photographs are decomposed into albedo, diffuse shading, and non-diffuse residuals, which condition the model. At inference time, the same conditioning is computed from a path-traced render of a coarse 3D reconstruction of the input under user-specified PBR lights. A transformer-based neural renderer then applies the target illumination to the source photograph, preserving fine image detail through a per-pixel affine modulation. PIXLRelight enables arbitrary PBR-style lighting control, achieves state-of-the-art relighting quality, and runs in under a tenth of a second per image. Code and models are available at https://mlfarinha.github.io/pixl-relight/. Figure 1: PIXLRelight is a feed-forward transformer that produces photorealistic relightings of in-the-wild photographs in a single pass. The user authors the target illumination in a physically based renderer, with full control over light type, position, color, and intensity. The model is conditioned on the resulting intrinsic decomposition of the target appearance – albedo, diffuse shading, and a non-diffuse residual. Above, the same source image is relit under six different illuminations. 1 Introduction Illumination is a primary component of image formation. In computer graphics, physically based rendering (PBR) engines such as Blender [ 3 ] and Unreal Engine [ 10 ] expose lighting controls directly: an artist places point, area, directional, environmental, or emissive light sources in a 3D scene, and a path tracer simulates their interaction with geometry and materials. Bringing this same physical control to in-the-wild photographs would unlock applications across computational photography, content creation, and visual effects – but is fundamentally harder, since recovering the geometry, materials, and light transport from an image is challenging. Recent approaches address this gap in three ways, none of which jointly enables physical control, photorealism, and speed. A first line conditions relighting on HDR environment maps [ 18 , 49 ] , reference images [ 47 ] , or screen-space scribbles [ 7 ] and masks [ 33 ] ; these interfaces struggle to express spatially localized, multi-source illumination. A second line treats relighting as an inverse-then-forward rendering pipeline that estimates G-buffers and re-renders them under a target illumination [ 50 , 28 , 40 , 11 ] ; the intermediate buffers cannot encode every cue the renderer needs (e.g. transparency, subsurface scattering), and errors compound across the two stages [ 14 ] . The third, and closest, line combines PBR with a neural renderer: 5 reconstruct an approximate textured mesh from a photograph, ray-trace it under user-specified illumination, and pass the CG render to a neural renderer that produces the final image. Two limitations follow. First, because the mesh is reconstructed with diffuse reflectance only, the ray-traced render is diffuse-only, and the network must guess every specular highlight, transparent surface, or refractive cue in the output. Second, training each input pair requires a per-image differentiable-rendering optimization that fits a 3D lighting environment to the source image; this optimization is stable only for a narrow lighting parameterization, bounding the lighting distribution the network sees at training. The natural alternative is to decompose the photograph into geometry and materials, place new lights in PBR, and re-render. This is exactly what computer graphics does well – given high-quality assets. For in-the-wild photographs, those assets are precisely what we cannot recover reliably: single-image geometry is coarse and material decomposition is under-constrained, so a path-traced render of such a reconstruction is riddled with artifacts. Our key insight is to harness both the advantages of PBR (controllability) and neural rendering (detailed photorealism). Our insight is to use each tool for what it does well: PBR for specifying what the target lighting should be, and a feed-forward neural model trained on real photographs for how to apply that lighting to the photograph photorealistically – absorbing the imperfections of the underlying scene reconstruction in the process. The two are bridged by an intrinsic decomposition of the target appearance into albedo, diffuse shading, and a non-diffuse residual [ 20 ] . The same input is produced from a real photograph at training and from a PBR render at inference. At training, paired captures from existing multi-illumination datasets [ 34 , 25 , 15 ] pass through a frozen intrinsic decomposition model [ 20 ] to directly supervise relighting which means there is no inverse-then-forward chain to train or per-image rendering optimization to run. At inference, the conditioning is produced by a path-traced render of a coarse 3D reconstruction in which the user freely controls lighting using arbitrary combinations of physically-based lights. Crucially, the model never sees the rendered RGB image; it sees only the intrinsic buffers, which carry a precise cue for the desired output lighting. We call this approach PIXLRelight , a feed-forward transformer that consumes a source image and the target intrinsics and predicts a relit RGB image with per-pixel detail. Following recent feed-forward dense-prediction architectures [ 42 , 16 , 29 , 17 ] , we tokenize the two inputs with asymmetric encoders – a ViT [ 9 ] for the source image and a ConvNeXt [ 31 ] for the smoother, lower-frequency intrinsic stack – and fuse them in a shared transformer trunk read out by a DPT head [ 35 ] . Rather than regressing RGB directly, the head produces an identity-initialized per-pixel affine modulation of the source: at initialization the network reproduces the input exactly, and during training it learns only the residual lighting transformation, preserving photorealistic detail by construction. In summary, our contributions are: (1) a target-appearance intrinsic decomposition as a single conditioning interface for single-image relighting, computed from a real image at training and from a PBR render at inference; (2) a direct training strategy supervised end-to-end on real multi-illumination photographs, without an inverse-then-forward rendering pipeline or per-image rendering optimization; and (3) PIXLRelight , which achieves state-of-the-art relighting quality while running in under a tenth of a second per image. 2 Related work Single-image relighting methods can be organized by the modality through which the user specifies the target lighting. Text and reference-image conditioning [ 51 , 47 ] hallucinate plausible illumination but offer no physical control. Environment-map conditioning [ 18 , 49 , 28 , 14 ] assumes lighting comes from infinity and cannot express the near-field sources common in indoor scenes. Parametric conditioning on a single light source placed in a reference view, as in GenLit [ 2 ] and SyncLight [ 38 ] , restores spatial control but is not designed for multi-light, area-light, or emissive-geometry edits. Pixel-aligned intrinsic conditioning sidesteps both limitations: RGB \leftrightarrow X [ 50 ] , Ouroboros [ 40 ] , and V-RGBX [ 11 ] drive diffusion-based renderers from intrinsic buffers that include shading, but stop short of CG-style authoring. Closest to ours, 5 achieve Blender-authored physical control by routing a PBR render through a feed-forward neural renderer, but their pipeline

Read the complete text