Hybrelighter: Combining Deep Anisotropic Diffusion and Scene Reconstruction for On-device Real-time Relighting in Mixed Reality

Hanwen Zhao, John Akers, Baback Elmieh, Ira Kemelmacher-Shlizerman · 2025

Read Hybrelighter: Combining Deep Anisotropic Diffusion and Scene Reconstruction for On-device Real-time Relighting in Mixed Reality by Hanwen Zhao, John Akers, Baback Elmieh, Ira Kemelmacher-Shlizerman (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2508-14930.txt.

Introduction

Hanwen Zhao and three co-authors present a fast method to relight mixed reality scenes on the device itself, in a 2025 paper. Relighting lets virtual changes in light play over real objects with true shadows. One use is viewing a room at different times of day when house hunting. Past deep learning methods run too slowly on current headsets.

Scene scans on the device are also rough, which hurts light quality. Simpler image filters miss depth and shadows. The new method joins image segmentation with light spread over a basic scene model. A diffusion step that favors certain directions carries the light. The method fixes scan errors as it goes. It reached speeds up to one hundred frames per second. The authors compare it with an industry standard and show a real estate demo.

Catalog record

CREATORHanwen Zhao, John Akers, Baback Elmieh, Ira Kemelmacher-Shlizerman
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE37 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2508-14930

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Hybrelighter: Combining Deep Anisotropic Diffusion and Scene Reconstruction for On-device Real-time Relighting in Mixed Reality Hanwen Zhao, John Akers, Baback Elmieh, Ira Kemelmacher-Shlizerman Abstract Mixed Reality scene relighting, where virtual changes to lighting conditions realistically interact with physical objects, producing authentic illumination and shadows, can be used in a variety of applications. One such application in real estate could be visualizing a room at different times of day and placing virtual light fixtures. Existing deep learning-based relighting techniques typically exceed the real-time performance capabilities of current MR devices. On the other hand, scene understanding methods, such as on-device scene reconstruction, often yield inaccurate results due to scanning limitations, in turn affecting relighting quality. Finally, simpler 2D image filter-based approaches cannot represent complex geometry and shadows. We introduce a novel method to integrate image segmentation, with lighting propagation via anisotropic diffusion on top of basic scene understanding, and the computational simplicity of filter-based techniques. Our approach corrects on-device scanning inaccuracies, delivering visually appealing and accurate relighting effects in real-time on edge devices, achieving speeds as high as 100 fps. We show a direct comparison between our method and the industry standard, and present a practical demonstration of our method in the aforementioned real estate example. \onlineid 1765 \vgtccategory Research \vgtcpapertype algorithm/technique \authorfooter Hanwen Zhao, University of Washington, Seattle, Washington, United States, [email protected] John Akers, UW Reality Lab, University of Washington, Seattle, Washington, United States Baback Elmieh, Computer Science, University of Washington, Seattle, Washington, United States, [email protected] Prof Ira Kemelmacher-Shlizerman, University of Washington , Seattle , Washington, United States \teaser Utilize the scene reconstruction capability and anisotropic diffusion (row 1) for real-time relighting of a room in various lighting conditions in mixed reality (row 2). Introduction keywords Relighting, Mixed Reality, Computer Vision Recent advancements in Mixed Reality (MR) technologies have significantly enhanced the capabilities of headsets, enabling them not only to visualize but also to comprehend real-world environments through passthrough cameras. This capability allows users to interact with their environments through a controllable virtual rendering layer, opening opportunities for various applications, notably scene relighting. By intelligently manipulating camera data, we can effectively alter the perceived lighting conditions of the environment, offering realistic simulations of diverse lighting scenarios without physically altering actual light sources. Scene relighting has numerous practical applications, including immersive integration of virtual objects with real-world scenes, creating visually compelling storytelling effects, or simulating environments under different lighting conditions for planning and visualization. Simple 2D filter-based relighting methods, such as adjustments to image temperature, contrast, or exposure, can be easily implemented with various image editing software packages. More advanced techniques involve masking-based image processing, which allows localized illumination changes, like simulating a spotlight effect. These learning-free approaches are highly efficient, stable, and predictable but suffer from limitations due to the absence of 3D geometric context, particularly when accurately representing complex shadows and occlusion interactions. In contrast, sophisticated 3D-based relighting methods utilize scene geometry derived from real-time reconstruction to achieve more accurate lighting simulations. These techniques leverage traditional rendering methods, such as rasterization or ray tracing, enabling more realistic representation of shadows and illumination interactions. However, current on-device scene reconstruction, typically performed using built-in lidar or depth sensors on MR headsets, often results in simplified meshes due to computational constraints and limited fidelity, adversely affecting the quality of relighting. More recent developments in deep learning have paved the way for highly detailed and realistic relighting solutions. Deep neural networks can infer intrinsic scene properties such as surface normals, roughness, albedo, and reflectance, greatly improving the accuracy and visual realism of relighting effects. However, these deep learning methods are typically computationally intensive, limiting their real-time applicability on edge devices such as MR headsets. In this paper, we propose a novel hybrid method that integrates the computational efficiency of filter-based approaches with the deeper semantic understanding provided by deep learning-guided anisotropic diffusion. Our approach specifically addresses the inaccuracies present in simplified meshes generated by real-time scene reconstruction. By leveraging RGB images captured from passthrough cameras, we employ a deep learning-based feature extractor to accurately identify high-frequency details and object boundaries. Subsequently, we apply anisotropic diffusion iteratively, guided by these learned features, to smoothly interpolate shading within object boundaries while maintaining sharp, precise edges. Recognizing the computational demands of conventional anisotropic diffusion, which typically require numerous iterations to achieve equilibrium, we introduce a cascaded diffusion strategy. This method capitalizes on the rapid propagation of gradients at lower image resolutions, progressively refining edge details through fewer iterations, significantly enhancing runtime efficiency. Figure 1 : An example of a synthetic scene being relit by a spotlight image filter rendered on an untextured version of the mesh data. Figure 2 : An example of the mesh quality directly from the LiDAR camera on the iPhone 16 Pro. The errors in such suboptimal mesh can significantly reduce the visual quality of the final relit image, especially around the edges. In short, the problem we are solving is: How can we relight a dynamically updating scene that is geometrically correct, and runs in real-time on device in Mixed Reality? And our primary contributions include: • A hybrid relighting approach that effectively combines deep learning feature extraction with filter-based anisotropic diffusion. • A modified anisotropic diffusion process optimized for fewer iterations, demonstrating that existing deep diffusion model architectures can be effectively adapted to run in real time on edge devices. • A practical demonstration and evaluation of our method on commercially available MR devices. We present how relighting can be practically applied in a real estate tour scenario. 1 Related Work Scene Reconstruction based Relighting Approaches Scene relighting techniques have historically relied on high-quality reconstructions as a basis for realistic lighting simulations. Early research demonstrated the potential of mesh-based methods to produce convincing relit images given sufficiently detailed geometry [ 9 ] . However, due to computational constraints, MR headsets typically employ mesh decimation strategies, resulting in reduced fidelity and diminished relighting quality. To address this limitation, several studies have proposed methods to incrementally refine meshes dynamically and efficiently. Hybrid voxel-octree fusion techniques [ 14 ] , and the Large Reconstruction Model (LRM) [ 22 ] , demonstrate that high-quality meshes can be rapidly generated even from sparse input views. Additionally, recent advancements in real-time dense scene reconstruction can achieve high-fidelity meshes at approximately 20 frames per second (fps) on powerful GPUs, although these methods remain computationally prohibitive for edge-device deployment [ 15 ] . Neural rendering methods, including Gaussian splatting and Neural Radiance Fields (NeRF), have also been explored for their impressive capabilities in large-scale scene relighting [ 16 ] [ 3 ] [ 26 ] . Despite achieving visually compelling results, these methods require extensive pre-training, ranging from several minutes to half an hour, thus limiting their practical usability in interactive MR scenarios. Other recent approaches seek to infer 3D scene information without explicitly generating 3D representations. Advances in depth estimation have enabled consistent depth image generation from extended video sequences, achieving high-performance processing speeds of approximately 9 ms per frame (over 100 fps) on high-end discrete GPUs [ 4 ] . Additionally, depth estimation methods tailored for edge devices have been developed, reaching speeds up to 300 fps on embedded Nvidia hardware, albeit with compromised image quality and

Read the complete text