Learning a Delighting Prior for Facial Appearance Capture in the Wild 307 Journal: TOG Journal: TOG Volume: 45 4 46 7 DOI: 10.1145/3811303 CCS: Computing methodologies Rendering
Yuxuan Han, Xin Ming, Tianxiao Li, Zhuofan Shen, Qixuan Zhang, Lan Xu, Feng Xu · 2026
Read Learning a Delighting Prior for Facial Appearance Capture in the Wild 307 Journal: TOG Journal: TOG Volume: 45 4 46 7 DOI: 10.1145/3811303 CCS: Computing methodologies Rendering by Yuxuan Han, Xin Ming, Tianxiao Li, Zhuofan Shen, Qixuan Zhang, Lan Xu, Feng Xu (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2605-05636.txt.
Introduction
Capturing a face's true appearance from casual smartphone photos is hard, because unknown lighting tangles with skin reflectance. Yuxuan Han and six co-authors propose learning a delighting network, which strips illumination away, as a prior for the reconstruction. Studio light-stage systems do this well but are costly and immobile.
Their network trains on both a light-stage dataset and rendered scans. A modulation scheme conditions the network on learnable source tokens, separating each dataset's style from the shared physics of delighting. The learned prior then constrains inverse rendering on ordinary photos, disentangling reflectance from light more reliably.
Catalog record
| CREATOR | Yuxuan Han, Xin Ming, Tianxiao Li, Zhuofan Shen, Qixuan Zhang, Lan Xu, Feng Xu |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 53 KB |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2605.05636
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Learning a Delighting Prior for Facial Appearance Capture in the Wild 307 Journal: TOG Journal: TOG Volume: 45 4 46 7 DOI: 10.1145/3811303 CCS: Computing methodologies Rendering Yuxuan Han, Xin Ming, Tianxiao Li, Zhuofan Shen, Qixuan Zhang, Lan Xu, Feng Xu Abstract . High-quality facial appearance capture has traditionally required costly studio recording. Recent works consider an in-the-wild smartphone-based setup; however, their model-based inverse rendering paradigm struggles with the complex disentanglement of reflectance from unknown illumination. To bridge this gap, we propose to shift the paradigm into training a powerful delighting network as a prior to constrain the optimization. We leverage the OLAT dataset and the rendered Light Stage scans for training, and propose Dataset Latent Modulation (DLM) to seamlessly integrate these heterogeneous data sources. Specifically, by conditioning the core network on learnable source-aware tokens, we decouple dataset-specific styles from physical delighting principles, enabling the emergence of a delighting prior that outperforms existing proprietary models. This powerful delighting prior enables a simple and automatic appearance capture pipeline that achieves high-quality reflectance estimation from casual video inputs, outperforming prior arts by a large margin. Furthermore, we leverage our appearance capture method to transform the multi-view NeRSemble dataset into NeRSemble-Scan, a large-scale collection of 4K-resolution relightable scans. By open-sourcing our model and the NeRSemble-Scan dataset, we democratize high-end facial capture and provide a new foundation for the research community to build photorealistic digital humans. Figure 1. Given a smartphone sequence captured in the wild (4 sampled views shown here), our method reconstructs high-quality facial assets, which can be exported to graphics engines like Blender for photo-realistic rendering. As shown in the close-up above, our method faithfully reconstructs facial detail patterns around the lip, eyebrow, and cheek. 1. Introduction Facial appearance capture lies at the core of digital human creation, enabling applications across visual effects generation, interactive entertainment, and immersive virtual experiences. Although production-level results have been achieved under delicately controlled lighting using specialized studio systems ( Alexander et al. 2009 ; Alexander et al. 2013 ) , such setups are costly and inaccessible to everyday users. In contrast, capturing facial appearance in the wild, i.e. , from casually recorded smartphone videos under unconstrained and unknown illumination, remains fundamentally challenging. The difficulty stems from faithfully disentangling clean and relightable reflectance maps from complex real-world lighting effects. Recent years have witnessed a growing interest in facial appearance capture from unconstrained smartphone videos ( Xu et al. 2025 ; Han et al. 2026 ; Bharadwaj et al. 2023 ) . Early works adopt a model-based inverse rendering framework ( Xu et al. 2025 ; Bharadwaj et al. 2023 ) , which is constrained by oversimplified lighting models, struggling to resolve complex light transport inherent in most in-the-wild scenarios. To address this, a recent concurrent work, WildCap ( Han et al. 2026 ) , proposes a hybrid approach that performs model-based inverse rendering on top of delighted images predicted by a data-driven method, thereby improving reconstruction quality in various lighting conditions. However, WildCap necessitates manual efforts to isolate regions where the delighted images exhibit artifacts, thereby hindering its scalability for large-scale applications. In addition, WildCap relies on a specialized region-based inverse rendering to clean these artifacts, which also increases system complexity and results in a discontinuous texture map. Crucially, we also observe that a strong delighting prior can effectively regularize model-based inverse rendering; by internalizing accurate light transport into the prior, it is possible to achieve intervention-free delighting and stable per-scene optimization. Driven by this insight, we shift the focus of in-the-wild facial appearance capture toward learning a robust delighting prior that effectively disentangles diffuse albedo from a single-view in-the-wild face image. However, training such a powerful delighting prior is non-trivial. On the one hand, state-of-the-art solutions remain proprietary in both training data and model weights ( Kim et al. 2024 ; Yoon et al. 2024 ; Futschik et al. 2023 ) , which makes it challenging for the research community to reproduce or build upon their performance. While FaceOLAT ( Rao et al. 2025 ) , a recent open-source dataset captured by a Light Stage in the One-Light-At-a-Time (OLAT) mode ( Debevec et al. 2000 ) , has attempted to bridge this gap, its utility for training a powerful delighting prior is compromised by spatial blurring stemming from imperfect OLAT frame alignment and the lack of physically correct diffuse albedo for supervision. On the other hand, even the leading proprietary model, SwitchLight ( Kim et al. 2024 ) , struggles with complex lighting and leaves noticeable baking artifacts in predictions, which ultimately necessitate manual interventions and specialized inverse rendering in WildCap ( Han et al. 2026 ) . In this paper, we propose OpenDelight, a powerful and open-source delighting prior for facial appearance capture in the wild. On the data front, to address the inherent quality limitations of FaceOLAT ( Rao et al. 2025 ) , we first collect a dataset captured by a Light Stage in the material mode ( Ghosh et al. 2011 ) , comprising relightable scans of 120 subjects. Our scan dataset and FaceOLAT are highly complementary: scan-based renderings provide the necessary sharpness and physically correct diffuse albedo, while OLAT-integrated images contribute to closing the gap between synthetic and real-world images. However, training a delighting prior that seamlessly inherits the strengths of such heterogeneous data sources remains a challenging task. If handled naively, the network tends to settle for a compromised solution. To resolve this, our key insight is to decompose the knowledge within these heterogeneous datasets into shared mechanisms (the high-level foundation of delighting) and dataset-specific styles (color distributions, degree of physical correctness, and spatial sharpness). Building on this insight, we propose a Dataset Latent Modulation (DLM) technique, which employs learnable source-aware tokens as additional conditions to the network. Given that the parameter count of these tokens is negligible compared to the core network, simply end-to-end training encourages a natural emergence of disentanglement: the tokens absorb the styles of each dataset, while the core network distills the principles of facial delighting. At inference time, we steer the network toward sharp and physically correct predictions by conditioning it on the tokens associated with our scan dataset. Thanks to the meticulously constructed dataset and effective training techniques, our model achieves significantly better results than the leading proprietary model SwitchLight ( Kim et al. 2024 ) , while keeping the method simple. With the powerful delighting prior at hand, we further unlock its potential to empower facial appearance capture in the wild. Given smartphone-captured multi-view images, we utilize OpenDelight to estimate diffuse albedo across views, which are then fused into a unified texture. This clean texture base enables performing conventional model-based inverse rendering within a patch-level diffusion prior ( Han et al. 2025 ; Han et al. 2026 ) in a more stable way. Compared to WildCap, our method not only eliminates the need for manual intervention to correct network artifacts but also removes its complex and specialized inverse rendering process, while achieving on par or better quality, demonstrating that a superior delighting prior can fundamentally simplify the appearance capture problem. Finally, leveraging our automatic appearance capture method, we transform an existing multi-view face dataset, NeRSemble ( Kirschstein et al. 2023 ) ,