Monocular Avatar Reconstruction via Cascaded Diffusion Priors and UV-Space Differentiable Shading

Hong Li, Minqi Meng, Yanjun Liang, Chongjie Ye, Houyuan Chen, Weiqing Xiao, Xianda Guo †, Guojun Lei, Xuhui Liu, Chaojie Yang, Yanlun Peng, Hao Zhao, Baochang Zhang ‡ · 2026

Read Monocular Avatar Reconstruction via Cascaded Diffusion Priors and UV-Space Differentiable Shading by Hong Li, Minqi Meng, Yanjun Liang, Chongjie Ye, Houyuan Chen, Weiqing Xiao, Xianda Guo †, Guojun Lei, Xuhui Liu, Chaojie Yang, Yanlun Peng, Hao Zhao, Baochang Zhang ‡ (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2606-28144.txt.

Introduction

Rebuilding a relightable 3D avatar from one casual photo is an ill-posed problem. PBR training data are scarce, and telling lighting apart from materials is hard. Hong Li and twelve colleagues present a data-efficient framework in 2026. It builds on a unified pretrained diffusion backbone. Cascaded low-rank adaptations handle each sub-task in UV space.

An inpainting module completes textures hidden by occlusion. A light-homogenisation module and a cross-intrinsic attention mechanism remove baked-in lighting. They build pixel-aligned maps for albedo, normal, roughness, specular, and displacement. A UV-space loss on shading keeps the result honest to physics during decomposition. The generative process must satisfy the rendering equation directly. Training used fewer than one hundred real 3D scans. The system still produces complete 4K PBR assets. Realism beats state-of-the-art methods in the reported tests.

Catalog record

CREATORHong Li, Minqi Meng, Yanjun Liang, Chongjie Ye, Houyuan Chen, Weiqing Xiao, Xianda Guo †, Guojun Lei, Xuhui Liu, Chaojie Yang, Yanlun Peng, Hao Zhao, Baochang Zhang ‡
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE64 KB
LICENSEhttp://creativecommons.org/licenses/by-sa/4.0/
RIGHTSCC BY SA
SOURCEarXiv

Source: https://arxiv.org/abs/2606.28144

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Monocular Avatar Reconstruction via Cascaded Diffusion Priors and UV-Space Differentiable Shading https://www.blender.org/ https://www.3dscanstore.com. https://github.com/fpgaminer/joycaption https://www.3dscanstore.com. Hong Li, Minqi Meng, Yanjun Liang, Chongjie Ye, Houyuan Chen, Weiqing Xiao, Xianda Guo †, Guojun Lei, Xuhui Liu, Chaojie Yang, Yanlun Peng, Hao Zhao, Baochang Zhang ‡ Abstract Reconstructing high-fidelity, relightable 3D avatars from a single in-the-wild image is a challenging ill-posed problem, primarily hindered by the scarcity of high-quality PBR data and the complexity of disentangling illumination from intrinsic materials. In this paper, we present a data-efficient framework that leverages the robust priors of a unified pre-trained diffusion backbone to sequentially address texture completion, delighting, and material decomposition. Unlike existing methods that rely on fragmented pipelines or extensive proprietary datasets, we utilize cascaded Low-Rank Adaptations (LoRAs) to adapt the strong generative prior of the diffusion model for each sub-task in UV space. Specifically, we first employ an Inpainting LoRA to complete missing UV textures caused by occlusion, leveraging the model’s semantic understanding to generate semantically and photometrically coherent details. Subsequently, a Light-Homogenization LoRA and a novel Cross-Intrinsic Attention mechanism are introduced to remove baked-in lighting and collaboratively synthesize pixel-aligned PBR maps (Albedo, Normal, Roughness, Specular, and Displacement). To ensure physical plausibility, we impose a UV-space differentiable BRDF shading loss during the decomposition stage, forcing the generative process to adhere to the rendering equation without the artifacts typical of rasterization-based supervision. Extensive experiments demonstrate that our method, trained on fewer than 100 real 3D scans, generates comprehensive, 4K-resolution PBR assets with superior realism and generalization compared to state-of-the-art methods, and all training code and model weights will be released upon acceptance. Figure 1 : High-Fidelity 3D Avatar Reconstruction . From a single input image, we reconstruct high-fidelity 3D geometry and PBR materials (albedo, normal, packed maps) to enable relightable avatar synthesis. As shown in the novel lighting results (right), our method accurately recovers fine details (e.g., wrinkles, moles) whilst maintaining identity consistency. 1 Introduction Photorealistic avatars are pivotal to immersive experiences in virtual reality, multimedia, and video games [ 67 , 60 ] . These personalized digital doubles serve as a crucial interface, bridging physical identities with digital environments. While Light Stage systems [ 15 ] have long established the gold standard for acquiring high fidelity avatars, their prohibitive cost and reliance on complex, studio based capture setups severely limit their accessibility for mass adoption. Therefore, monocular 3D face reconstruction, recovering geometry and texture from a single in-the-wild image, remains a fundamental challenge in computer vision. In this paper, we focus on relightable face and head PBR asset reconstruction rather than full-body, dynamic, or accessory-complete avatars. Traditional approaches rely mainly on 3D Morphable Models (3DMMs) [ 6 , 45 ] , which represent facial variations within a compact linear subspace. While robust, these linear models inherently struggle to capture high frequency details. To overcome this, recent research has shifted towards neural synthesis paradigms. Early works focused on texture completion [ 16 , 23 , 22 , 38 , 68 ] ; however, they typically produce textures with baked-in illumination, rendering the assets unsuitable for relighting in physically based rendering (PBR) pipelines. To achieve disentangled PBR reconstruction, recent methods typically train generative networks (GANs or Diffusion) to predict nonlinear texture bases [ 34 , 33 , 32 , 21 , 44 , 19 , 2 , 11 ] . However, these approaches face a critical data efficiency and generalization dilemma. First, high quality PBR training data is scarce: synthetic data often lacks realism, while real Light Stage scans are difficult to scale. Second, methodological bottlenecks persist: GAN-based methods suffer from mode collapse and over smoothing, while recent attempts to finetune diffusion models on small real datasets (e.g., UltraAvatar [ 72 ] ) often degrade the model’s prior knowledge, leading to poor generalization on in-the-wild images. Furthermore, most existing frameworks rely on differentiable rasterization with 2D screen-space supervision. This optimization paradigm is sensitive to occlusions (e.g., hair, glasses) present in input images, often inadvertently "baking" these artifacts into the reconstructed texture. In this work, we present a novel framework that addresses these challenges by synergizing the robust priors of large-scale generative models with a modern rendering workflow (Fig. ). Our core insight is that we can circumvent the need for massive proprietary datasets by effectively augmenting a compact set of high-quality scans via modern rendering engines, incorporating pose and occlusion priors derived from public in-the-wild face datasets. By combining these augmented physical priors with a pre-trained modern diffusion backbone, our method achieves high-fidelity material estimation without requiring extensive real-world data. Crucially, we enforce physically based rendering constraints directly on the estimated materials in UV space. This strategy ensures the physical consistency and high quality of the textures while mitigating the occlusion artifacts commonly observed in previous rasterization-based approaches. Our pipeline begins with a precise geometric alignment step. To support high-fidelity texture generation, we employ a robust geometry estimator enhanced by DINOv3 features, ensuring accurate UV mapping. With the geometry fixed, our core contribution lies in a unified diffusion-based texture framework adapted via cascaded LoRA modules. This backbone sequentially performs texture inpainting, light homogenization, and intrinsic material estimation. To ensure physical plausibility, we introduce a Cross-Intrinsic Attention mechanism that coordinates distinct material branches (Albedo, Normal, Roughness, Specular, Displacement), ensuring coherent high-frequency details. Trained on fewer than 100 real 3D scans, our model generates comprehensive, 4K-resolution PBR assets that generalize seamlessly to diverse in-the-wild images. In summary, our key contributions are: • We propose a data-efficient framework for high-fidelity 3D avatar reconstruction that achieves state-of-the-art quality using fewer than 100 real 3D scans, effectively solving the data scarcity bottleneck via a Blender-augmented synthetic data pipeline. • We design a unified diffusion-based texture pipeline featuring a novel Cross-Intrinsic Attention mechanism, which enforces structural coherence and physical plausibility across generated material modalities. • Leveraging generative priors and UV-space physical constraints, our method effectively avoids the pitfalls of direct 2D supervision, producing 4K-resolution PBR assets robust to occlusions and generalizing well to in-the-wild inputs. 2 Related Works 2.1 3D Geometry Reconstruction Monocular 3D face geometry reconstruction remains a fundamental challenge [ 61 ] . The dominant paradigm relies on parametric representations, utilizing 3DMMs [ 6 , 4 , 62 , 40 ] or blendshapes [ 41 ] to regress coefficients via deep learning [ 50 , 17 , 24 , 64 , 20 , 12 , 73 , 3 ] . While semantically controllable, these methods are inherently constrained by the linear subspace, failing to capture high-frequency details. To address this, HRN [ 36 ] proposes a hierarchical iterative refinement strategy but suffers from optimization instability (e.g., overfitting artifacts). Similarly, HiFace [ 9 ] attempts to enhance expressiveness by expanding the high-fidelity 3DMM basis, yet it still struggles to resolve intricate micro-structures like wrinkles and pores. The second category, non-linear reconstruction models, endeavors to surmount the inherent limitations of linear models in representing high-frequency geometric details. Instead of relying on fixed linear bases, these approaches employ DNNs, GNNs [ 59 , 58 , 1 , 47 , 19 ] , or Implicit Neural Representations (INRs) [ 14 , 69 , 70 ] to learn complex mappings directly from data. This enables the capture of subtle facial variations and intricate details like

Read the complete text