PercHead: Perceptual Head Model for Single-Image 3D Head Reconstruction & Editing

Antonio Oroz, Matthias Nießner, Tobias Kirschstein · 2025

Read PercHead: Perceptual Head Model for Single-Image 3D Head Reconstruction & Editing by Antonio Oroz, Matthias Nießner, Tobias Kirschstein (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2511-02777.txt.

Introduction

Antonio Oroz, Matthias Niessner, and Tobias Kirschstein present PercHead, a model that rebuilds a 3D head from one photo and allows separate edits to shape and look. Both tasks are ambiguous, since many heads could explain the same image. Their 2025 paper rests on a new perceptual loss built from the vision models DINOv2 and SAM 2.1.

Unlike pixel-level losses, the new loss judges images by deep visual understanding, giving cleaner guidance in fine detail areas. It can replace standard losses directly. The model uses vision transformers, trains on multi-view and in-the-wild photos, and reaches state-of-the-art quality on new views, even at extreme angles. For editing, a segmentation map steers shape while a prompt or photo sets appearance, all through an interactive interface.

Catalog record

CREATORAntonio Oroz, Matthias Nießner, Tobias Kirschstein
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE35 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2511-02777

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

PercHead: Perceptual Head Model for Single-Image 3D Head Reconstruction & Editing Antonio Oroz, Matthias Nießner, Tobias Kirschstein Abstract We present PercHead, a method for single-image 3D head reconstruction and semantic 3D editing — two tasks that are inherently challenging due to severe view occlusions, weak perceptual supervision, and the ambiguity of editing in 3D space. We develop a unified base model for reconstructing view-consistent 3D heads from a single input image. The model employs a dual-branch encoder followed by a ViT-based decoder that lifts 2D features into 3D space through iterative cross-attention. Rendering is performed using Gaussian Splatting. At the heart of our approach is a novel perceptual supervision strategy based on DINOv2 Oquab et al. 2023 and SAM2.1 Ravi et al. 2024 , which provides rich, generalized signals for both geometric and appearance fidelity. Our model achieves state-of-the-art performance in novel-view synthesis and, furthermore, exhibits exceptional robustness to extreme viewing angles compared to established baselines. Furthermore, this base model can be seamlessly extended for semantic 3D editing by swapping the encoder and fine-tuning the network. In this variant, we disentangle geometry and style through two distinct input modalities: a segmentation map to control geometry and either a text prompt or a reference image to specify appearance. We highlight the intuitive and powerful 3D editing capabilities of our model through a lightweight, interactive GUI, where users can effortlessly sculpt geometry by drawing segmentation maps and stylize appearance via natural language or image prompts. Project Website: https://antoniooroz.github.io/PercHead/ 1 Introduction Accurate reconstruction of 3D head geometry from a single image, while maintaining robust consistency across arbitrary input and target viewpoints, is a crucial yet unsolved challenge. Success in this task would enable highly realistic and immersive applications, including virtual conferencing, gaming, and personalized retail experiences, as well as open the door to advanced capabilities such as camera control over portraits, 3D virtual try-on, and editing that disentangles geometry from style. Faithful 3D head reconstruction from a single image is an inherently under-constrained problem, as many plausible 3D shapes can explain the same 2D observation. While learning strong priors from large datasets is critical to resolve this ambiguity, high-quality multi-view datasets such as NeRSemble Kirschstein et al. 2023 and Ava-256 Martinez et al. 2024 remain limited in size, and large-scale datasets like FFHQ Karras et al. 2019 and VFHQ Xie et al. 2022 provide only single-view 2D supervision or limited angle-variety. As a consequence, existing methods tend to perform well for near-frontal views or viewpoints close to the input, but deteriorate in identity fidelity or geometric realism under large viewpoint deviations. Furthermore, humans are exceptionally sensitive to even small errors in facial structure and appearance, demanding extremely high visual fidelity from reconstructed 3D heads. Conventional supervision strategies based on reconstruction or perceptual losses often fall short of achieving the required realism, especially for high-frequency or occluded regions. Finally, downstream editing tasks that aim to disentangle style (e.g., hair, skin tone) from geometry (e.g., facial structure) are inherently ambiguous, as multiple plausible outputs can explain the same input. Existing methods address the challenges of 3D head reconstruction through various forms of prior constraints and supervision. Early approaches such as ROME Khakhulin et al. 2022 relied on strong geometric constraints via mesh-based 3D Morphable Models (3DMMs), limiting realism and expressiveness. GAN-based methods Chan et al. 2021 ; An et al. 2023 leveraged large-scale 2D datasets and adversarial losses to synthesize realistic outputs, but often struggle with identity preservation and robustness under extreme viewpoints. GAGAvatar Chu and Harada 2024 introduced a dual-lifting strategy trained on multi-frame video data, which improves reconstruction quality near the input view but fails to generalize well to more diverse angles. The Large Avatar Model (LAM) He et al. 2025 employs multi-frame video supervision within a transformer-based architecture, achieving strong frontal reconstructions but still exhibiting degradation for views far from the input. LGM Tang et al. 2024 employs a multi-view diffusion prior, but still falls short of the realism required for high-quality human head synthesis. To overcome these limitations, we propose a novel architecture combining real multi-view data from Nersemble Kirschstein et al. 2023 , artificial multi-view head data from Cafca Buehler et al. 2024 , and diverse single-view images from FFHQ Karras et al. 2019 to achieve the best trade-off between diversity and strong 3D consistency. Our approach employs a dual-branch encoder integrating intermediate DINOv2 features Oquab et al. 2023 ; He et al. 2025 with a task-specific Vision Transformer (ViT) Dosovitskiy et al. 2020 . Our view-agnostic ViT-Decoder exclusively cross-attends to 2D features, lifting their information into 3D regions. Critically, to train our 3D deocder pipeline, our method replaces conventional perceptual and reconstruction losses, such as LPIPS, SSIM, or L1, with the supervision of powerful foundational models: DINOv2 and SAM2.1 Ravi et al. 2024 . Surprisingly, this departure from traditional objectives leads to more robust and generalized training signals, ultimately enhancing both visual fidelity and 3D consistency. Our method significantly surpasses state-of-the-art approaches, such as PanoHead, GAGAvatar, LGM and LAM, particularly in reconstruction tasks involving extreme viewpoints. Furthermore, our generalized 3D reconstruction model serves as a versatile backbone that can be extended to tasks such as 3D editing. By replacing the encoder and fine-tuning the model with segmentation maps Zheng et al. 2021 and CLIP embeddings Radford et al. 2021 as inputs, we enable advanced editing capabilities that disentangle geometric control from style modulation. In summary, our contributions include: • A novel architecture achieving superior single-image 3D head reconstruction, especially at challenging viewing angles. • A perceptual loss formulation based on generalized vision models DINOv2 and SAM2.1, significantly improving visual quality. • Advanced 3D face editing capabilities are enabled by fine-tuning our strong, generalized base model on FARL segmentation maps and CLIP embeddings, demonstrating both its versatility in handling diverse inputs and its transferability to novel downstream tasks. 2 Related Work 2.1 Single-Image 3D Face Reconstruction Early single-image 3D reconstruction methods used mesh-based models Wang et al. 2022a ; Bai et al. 2023 ; Khakhulin et al. 2022 ; Feng et al. 2021 ; Lattas et al. 2020 ; Lattas et al. 2021 ; Lin et al. 2020 ; Gecer et al. 2019 ; Gecer et al. 2021 , offering efficient geometry but limited detail and flexibility - such as hair. GAN-based refinements Gecer et al. 2019 ; Gecer et al. 2021 improve realism but still inherit the mesh constraints. Early NeRF-based GANs Chan et al. 2020 ; Or-El et al. 2022 ; Gu et al. 2022 achieved high-quality 3D head synthesis but were slow. EG3D Chan et al. 2021 introduced tri-plane representations to improve efficiency, enabling faster rendering with great visual quality. PanoHead An et al. 2023 extended this to 360° head generation. Nonetheless, these GAN-based methods require computationally expensive latent inversion Roich et al. 2021 for 3D head reconstruction. Encoder-based variants Bhattarai et al. 2024 improve inference speed but typically reduce reconstruction fidelity. Most GAN-based methods are also trained on 2D data and lack strong multi-view supervision, limiting 3D consistency. Portrait4D Deng et al. 2024a and Portrait4Dv2 Deng et al. 2024b address this by introducing multi-view supervision, resulting in more consistent geometry. Nonetheless, NeRFs remain computationally heavy and can still suffer from view-dependent variability. With the emergence of

Read the complete text