Pixel3DMM: Versatile Screen-Space Priors for Single-Image 3D Face Reconstruction

Simon Giebenhain, Tobias Kirschstein, Martin Rünz, Lourdes Agapito, Matthias Nießner · 2025

Read Pixel3DMM: Versatile Screen-Space Priors for Single-Image 3D Face Reconstruction by Simon Giebenhain, Tobias Kirschstein, Martin Rünz, Lourdes Agapito, Matthias Nießner (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2505-00615.txt.

Introduction

Simon Giebenhain and co-authors rebuild 3D faces from a single photo using per-pixel predictions in a 2025 paper. Fitting a morphable face model to one image is underdetermined: many faces could explain the same pixels. Strong learned priors are needed to pin the shape down.

Their system, Pixel3DMM, uses vision transformers built on a foundation model's features to predict surface normals and texture coordinates for every pixel. Those cues then guide a fitting routine that solves for the face model's parameters. Training drew on three high-quality 3D face data sets, over a thousand people and nearly a million images. A new benchmark tests posed and neutral expressions across angles and ethnicities. The method beat top rivals by over fifteen percent on posed faces.

Catalog record

CREATORSimon Giebenhain, Tobias Kirschstein, Martin Rünz, Lourdes Agapito, Matthias Nießner
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE34 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2505-00615

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Pixel3DMM: Versatile Screen-Space Priors for Single-Image 3D Face Reconstruction Simon Giebenhain, Tobias Kirschstein, Martin Rünz, Lourdes Agapito, Matthias Nießner Abstract We address the 3D reconstruction of human faces from a single RGB image. To this end, we propose Pixel3DMM, a set of highly-generalized vision transformers which predict per-pixel geometric cues in order to constrain the optimization of a 3D morphable face model (3DMM). We exploit the latent features of the DINO foundation model, and introduce a tailored surface normal and uv-coordinate prediction head. We train our model by registering three high-quality 3D face datasets against the FLAME mesh topology, which results in a total of over 1,000 identities and 976K images. For 3D face reconstruction, we propose a FLAME fitting opitmization that solves for the 3DMM parameters from the uv-coordinate and normal estimates. To evaluate our method, we introduce a new benchmark for single-image face reconstruction, which features high diversity facial expressions, viewing angles, and ethnicities. Crucially, our benchmark is the first to evaluate both posed and neutral facial geometry. Ultimately, our method outperforms the most competitive baselines by over 15% in terms of geometric accuracy for posed facial expressions. Figure 1 : We present Pixel3DMM, a set of two ViTs [ 9 ] , which are tailored to predict per-pixel surface normals and uv-coordinates. Here, we demonstrate the fidelity and robustness of our prior networks on examples from the FFHQ [ 20 ] dataset. From top to bottom we show input RGB, predicted surface normals, 2D vertices extracted from the uv-coordinate prediction, and our FLAME fitting results. † † footnotetext: Project Page: https://simongiebenhain.github.io/pixel3dmm 1 Introduction 3D reconstruction of faces, tracking facial movements, and ultimately extracting expressions for animation tasks are fundamental problems in many domains such as computer games, movie production, telecommunication, and AR/VR applications. Recovering 3D head geometry from a single image is a particularly important task due to the vast amount of available image collections. Unfortunately, reconstructing faces from a single input image is also inherently under-constrained. Not only depth ambiguity renders this task challenging, but also ambiguities between albedo and lighting/shadow effects. In addition, properly disentangling identity and expression information – which is critical for many downstream applications – makes the problem difficult. Finally, occlusions and unobserved facial regions further complicate the problem in real application scenarios, thus highlighting the need for strong data priors. A typical approach to address single-image face reconstruction is to exploit 3D parametric head models (3DMMs) [ 3 , 26 ] which provide a comparatively low-dimensional parametric representation for the underlying 3D geometry. Optimizing within a 3DMM’s disentangled parameter space heavily constrains the search space with built-in assumptions about plausible facial structure and expressions, and allows to extract disentangled identity and expression information. Nonetheless, despite relying on 3DMMs, many ambiguities remain and their simplifying assumptions about our world often cannot explain the complexity of an observed RGB signal. This necessitates additional priors in order to obtain compelling fitting results such as sparse [ 37 ] and dense [ 5 , 47 ] facial landmarks, or UV coodinate predictions [ 41 ] In recent years, we have also seen significant progress in feed-forward 3DMM regressors [ 38 , 10 , 8 , 34 , 53 , 50 ] . However, it is complicated to extend feed-forward regressors, e.g . to a multi-view or temporal domain, and, as we will show later, they fall behind optimization-based approaches on inputs with strong facial expressions. Overall, accurate 3D face reconstruction from single images remains a challenging and highly relevant problem. Therefore, we propose Pixel3DMM, a novel optimization-based 3D face reconstruction approach. Our main idea is to exploit and further develop broadly generalized and powerful foundation models to predict pixel-aligned geometric cues that effectively constrain the 3D state of an observed face. Given a single image at test time, we propose normal and uv-coordinate predictions as optimization constraints from which we fit a 3D FLAME model. Instead of a simple rendering loss of uv-coordinates, we then transfer the information into a 2D vertex loss, which offers a wider basin of attraction during optimization. We argue that this strategy is superior to traditional photometric terms, or sparse landmarks, which often struggle with extreme view points and facial expressions. In order to train our approach, we unify three recent, high-fidelity 3D face datasets [ 13 , 52 , 29 ] by registering them against the FLAME [ 26 ] model.Our approach outperforms all available normal estimators for human faces in the NeRSemble [ 24 ] dataset. In order to advance the evaluation of single-image 3D face reconstruction methods, we further propose a new benchmark based on the multi-view video dataset NeRSemble [ 24 ] , which includes a wider variety of facial expressions than existing benchmarks [ 38 , 52 , 11 , 6 ] . Our benchmark is the first to allow for the simultaneous evaluation of posed and neutral facial geometry. This enables a more direct comparison of methods, especially regarding fitting fidelity and ability to disentangle expression and identity information. Finally, we show that compared to our strongest baselines, our approach improves the L2-Chamfer reconstructions loss by over 15% for posed geometry, while slightly improving over neutral geometry predictions. To summarize, our main contributions are as follows: • A new formulation to exploit foundation model features for 3D-related, pixel-aligned predictions, facilitating state-of-the-art normal estimations for human faces. • A novel 3D face reconstruction approach based on predicted uv-map correspondences and surface normals. • A 3D face reconstruction benchmark and evaluation protocol from high-fidelity multi-view face captures. We plan to make the model, code, and our new benchmark publicly available to promote progress in single image 3D face reconstruction and encourage quantitative benchmarking on challenging facial expressions. Figure 2 : Method Overview: Pixel3DMM consists of (a) learning pixel-aligned geometric priors (left) and (b) test-time optimization against predicted uv-coordinates and normals (right). On the left we illustrate our network architecture and examples from the training set. On the right we illustrate to process of finding per-vertex 2D locations using a nearest neighbor (N.N.) look up, and our loss terms. 2 Related Work Single-Image 3DMM Fitting Tracking morphable models from single images is a well-studied problem in the context of 3D face reconstruction and tracking. Early works [ 2 , 31 , 26 ] , introduced statistical shape and texture priors to estimate 3D face geometry from 2D images. Such methods rely on photometric fitting and subsequent approaches improve modeling capabilities using learned implicit representations [ 27 , 14 ] . While some methods [ 43 , 15 ] favor a high tracking frame rate for real-time applications, others favor reconstruction accuracy [ 53 ] . Facial Landmark Prediction Numerous reconstruction methods [ 26 , 5 ] for faces rely on accurate landmark predictions, which are usually coupled with vertices of a template mesh. Pioneering work on detecting such landmarks already relies on statistical learning [ 7 ] and more recent models exploit large datasets [ 46 , 48 ] and neural networks to improve the performance [ 4 , 1 ] . MediaPipe [ 1 ] , for instance, uses a convolutional network inspired by MobileNet [ 17 ] . Another line of work focuses on densely

Read the complete text