Generating 3D models from sketches of human faces using a combined approach of Convolutional Neural Networks, Procedural Modeling, and Contour Mapping

Nancy Iskander · 2026

Read Generating 3D models from sketches of human faces using a combined approach of Convolutional Neural Networks, Procedural Modeling, and Contour Mapping by Nancy Iskander (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2605-25418.txt.

Introduction

Nancy Iskander builds 3D face models from sketches by combining expression detection with model generation. Because expressions reshape facial contours, her method first reads the expression in a sketch, then reproduces it on a 3D model. Convolutional networks trained on her own dataset detect the expression through facial action units. The detected expression is copied onto a parametric face model called Valley Girl. Active snake contours then compute the warps that close gaps between that model and the sketch.

Catalog record

CREATORNancy Iskander
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE45435
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2605.25418

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Generating 3D models from sketches of human faces using a combined approach of Convolutional Neural Networks, Procedural Modeling, and Contour Mapping Nancy Iskander Abstract Generating 3D models from face sketches is an active topic of research in Computer Graphics due to its potential to tremendously facilitate the modeling of faces for both professional 3D arists and novices. Motivated by the observation that facial expressions are responsible for significantly altering and shaping the contours in our faces, we combine both expression detection and 3D model generation in our approach. The result is a novel approach to generating 3D models from sketches which relies on three components: Convolutional Neural Networks, a parametric 3D face model (Valley Girl), and Active Snake Contours. For the first time in the literature, CNNs are trained (using our own generated dataset) to detect the expression in the given sketch through detecting the active FACS Action Units. The expression is then duplicated on Valley Girl to obtain a 3D model with a similar expression. Active Snake Contours are then used to find the transforms needed to close the gaps between that model and the given sketch. {preliminary} Acknowledgements Thank you to my supervisor Karan Singh for going above and beyond in order to help me grow as a researcher and as a person, for believing in me, and for general awesomeness. Thank you to Chris Landreth for sharing his parametric face model (Valley Girl) and for imparting his knowledge of FACS, Maya, and facial animation on me. Thank you to Sanja Fidler for detangling Deep Learning and making it seem accessible. Thank you to Lynda Barnes for exceptional response times whenever I needed information on the program. Thank you to my parents Eman Soliman and Magdy Hanna, my brother Samir Iskander, and Pierre-Luc Loyer for their support and motivation. Finally, thank you to my employer Behaviour Digital for being accomodating of my schedule when I needed to work on this thesis. Chapter 1 Introduction Generating 3D models from line sketches is an interesting problem that has received a lot of attention in the literature, beginning with approaches that do contour matching between an existing model and the sketch [ 13 ] , to those relying on deep learning to directly generate the model from the sketch [ 9 ] . This work develops a combined approach where both deep learning and contour matching between an existing model and the sketch are used. When drawn as a sketch, a face contorted into an expression will contain a greater number of contours than a blank face. Oftentimes, artists use a face sketch to portray a particular emotion, as seen in figure 1.1 . This is why we aim to detect and reproduce the expression from the given sketch on our model before attempting to do contour matching between them. In order to detect the expression, we generated our own dataset of 3D models and corresponding sketches, which is used to train Convolutional Neural Networks to detect facial actions in the given sketch. Figure 1.1: Sketches of expressive faces found on the internet, where a wide range of active AUs can be observed (e.g. “smile”, “brow furrow”, “sneer”, etc). Applications. There is a wide range of applications for generating 3D models from sketches using our method in professional and non-professional settings: • 3D artists can quickly prototype their character designs and obtain a starting point for their model. • Novices and Indie developers can generate character heads with little to no knowledge of 3D modelling software. • Witnesses and sketch-artists can obtain a higher level of accuracy by being able to quickly visualize their sketches as renders and iterate on them. • Children can effortlessly see their simple sketches come to life. • An expression from a sketch can be replicated on any model. Before delving into the details of our method, there are three components that are important to understand: FACS (described in 1.1 ), Valley Girl (described in 1.2 ), and Snakes (described in 1.3 ). 1.1 Overview of FACS The Facial Action Coding System (FACS) was developed by Ekman et al. [ 6 ] in 1977. Describing the emotion portrayed by a facial expression (e.g. “happiness”, “sadness”, “anger”, etc) is not precise enough and does not tell you what the facial expression actually looks like (see figure 1.2 ). On the other hand, detecting the exact muscle groups which are active in a facial expression is nearly impossible for both humans and computers (if the input is an image), because the relation between muscle actions and observable changes in facial appearance is many-to-many. For example, the frontalis muscle can be used to raise one brow, to wrinkle the entire forehead, and to wrinkle just the center of the forehead (one muscle mapping to many facial changes). Conversely, the risorius and platysma muscles are always fired together and produce a grimace (many muscles mapping to one change). FACS encodes only observable components of facial movement as AUs, and each facial expression can be broken down into its constituent AUs, which is what we aim to vary in the generated dataset (section 2.1 ) and later detect (section 2.2 ). Figure 1.2: Three facial expressions that can be construed as “happy” that are in fact very different. The first column shows the expression on Valley Girl. The second column shows which muscles are active in each expression. 1.2 Overview of Valley Girl Valley Girl is a physiologically accurate 3D facial model developed in Maya by Chris Landreth [ 19 ] . All facial muscle actions are modelled using blendshapes and exposed as parameters modelled after FACS AUs. Table 1.1 shows the mapping between a subset of Valley Girl facial actions, muscles used, and FACS AUs. The FACS AU parameters are divided into 12 upper-face controls, 11 mid-face controls, and 24 lower-face controls. In addition, there are three controls for rotating the head in 3D space, as well as three for rotating the neck. There are parameters for choosing the ethnicity and the gender which can be used to vary the facial features. I have also augmeneted the original Valley Girl to add parameters for varying facial features more explicitly (e.g. eye shape, jawline shape, etc). Figure 1.3 provides an idea of what Valley Girl’s UI looks like in Autodesk MAYA. These parameters can be used for posing and facial animation. (The same parameters can be controlled programmatically.) Figure 1.3: Valley Girl’s UI in Autodesk MAYA. Table 1.1: Mapping between a subset of face actions used in Valley Girl UI, muscles used and FACS AUs. Action Signature Muscles Involved FACS AUs Valley Girl Raising eyebrows Worry lines Frontalis 1, 2 Lowering brow Furrowed eyebrows Currogator 4 Blinking Open/Closed eyes Levator palpebrae superioris 5 Squinting Furrowed eyes Orbicularis Oculi 6, 44 Wincing Squared lips, nasal folds Levator Labii Superioris Alaeque Nasi 9 Sneering Squared lips Levator Labii Superioris 10 Smiling Upturned mouth Zygomatic Major 12 Puckering Puckered lip corners Buccinator 14 Frowning Downturned mouth Triangularis 15 Lowering lower lip Mouth shape for ”s” sound Depressor Labii Inferioris 16 Shrugging Squashed lower lip Mentalis 17 Pursing lips Mouth shape for kissing Orbicularis Oris 18 Grimacing Stretched lower lip Risorius/Platysma 20 Lip loosening Mouth

Read the complete text