A Responsible Face Recognition Approach for Small and Mid-Scale Systems Through Personalized Neural Networks

Sebastian Groß, Stefan Heindorf, Philipp Terhörst · 2025

Read A Responsible Face Recognition Approach for Small and Mid-Scale Systems Through Personalized Neural Networks by Sebastian Groß, Stefan Heindorf, Philipp Terhörst (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2505-19920.txt.

Introduction

Sebastian Groß, Stefan Heindorf, and Philipp Terhörst propose in 2025 a face recognition method that stores a small personal network for each identity. Standard systems store fixed face vectors, which can be hard to explain and raise fairness and privacy fears. Small systems feel those costs most.

At sign-up, the method trains one small yes-no network per person from a single photo. Fake balanced samples let staff tune fairness for each person at that stage. Tests across data sets and base systems show large gains in fairness and strong gains in privacy. The trade-off is slower checks and more storage, a fair price for small setups.

Catalog record

CREATORSebastian Groß, Stefan Heindorf, Philipp Terhörst
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE35 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2505-19920

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

A Responsible Face Recognition Approach for Small and Mid-Scale Systems Through Personalized Neural Networks Sebastian Groß, Stefan Heindorf, Philipp Terhörst Abstract Traditional face recognition systems rely on extracting fixed face representations, known as templates, to store and verify identities. These representations are typically generated by neural networks that often lack explainability and raise concerns regarding fairness and privacy. In this work, we propose a novel model-template (MOTE) approach that replaces vector-based face templates with small personalized neural networks. This design enables more responsible face recognition for small and medium-scale systems. During enrollment, MOTE creates a dedicated binary classifier for each identity, trained to determine whether an input face matches the enrolled identity. Each classifier is trained using only a single reference sample, along with synthetically balanced samples to allow adjusting fairness at the level of a single individual during enrollment. Extensive experiments across multiple datasets and recognition systems demonstrate substantial improvements in fairness and particularly in privacy. Although the method increases inference time and storage requirements, it presents a strong solution for small- and mid-scale applications where fairness and privacy are critical. 1 Introduction Face recognition technology has emerged as a cornerstone of modern biometric systems, fundamentally transforming security, authentication, and human-computer interactions [ 1 ] . Recent advances in deep learning, particularly through architectures like Convolutional Neural Networks (CNNs) [ 2 , 3 ] and Transformers [ 4 , 5 ] , have pushed face recognition accuracy to unprecedented levels, rivaling and exceeding human capabilities [ 5 ] . Despite recent advances, face recognition systems continue to face critical challenges related to responsibility, particularly in terms of privacy, fairness, and explainability. In terms of privacy, deep learning–based face templates often contain more than just identity-related information [ 6 , 7 ] . Attributes such as gender, age, ethnicity, and even health status can be inferred from these templates [ 8 , 9 ] . Since these representations are typically intended for recognition purposes only, the presence of such additional personal information poses significant privacy risks [ 10 ] . Unauthorized access to this data can result in unfair or discriminatory treatment, undermining individual rights and trust in the technology. The presence of this information might also lead to fairness issues [ 11 , 12 , 13 ] . Most biometric systems rely on representation learning strategies that are optimized for overall recognition performance. However, these strategies are highly sensitive to the distribution of the training data properties. As a result, the system’s accuracy can vary across different population groups, leading to biased outcomes [ 14 ] . This becomes particularly problematic in critical applications such as law enforcement or forensic investigations, where such disparities can have serious consequences. For these critical decision-making processes, explainability remains a substantial challenge as well. Most systems provide only a match score or decision, without insight into how that result was reached. This lack of transparency makes it difficult to interpret or contest outcomes, especially in high-stakes settings. Clear, understandable reasoning is essential to support trust and responsibility in these technologies [ 15 , 16 , 17 , 18 , 19 ] . To address these fundamental challenges simultaneously, we propose an alternative solution in the form of personalized neural networks. Rather than training a single model to handle all identities by extracting and storing templates, our approach creates and stores model-templates (MOTE) in the form of a dedicated small-scale binary classifier for each enrolled individual. These personalized neural network classifiers can decide if the probe identity belongs to the enrolled reference identity. To overcome the challenge of training the model with only one positive reference sample, we introduce a pipeline to create synthetic training data that further allows to balance the training data and thus, to adjust the fairness at the level of a single individual. With extensive experimentation across multiple datasets and face recognition systems, the results show that changing from storing vector-templates to model-templates can significantly enhance fairness and privacy-protection. Moreover, since our approach transforms face recognition from a complex representation learning task to a set of simple binary classification tasks, well-researched approaches for explainability can be applied to get more understandable decisions. Our reliable face recognition approach shows strong advantages in terms of privacy, fairness, and explainability at the cost of higher storage time and computation requirements, making our MOTE approach highly useful for small- and mid-scale recognition systems that prioritize responsibility factors such as fairness and privacy. 2 Related Work Current face recognition systems predominantly leverage deep learning (DL) models, primarily Convolutional Neural Networks (CNNs) and more recently Transformers, to achieve high accuracy [ 15 ] . The standard paradigm involves training these complex models to map facial images onto low-dimensional, discriminative feature vectors, commonly referred to as templates or embeddings [ 20 , 2 , 3 , 21 ] . While highly effective in terms of recognition accuracy, this prevailing template-based DL approach presents significant challenges concerning algorithmic fairness, data privacy, and system explainability [ 15 ] . The learned embeddings, designed to capture identity, often inadvertently encode sensitive demographic attributes [ 7 , 6 ] (like gender, age, or race), leading to potential biases [ 12 ] in performance across different groups and raising privacy concerns if templates are compromised or misused [ 8 , 10 ] . Furthermore, the inherent complexity and “black-box” nature of deep neural networks make it difficult to understand or explain why a particular recognition decision was made. 2.1 Privacy in Face Recognition The inadvertent encoding of demographic attributes in face embeddings has spurred research into privacy-enhancing techniques, as attackers might be able to extract privacy-sensitive information from the stored templates. Privacy-enhancing techniques usually aim to preserve the recognition performance while making it hard to extract information by either (a) concealing attribute patterns [ 22 ] , (b) changing the recognition process and the stored information [ 23 , 24 ] , or (c) by directly removing the attribute patterns [ 25 , 26 , 27 ] or indirectly removing these patterns through disentangling identity and attribute information [ 28 , 29 ] . Despite these advances, recent work by Osorio-Roig et al. [ 9 ] demonstrated that privacy-enhancement techniques remain vulnerable to sophisticated inference attacks, underscoring the need for stronger protections against them. 2.2 Fairness in Biometric Systems Demographic performance disparities in face recognition systems have been extensively documented. Drozdowski et al. [ 14 ] survey sources of bias in biometric systems, highlighting issues in data collection, algorithm development, and evaluation protocols. Mitigation strategies include dataset balancing approaches [ 30 ] , fairness-enhancing model based approaches [ 31 , 32 , 33 ] , and supervised [ 34 ] and unsupervised score normalization approaches [ 35 , 36 ] . These approaches attempt to achieve equitable performance within the constraints of the universal embedding paradigm. While strong advancements have been made, the complex nature of the large neural networks utilized has made it difficult to address fairness, especially on the individual level. 2.3 Explainability in Face Recognition Understanding model decisions remains challenging, especially for face recognition systems based on embeddings and similarity metrics rather than explicit classifications. White-box methods like xSSAB [ 18 ] and EBP [ 17 ] analyze gradient

Read the complete text