VariFace: Fair and Diverse Synthetic Dataset Generation for Face Recognition

Michael Yeung [email protected], Toya Teramoto [email protected], Songtao Wu [email protected], Tatsuo Fujiwara [email protected], Kenji Suzuki [email protected], Tamaki Kojima [email protected] · 2024

Read VariFace: Fair and Diverse Synthetic Dataset Generation for Face Recognition by Michael Yeung [email protected], Toya Teramoto [email protected], Songtao Wu [email protected], Tatsuo Fujiwara [email protected], Kenji Suzuki [email protected], Tamaki Kojima [email protected] (2024) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2412-06235.txt.

Introduction

Michael Yeung and co-authors present VariFace, a way to build fake face photo sets for training face recognition systems, in a research paper from 2024. Large sets scraped from the web raise privacy and bias concerns. Fake sets are safer, but past sets lacked variety and trained weaker systems than real photos did.

VariFace works in two stages with a diffusion model, a tool that builds images step by step. The team added three fixes: one cleans up group labels, one raises variety between people, and one balances keeping a person's look against varying their photos. At the same set size, VariFace beat older fake sets on test scores. In a larger run, it even beat a well-known real photo set across six test sets.

Catalog record

CREATORMichael Yeung [email protected], Toya Teramoto [email protected], Songtao Wu [email protected], Tatsuo Fujiwara [email protected], Kenji Suzuki [email protected], Tamaki Kojima [email protected]
YEAR2024
COLLECTIONPAPERS
FORMATTXT
FILE SIZE62 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2412-06235

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

VariFace: Fair and Diverse Synthetic Dataset Generation for Face Recognition Michael Yeung [email protected], Toya Teramoto [email protected], Songtao Wu [email protected], Tatsuo Fujiwara [email protected], Kenji Suzuki [email protected], Tamaki Kojima [email protected] Abstract The use of large-scale, web-scraped datasets to train face recognition models has raised significant privacy and bias concerns. Synthetic methods mitigate these concerns and provide scalable and controllable face generation to enable fair and accurate face recognition. However, existing synthetic datasets display limited intraclass and interclass diversity and do not match the face recognition performance obtained using real datasets. Here, we propose VariFace, a two-stage diffusion-based pipeline to create fair and diverse synthetic face datasets to train face recognition models. Specifically, we introduce three methods: Face Recognition Consistency to refine demographic labels, Face Vendi Score Guidance to improve interclass diversity, and Divergence Score Conditioning to balance the identity preservation-intraclass diversity trade-off. When constrained to the same dataset size, VariFace considerably outperforms previous synthetic datasets (0.9200 → \rightarrow 0.9405) and achieves comparable performance to face recognition models trained with real data (Real Gap = -0.0065). In an unconstrained setting, VariFace not only consistently achieves better performance compared to previous synthetic methods across dataset sizes but also, for the first time, outperforms the real dataset (CASIA-WebFace) across six evaluation datasets. This sets a new state-of-the-art performance with an average face verification accuracy of 0.9567 (Real Gap = +0.0097) across LFW, CFP-FP, CPLFW, AgeDB, and CALFW datasets and 0.9366 (Real Gap = +0.0380) on the RFW dataset. 1 Introduction Figure 1: Face verification accuracy using synthetic datasets. Face verification accuracy is the average performance across LFW, CFP-FP, CPLFW, AgeDB, and CALFW datasets. VariFace is trained only with data from CASIA-WebFace (real), the performance of which is shown for reference. All other results are taken from their respective papers. A decade after the breakthrough performances of DeepFace ( Taigman et al., 2014 ) and DeepID ( Sun et al., 2014 ) , deep learning remains the state-of-the-art approach for face recognition (FR) ( Wang & Deng, 2021 ; Gururaj et al., 2024 ) . Deep learning performance is limited by training dataset size ( Zhu et al., 2021 ) , and the creation of large-scale FR datasets such as CASIA-WebFace ( Yi et al., 2014 ) , MS-Celeb-1M ( Guo et al., 2016 ) and MegaFace ( Kemelmacher-Shlizerman et al., 2016 ) was central to the success of deep learning in FR. However, the development of these massive face datasets involves scraping face image data from the internet without permission from subjects, and all but one of the aforementioned datasets have since been retracted or decommissioned ( Boutros et al., 2022 ) . Legally, Article 9 (1) General Data Protection Regulation (GDPR) ( Parliament, 2016 ) in the EU prohibits the processing of biometric data for the purpose of uniquely identifying a natural person, except when the data subject has given explicit consent to the processing of those personal data for one or more specified purposes (Article 9(2)(a) GDPR). Moreover, the AI Act (AIA) in the EU came into effect on August 1st, 2024, where Article 5(1)(e) AIA prohibits the placing on the market or the use of AI systems that create or expand facial recognition databases through the untargeted scraping of facial images ( Parliament, 2024 ) . Together, these regulations illustrate a growing legal concern over the use of large-scale web-scraped face datasets for training face recognition models. Besides privacy concerns, real face datasets suffer from data imbalance, including limited face pose and lighting variation ( Liu et al., 2022a ) , as well as under-representation of protected characteristics such as race and gender ( Liu et al., 2022a ; Karkkainen & Joo, 2021 ; Thong et al., 2023 ) . This results in FR models trained on these datasets exhibiting robustness and fairness issues, with the latter raising significant legal and ethical implications ( Mehrabi et al., 2021 ) . To address the privacy risks and biases associated with real face datasets, there is increasing interest in the development of synthetic face datasets ( Baltsou et al., 2024 ; Qiu et al., 2021 ; Liu et al., 2022b ; Bae et al., 2023 ; Boutros et al., 2023a ; Melzi et al., 2023 ; Kim et al., 2023 ) . Moreover, synthetic methods not only facilitate scalable dataset generation to support deep FR model training ( Trigueros et al., 2021 ; Bae et al., 2023 ) , but also provide precise control over the composition of the dataset ( Liu et al., 2022b ; Melzi et al., 2023 ; Banerjee et al., 2023 ; Baltsou et al., 2024 ) . The most widely used synthetic data generation methods are 3D parametric ( Wood et al., 2021 ; Bae et al., 2023 ) and deep generative ( Trigueros et al., 2021 ; Qiu et al., 2021 ; Liu et al., 2022b ; Boutros et al., 2023b ; Boutros et al., 2023a ; Melzi et al., 2023 ; Kim et al., 2023 ) models, the latter encompassing GAN ( Deng et al., 2020 ; Shen et al., 2020 ; Qiu et al., 2021 ; Boutros et al., 2022 ; Liu et al., 2022b ; Boutros et al., 2023b ; Boutros et al., 2024 ; Wu et al., 2024a ) and diffusion-based approaches ( Boutros et al., 2023a ; Melzi et al., 2023 ; Kim et al., 2023 ; Kansy et al., 2023 ; Papantoniou et al., 2024 ; Li et al., 2024 ) . Despite the considerable progress in the development of synthetic face data, there remains a substantial gap in FR performance between models trained on real compared to synthetic data ( Boutros et al., 2023c ; Shahreza et al., 2024 ; DeAndres-Tame et al., 2024 ) . This is because synthetic datasets tend to amplify biases inherent in the real face datasets used for training ( Thong et al., 2023 ; Leyva et al., 2024 ) , while also introducing unique challenges that affect FR performance ( Bae et al., 2023 ; Kim et al., 2023 ) . To mitigate the demographic biases in real datasets, several synthetic methods are able to generate datasets with balanced demographic distributions ( Qiu et al., 2021 ; Kim et al., 2023 ; Melzi et al., 2023 ) . However, these methods rely on using either existing demographic labels or labels generated by supervised models ( Taigman et al., 2014 ; Kim et al., 2023 ; Melzi et al., 2023 ) , which exhibit poor performance on minority classes (see Fig. S2 ). Another issue inherent with synthetic methods is the challenge of balancing identity (ID) preservation with generating diverse images for each individual. Although ID preservation can be achieved using pretrained FR model embeddings as a conditioning signal ( Boutros et al., 2023a ; Kim et al., 2023 ; Li et al., 2024 ) , generating sufficient diversity to reflect ‘in-the-wild’ variation remains challenging. To generate image variations for an individual, current methods rely on extracting specific attributes such as face pose, expression, and illumination ( Melzi et al., 2023 ; Papantoniou et al., 2024 ) . However, this requires using supervised models trained on those attributes and ignores properties that cannot be easily classified

Read the complete text