FairDeFace: Evaluating the Fairness and Adversarial Robustness of Face Obfuscation Methods

Seyyed Mohammad Sadegh Moosavi Khorzooghi 2 , Poojitha Thota 2 , Mohit Singhal 3 , Abolfazl Asudeh 4 Gautam Das 2 , Shirin Nilizadeh 2 · 2025

Read FairDeFace: Evaluating the Fairness and Adversarial Robustness of Face Obfuscation Methods by Seyyed Mohammad Sadegh Moosavi Khorzooghi 2 , Poojitha Thota 2 , Mohit Singhal 3 , Abolfazl Asudeh 4 Gautam Das 2 , Shirin Nilizadeh 2 (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2503-08731.txt.

Introduction

Seyyed Moosavi Khorzooghi and co-authors grade face-blurring tools for both toughness and fairness. Blurring faces protects privacy in photos shared or stored at scale. Each tool had been tested on its own turf with its own yardsticks. Their FairDeFace platform lines up data sets, detectors, attacks, and fairness scores in one place.

Six attacks and several privacy, utility, and fairness metrics ship with the frame. Over five hundred runs compared seven blurring methods head to head. Some tools cracked more easily under attack than their claims suggested. Others guarded some gender or racial groups less well than the rest. Heat maps show which facial zones each method changes and misses.

Catalog record

CREATORSeyyed Mohammad Sadegh Moosavi Khorzooghi 2 , Poojitha Thota 2 , Mohit Singhal 3 , Abolfazl Asudeh 4 Gautam Das 2 , Shirin Nilizadeh 2
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE84 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2503-08731

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

FairDeFace: Evaluating the Fairness and Adversarial Robustness of Face Obfuscation Methods Seyyed Mohammad Sadegh Moosavi Khorzooghi 2 , Poojitha Thota 2 , Mohit Singhal 3 , Abolfazl Asudeh 4 Gautam Das 2 , Shirin Nilizadeh 2 Abstract The lack of a common platform and benchmark datasets for evaluating face obfuscation methods has been a challenge, with every method being tested using arbitrary experiments, datasets, and metrics. Moreover, while prior work has demonstrated that face recognition systems exhibit bias against some demographic groups, there exists a substantial gap in our understanding regarding the fairness of face obfuscation methods. Providing fair face obfuscation methods can ensure equitable protection across diverse demographic groups, especially since they can be used to preserve the privacy of vulnerable populations, such as activists, survivors of abuse, etc. To address these gaps, this paper introduces a comprehensive framework, named 𝖥𝖺𝗂𝗋𝖣𝖾𝖥𝖺𝖼𝖾 \mathsf{FairDeFace} , designed to assess the adversarial robustness and fairness of face obfuscation methods. The framework introduces a set of modules encompassing data benchmarks, face detection and recognition algorithms, adversarial models, utility detection models, and fairness metrics. 𝖥𝖺𝗂𝗋𝖣𝖾𝖥𝖺𝖼𝖾 \mathsf{FairDeFace} serves as a versatile platform where any face obfuscation method can be integrated, allowing for rigorous testing and comparison with other state-of-the-art methods. In its current implementation, 𝖥𝖺𝗂𝗋𝖣𝖾𝖥𝖺𝖼𝖾 \mathsf{FairDeFace} incorporates 6 attacks, and several privacy, utility and fairness metrics. Using 𝖥𝖺𝗂𝗋𝖣𝖾𝖥𝖺𝖼𝖾 \mathsf{FairDeFace} , and by conducting more than 500 experiments, we evaluated and compared the adversarial robustness of seven face obfuscation methods. This extensive analysis led to many interesting findings both in terms of the degree of robustness of existing methods and their biases against some gender or racial groups. FairDeFace also uses visualization of focused areas for both obfuscation and verification attacks to show not only which areas are mostly changed in the obfuscation process for some demographics, but also why they failed through focus area comparison of obfuscation and verification. I Introduction With the proliferation of visual content shared online, there is a growing risk of unauthorized identification and misuse of personal data, particularly through advancements in facial recognition technology. The computer security and vision communities have proposed various face obfuscation (also called face de-identification or anonymization) methods to safeguard individuals’ privacy and mitigate the risks associated with the widespread dissemination of visual data. While traditional methods, such as blurring and pixelation [ 1 ] solely aimed at eliminating identity information to anonymize or obfuscate facial features, the more recent Generative Adversarial Networks (GANs)-based methods like DeepPrivacy [ 2 ] and CIAGAN [ 3 ] have emerged to balance privacy preservation while maintaining some attributes of the face, such as their emotions and pose. The effectiveness of face obfuscation methods has been assessed based on factors like the ability to prevent re-identification, the impact on Face Recognition (FR) algorithms, and the overall usability of the obfuscated face in specific applications, e.g., detecting the emotions of faces in an already anonymized dataset. However, the lack of a common platform and benchmark datasets for evaluating face obfuscation methods has been a challenge. Every method tests different threat models, implements arbitrary experiments, and uses arbitrary datasets and metrics, making it difficult to compare different methods objectively and understand their relative strengths and weaknesses. For example, the evaluation of CIAGAN’s privacy, which focuses on anonymizing faces while maintaining their pose, is conducted using the LFW dataset and FaceNet re-identification attacks [ 3 ] . However, this evaluation overlooks other potential attacks, such as verification attacks, its performance is only compared with that of a few other methods [ 4 ] . Moreover, its performance in preserving the face pose is not systematically tested. This trend is not unique to CIAGAN and is observed across evaluations of various other methods. Prior research has extensively studied the fairness of FR systems, showing their strong performance differences across diverse demographic categories, including gender, race, and age [ 5 , 6 , 7 ] . Like FR systems, face obfuscation systems might not be fair towards certain demographics, with consequences as severe or even worse than those observed with bias in FR systems. For example, obfuscated faces of marginalized or underrepresented groups might be more likely to be reconstructed or identified. This can lead to various negative outcomes, including increased risks of privacy infringements, targeted surveillance, or harassment. However, to the best of our knowledge, only one study has investigated the fairness properties of a class of face obfuscation systems, i.e., adversarial-based methods [ 8 ] . Rosenberg et al. [ 8 ] showed that larger perturbations are needed for the minority groups to have the same obfuscation as the majority groups, which can be unfair because larger perturbations mean less imperceptibility. There exists a significant gap in our understanding regarding the fairness of other classes of face obfuscation systems in terms of their performance under various threat models and possible attacks and their ability to preserve utility. To address these gaps, this paper introduces \mathsf{FairDeFace} framework for evaluating the fairness and adversarial robustness of face obfuscation methods. The framework introduces a set of modules encompassing data benchmarks, face detection and recognition algorithms, adversarial models, utility detection models, and fairness metrics. Additionally, it incorporates visualization tools using saliency maps and automated feature recognition capabilities through MediaPipe face mesh model to analyze obfuscation patterns. The proposed framework serves as a versatile platform, where any face obfuscation method can be integrated for testing and comparison with other state-of-the-art methods. Using \mathsf{FairDeFace} , and by conducting more than 500 experiments, we evaluated and compared the fairness and adversarial robustness of seven face obfuscation methods. Some of this paper’s contributions are: (1) \mathsf{FairDeFace} is a versatile framework for systematically evaluating and comparing any face obfuscation method with existing ones. (2) Created benchmark datasets for evaluating the fairness and robustness of face obfuscation methods. (3) Using \mathsf{FairDeFace} , we evaluated and compared the performance of seven face obfuscation methods concerning quality, privacy, and utility through state-of-the-art methods. Our analysis resulted in some new findings, e.g., GAN-based methods, especially DeepPrivacy2, having high performance in all the three stages, and low performance of Fawkes even with strong perturbations. (4) We found that the bias of face obfuscation methods is not necessarily the opposite of the bias in face recognition as Black Males usually had high Bias in both due to large-scale changes done during the obfuscation process. (5) Measured how the fairness evaluation is sensitive to the quality of testing datasets, and if the findings are consistent across different datasets. We found that the fairness was more dependent on the obfuscation and face recognition method rather than datasets. (6) We proposed and analyzed attribute-preserving rates of GAN-based methods using both open-source and third-party tools. (7) Proposed a novel approach combining saliency maps and automated feature recognition to analyze focus regions of face obfuscation systems and their corresponding face recognition attacks. Our analysis revealed that obfuscation methods, such as Fawkes, rely more on limited or unrelated features than the whole face in robust face recognition systems to change the identity. This is more severe for some demographics, with a lower correlation, which is compatible with the privacy attack results. This can explain their poor performance and bias. (8) Analysing passing rates or successful

Read the complete text