Evaluation of Human Visual Privacy Protection: A Three-Dimensional Framework and Benchmark Dataset

Sara Abdulaziz, Giacomo D’Amicantonio, Egor Bondarev · 2025

Read Evaluation of Human Visual Privacy Protection: A Three-Dimensional Framework and Benchmark Dataset by Sara Abdulaziz, Giacomo D’Amicantonio, Egor Bondarev (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2507-13981.txt.

Introduction

Abdulaziz, D'Amicantonio, and Bondarev build a way to grade tools that protect privacy in images of people in twenty twenty-five. AI cameras can now gather fine detail about a person from one photo. Tools that blur or mask that detail need a fair test. The team scores such tools on three axes: privacy, usefulness, and ease of real use.

They also built a public photo set with labels for body traits, soft traits, and other facts. Eleven tools were tested, from simple blurs to deep learning methods. The tests matched how humans judge privacy in a photo. They also showed the trade offs each tool makes among the three axes.

Catalog record

CREATORSara Abdulaziz, Giacomo D’Amicantonio, Egor Bondarev
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE35 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2507-13981

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Evaluation of Human Visual Privacy Protection: A Three-Dimensional Framework and Benchmark Dataset Sara Abdulaziz, Giacomo D’Amicantonio, Egor Bondarev Abstract Recent advances in AI-powered surveillance have intensified concerns over the collection and processing of sensitive personal data. In response, research has increasingly focused on privacy-by-design solutions, raising the need for objective techniques to evaluate privacy protection. This paper presents a comprehensive framework for evaluating visual privacy-protection methods across three dimensions: privacy, utility, and practicality. In addition, it introduces HR-VISPR, a publicly available human-centric dataset with biometric, soft-biometric, and non-biometric labels to train an interpretable privacy metric. We evaluate 11 privacy protection methods, ranging from conventional techniques to advanced deep-learning methods, through the proposed framework. The framework differentiates privacy levels in alignment with human visual perception, while highlighting trade-offs between privacy, utility, and practicality. This study, along with the HR-VISPR dataset, serves as an insightful tool and offers a structured evaluation framework applicable across diverse contexts. 1 Introduction Recent AI advancements have enabled intelligent monitoring systems capable of real-time event detection and behavioral analysis [ 28 ] . However, these systems raise significant privacy concerns due to the extensive collection and processing of private data. Consequently, data protection regulations, such as the GDPR [ 17 ] , have mandated privacy-by-design strategies, driving extensive research in visual privacy protection [ 4 , 10 , 12 , 14 ] . As visual privacy-protection methods inherently cause information loss through the obfuscation of sensitive regions, they often degrade the performance of downstream computer vision tasks, i.e., utility tasks. Even though utility models should ideally be independent of sensitive human attributes, such as face, skin-color, or clothing, performance drops presist even when models are re-trained on anonymized data [ 10 , 7 , 6 , 25 ] . Prior work on the privacy-utility trade-off analysis for human-related vision tasks often targets narrow protection methods [ 6 , 45 ] . For instance, in [ 6 ] , the authors assess the utility performance across the protection degree of a single method (e.g., blurring). In [ 45 ] , the performance is evaluated across the degree of sensitive object removal (e.g., bounding-box vs. segment). Other works, which include a broader assessment of protection methods, focus either on image-level, private vs. public [ 48 ] , or on face-level de-identification [ 9 ] . Despite offering insights into the utility performance under narrow protection methods, they remain insufficient to evaluate complex protection approaches. Recently, trainable privacy metrics have gained attention with the introduction of multi-label visual privacy datasets, such as VISPR [ 27 ] . The primary approach involves training a privacy-attribute classifier on the unprotected private data, then computing the class-based mean average precision on the protected test sets to rank the privacy-protection level [ 27 , 10 , 7 , 42 ] . Despite its potential, the reliability of this approach is compromised by several shortcomings in prior implementations [ 41 , 10 , 7 , 42 ] . These include the use of inaccurate privacy-labeled data and evaluation on a narrow subset of privacy attributes and protection methods. In this paper, we propose an objective three-dimensional privacy evaluation framework assessing visual privacy-protection methods across privacy, utility, and practicality. It employs a human-centric privacy metric trained on HR-VISPR, a newly introduced dataset with comprehensive human-related labels, including biometric, soft-biometric, and non-biometric attributes. HR-VISPR serves as a benchmark for future research. Using this framework, we evaluate 11 visual privacy-protection methods, ranging from traditional to advanced deep-learning based methods, offering a more robust and explainable assessment of the privacy-utility-practicality trade-off. 2 Related Work Privacy-protection evaluation spans two domains. The first, focused on guiding online data-sharing decisions [ 27 , 3 , 37 , 38 ] , lies outside our scope. The second evaluates privacy-by-design solutions across utility domains [ 10 , 9 , 34 , 8 ] , with schemes tailored to specific tasks. These evaluation methods primarily assess two aspects: privacy protection level and the utility performance of the anonymized data. Face-Level Privacy Evaluation. In the domain of face recognition, privacy protection involves altering facial details either by face synthesis [ 11 , 15 ] , or face perturbation [ 2 , 23 ] . In this context, the objective of privacy evaluation is to ensure anonymity against AI-based recognition models. Several metrics are employed to measure the face protection level, including face de-identification rate [ 9 , 40 , 24 , 16 ] , face-shape retention rate [ 20 ] , face verification [ 21 ] , k-anonymity score [ 26 , 5 ] , and user survey metrics [ 11 , 29 , 39 , 2 ] . Human-Level Privacy Evaluation. When the entire human body is targeted for privacy protection, it causes a greater loss of visual information, leading to a higher performance drop in utility tasks that rely on human features, such as action recognition and anomaly detection. One approach to evaluate the protection level in this case is by assessing human recognition scores in user surveys [ 39 ] . However, to avoid their time-consuming nature and the potential bias, recent methods employ deep learning models in place of human evaluators, measuring privacy attribute prediction accuracy [ 10 , 7 , 30 , 41 ] . A common approach trains a privacy-attribute classifier on non-anonymized visual data to detect human attributes like face, hair color, race, and gender [ 41 ] . The classifier is then applied to anonymized data, and the class-based mean average precision (cMAP) is computed to quantify the privacy level of each anonymization method. To analyze the privacy-utility trade-off, the anonymized data is applied to a utility task, and the privacy-utility scores are analyzed to help in selecting the best privacy-protection method [ 10 , 7 ] . 3 Privacy Evaluation Framework This section presents the human-centric visual privacy dataset and the privacy evaluation framework. 3.1 HR-VISPR Dataset Existing visual privacy datasets vary widely in their annotation type due to the differing research assumptions. The common and popular privacy datasets include PicAlert [ 46 ] , YourAlert [ 35 ] , VizWiz-Priv [ 13 ] , PrivacyAlert [ 48 ] , DIPA [ 43 ] , BIV-Priv [ 31 ] , and DIPA2 [ 44 ] . These datasets are mostly annotated with a binary image label, e.g. private or public, according to the presence of personally identifiable information. In contrast, the Visual Privacy dataset (VISPR) [ 27 ] , proposed initially for guiding online image sharing decisions, incorporates a wide range of human-related privacy attributes. It consists of 22k personal and public images, annotated using 68 binary privacy attributes, including face, gender, age, nudity, hobbies, race, skin color, ethnic clothing, relationships, ID documents, address, and so on. Due to the wide coverage of private attributes, the dataset has been serving as a useful benchmark for privacy-protection evaluation [ 10 , 7 , 42 ] . However, despite its diversity, the dataset has several major limitations that significantly impact its fairness for the privacy evaluation purposes. These limitations, illustrated in Fig. 1 , include the following. Extreme Complexity. the dataset combines a wide range of complex scenes, including distant crowd scenes, wide-view images with humans represented by a

Read the complete text