Who Can See Through You? Adversarial Shielding Against VLM-Based Attribute Inference Attacks
Yucheng Fan, Jiawei Chen, Yu Tian, Zhaoxia Yin, [0.5em] East China Normal University, Shanghai, China · 2025
Read Who Can See Through You? Adversarial Shielding Against VLM-Based Attribute Inference Attacks by Yucheng Fan, Jiawei Chen, Yu Tian, Zhaoxia Yin, [0.5em] East China Normal University, Shanghai, China (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2512-18264.txt.
Introduction
Yucheng Fan, Jiawei Chen, Yu Tian, and Zhaoxia Yin shield social media photos from AI snooping. Modern vision language models can guess private traits, like age or job, from a casual picture. Older shields either wrecked the photo's looks or broke the site's own image features.
Their method tunes a photo guard that hides private traits while keeping harmless content readable and the image looking the same. To judge fairly, they built VPI-COCO, a public set of 522 images with layered privacy questions and clean twin questions. Across several models, their guard pushed private guesses below 25 percent while harmless answers stayed above 88 percent, and it held up on fresh, reworded questions.
Catalog record
| CREATOR | Yucheng Fan, Jiawei Chen, Yu Tian, Zhaoxia Yin, [0.5em] East China Normal University, Shanghai, China |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 45 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2512-18264
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Who Can See Through You? Adversarial Shielding Against VLM-Based Attribute Inference Attacks Yucheng Fan, Jiawei Chen, Yu Tian, Zhaoxia Yin, [0.5em] East China Normal University, Shanghai, China Abstract As vision-language models (VLMs) become widely adopted, VLM-based attribute inference attacks have emerged as a serious privacy concern, enabling adversaries to infer private attributes from images shared on social media. This escalating threat calls for dedicated protection methods to safeguard user privacy. However, existing methods often degrade the visual quality of images or interfere with vision-based functions on social media, thereby failing to achieve a desirable balance between privacy protection and user experience. To address this challenge, we propose a novel protection method that jointly optimizes privacy suppression and utility preservation under a visual consistency constraint. While our method is conceptually effective, fair comparisons between methods remain challenging due to the lack of publicly available evaluation datasets. To fill this gap, we introduce VPI-COCO, a publicly available benchmark comprising 522 images with hierarchically structured privacy questions and corresponding non-private counterparts, enabling fine-grained and joint evaluation of protection methods in terms of privacy preservation and user experience. Building upon this benchmark, experiments on multiple VLMs demonstrate that our method effectively reduces PAR below 25%, keeps NPAR above 88%, maintains high visual consistency, and generalizes well to unseen and paraphrased privacy questions, demonstrating its strong practical applicability for real-world VLM deployments. 1 Introduction In recent years, the rapid progress of Vision-Language Models (VLMs) has emerged as one of the most remarkable advances in Artificial Intelligence. By integrating visual and linguistic modalities, VLMs achieve powerful cross-modal understanding and generalization across real-world scenarios. However, their growing deployment also raises increasingly diverse and complex concerns about privacy and regulatory compliance 5 ; 30 ; 20 ; 29 . Figure 1 : Illustration of VLM-based attribute inference attack. Users often share daily-life photos on social media. Attackers can exploit VLMs to infer personal privacy attributes from visual cues, even when such attributes are never explicitly disclosed. In particular, VLM-based attribute inference attacks have attracted increasing attention. Specifically, 16 ; 8 ; 14 ; 6 demonstrated that VLMs can infer an image’s geographic location from subtle visual cues without geo-annotations, whereas 22 ; 13 ; 28 further showed that they can uncover private attributes from ordinary images through semantic cues. An illustrative example of such attacks is shown in Fig. 1 , users on social platforms share volumes of daily-life images. By automatically collecting these images, attackers can leverage VLMs to analyze visual content and uncover latent private attributes about users. Such attacks are highly stealthy, low-cost, and scalable, posing an emerging threat to online privacy and security. To address these concerns, prior efforts have focused on enhancing model-level safety alignment and content filtering 18 . However, these safeguards can be easily circumvented by simple evasion techniques 2 ; 15 , rendering them ineffective against these attacks. This calls for an input-level privacy protection mechanism to defend against such threats, which poses two key challenges: (1) Trade-off between privacy protection and user experience. An effective privacy protection method is expected to induce the model to refuse privacy questions, thereby preventing potential privacy leakage. Meanwhile, it should also minimize unintended refusals of non-privacy questions that arise from the protection process to ensure that VLM-powered functions on social platforms, such as tagging and content recommendation, remain responsive for a satisfactory user experience. Existing input-level image privacy protection methods mainly fall into two categories: anonymization-based protection 17 ; 23 ; 19 ; 27 and encryption-based protection 32 ; 31 ; 4 ; 21 . The former anonymizes images by masking or replacing privacy-sensitive regions, while the latter enhances privacy by adding noise or applying image transformations to the input. However, these approaches often cause significant degradation in visual quality and overlook their potential negative impact on non-privacy questions, thereby failing to achieve a trade-off between privacy protection and user experience. (2) Lack of publicly available targeted evaluation datasets. Existing related datasets 22 ; 13 ; 16 ; 28 exhibit significant limitations in evaluating protection methods against VLM-based attribute inference attack. (1) Restricted Public Accessibility. Most existing datasets rely on privacy annotations collected from real individuals, making public release difficult due to privacy regulations and thus hindering standardized comparison across protection methods. (2) Lack of Targeted Privacy Annotation. Most existing datasets adopt explicit privacy questions that directly ask about private attributes, failing to reflect realistic inference-based attacks, where adversaries leverage contextual semantics such as scene, co-occurring objects, or human activities 13 . (3) No Support for Privacy–Utility Joint Evaluation. Most existing datasets typically lack non-privacy question annotations, hindering comprehensive evaluation of the trade-off between privacy protection and user experience, and thus making it difficult to measure their practicality in real-world deployment. To address the above challenges, we make the following key technical contributions as summarized below: (1) Input-level joint protection method balancing privacy and user experience. This work proposes an input-level joint protection method that formulates the protection process as a dual-objective optimization targeting privacy protection and user experience. To this end, it jointly optimizes a privacy suppression loss and a utility preservation loss under a visual-consistency constraint, achieving a balanced trade-off between the two objectives. (2) VPI-COCO: A Public, Fine-Grained, and Joint Evaluation Benchmark. We construct Visual Privacy Inference COCO (VPI-COCO), a public benchmark derived from the COCO 11 , containing 522 carefully selected images. Following three key principles of privacy-non-disclosive labeling, hierarchical privacy question design, and non-privacy question integration, VPI-COCO effectively overcomes the limitations of existing datasets, providing a unified benchmark for evaluating protection methods against VLM-based attribute inference attack. (3) Comprehensive Experimental Evaluation and Analysis. Experiments validate the effectiveness of the proposed protection method across multiple VLMs. It achieves the lowest Privacy Answer Rate (PAR) of 17.3% and the highest Non-Privacy Answer Rate (NPAR) of 92.6% on average, while maintaining high visual fidelity (PSNR 35.6 dB), transferability across paraphrased and unseen questions, confirming its practicality for real-world deployment. 2 Related Work 2.1 Traditional Attribute Inference Attack Attribute Inference Attack (AIA) aims to infer an individual’s privacy attributes by analyzing accessible data, even when such information is not explicitly disclosed 26 . Lindamood et al. 12 introduced an early scalable attack that infers hidden user attributes from social network data using a modified Bayes model. Weinsberg et al. 25 demonstrated that a recommender system can accurately infer users’ gender solely from their movie rating patterns. 2.2 VLM-Based Attribute Inference Attack Vision-language models (VLMs) extend AIA to the visual domain, extracting privacy attributes from images through cross-modal reasoning and implicit visual cues. Specifically, 16 ; 8 ; 14 ; 6 demonstrated that VLMs are capable of inferring an image’s geographic location by leveraging subtle visual cues. In a broader context, 22 ; 13 ; 28 further revealed that VLMs are also capable of uncovering private attributes from seemingly ordinary images by leveraging latent semantic cues embedded in the visual content. 2.3 Input-Level Image Privacy Protection Method Recent efforts have explored input-level protection methods to safeguard privacy information in images. These methods mainly fall into two categories: anonymization-based protection 17 ; 23 ; 19 ; 27 and encryption-based protection 32 ; 31 ; 4 ; 21 . However, these approaches often cause significant