Unsafe2Safe: Controllable Image Anonymization for Downstream Utility
Minh Dinh, SouYoung Jin · 2026
Read Unsafe2Safe: Controllable Image Anonymization for Downstream Utility by Minh Dinh, SouYoung Jin (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2603-28605.txt.
Introduction
Dinh and Jin present Unsafe2Safe, a pipeline that anonymizes images while keeping them useful for training. Large image datasets often hold identifiable or sensitive content. Their system uses a vision-language model to flag privacy risks. It writes paired private and public captions and turns the public one into neutral edit instructions. Diffusion editors then rewrite only the sensitive regions. Across three datasets, face and text similarity and demographic predictability fell a lot. Downstream accuracy stayed close to training on raw data.
Catalog record
| CREATOR | Minh Dinh, SouYoung Jin |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 77507 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2603.28605
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Unsafe2Safe: Controllable Image Anonymization for Downstream Utility Minh Dinh, SouYoung Jin Abstract Large-scale image datasets frequently contain identifiable or sensitive content, raising privacy risks when training models that may memorize and leak such information. We present Unsafe2Safe 1 1 1 https://see-ai-lab.github.io/unsafe2safe/ , a fully automated pipeline that detects privacy-prone images and rewrites only their sensitive regions using multimodally guided diffusion editing. Unsafe2Safe operates in two stages. Stage 1 uses a vision–language model to (i) inspect images for privacy risks, (ii) generate paired private and public captions that respectively include and omit sensitive attributes, and (iii) prompt a large language model to produce structured, identity-neutral edit instructions conditioned on the public caption. Stage 2 employs instruction-driven diffusion editors to apply these dual textual prompts, producing privacy-safe images that preserve global structure and task-relevant semantics while neutralizing private content. To measure anonymization quality, we introduce a unified evaluation suite covering Quality , Cheating , Privacy , and Utility dimensions. Across MS-COCO, Caltech101 and MIT Indoor67, Unsafe2Safe reduces face similarity, text similarity, and demographic predictability by large margins, while maintaining downstream model accuracy comparable to training on raw data. Fine-tuning diffusion editors on our automatically generated triplets (private caption, public caption, edit instruction) further improves both privacy protection and semantic fidelity. Unsafe2Safe provides a scalable, principled solution for constructing large, privacy-safe datasets without sacrificing visual consistency or downstream utility. 1 Introduction Figure 1 : Examples from Unsafe2Safe (U2S). For each case, the model converts an unsafe image into a privacy-preserving safe version. Examples demonstrate key capabilities that may appear simultaneously: (1) structure-preserving full body anonymization, (2) demographic neutralization (race entropy \uparrow ), and (3) obfuscation of non-human confidential details. Large-scale image datasets underpin modern computer vision, yet they contain faces, ID badges, personal documents, or other signals that expose individuals to privacy risks. These risks are amplified by the tendency of deep networks to memorize training data 43 ; 50 and thus are vulnerable to sensitive information extraction through model inversion 31 ; 41 or membership attack 15 ; 39 . As datasets grow larger and more heterogeneous, exhaustive manual auditing becomes infeasible, forming a barrier to sharing data and training models responsibly at scale. Existing anonymization methods offer only partial protection. Preceeding redaction, e.g. blurring, masking, and inpainting, there is often a detector that often overlooks privacy cues outside predefined regions and introduces artifacts that impair downstream utility. The core challenge is that anonymization is not merely a removal problem: sensitive content must be rewritten so that identity cues are suppressed without destroying the semantic structure needed for learning. An effective solution must therefore combine strong privacy protection with semantic fidelity and broad applicability in open-domain images, without relying on expensive manual annotation. Anonymization thus requires fine-grained control over which pixels to edit and how to change them without compromising non-private semantics. In this work, we introduce Unsafe2Safe (U2S) , a scalable anonymization framework that uses multimodal reasoning and text-guided diffusion to produce privacy-safe yet utility-preserving versions of image datasets. Our key idea is to combine two complementary textual signals that jointly capture the semantics of the original scene while specifying how sensitive content should be altered. The overall process proceeds in two steps: first generating privacy-aware textual guidance, and then applying controlled image editing conditioned on this guidance. Firstly, a vision–language model (VLM) inspects each image using predefined privacy criteria and flags those containing sensitive content. For unsafe images, it produces two captions: a private caption describing the full scene and a public caption that removes identity-specific details while preserving non-sensitive semantics. As the public caption specifies the desired safe outcome but not the transformation itself, a large language model (LLM) further generates a structured edit instruction that proposes identity-neutral substitutions and defines the required modifications. Together, the public caption and edit instruction determine what should be preserved and what should be altered. Conditioned on these textual signals, a text-guided diffusion editor performs targeted anonymization while maintaining visual coherence. The public caption acts as a semantic anchor that preserves global layout and object relationships, whereas the edit instruction guides geometry- and style-consistent rewriting of sensitive attributes. Unlike detector-based pipelines, Unsafe2Safe does not require segmentation masks, attribute labels, or predefined privacy taxonomies. Instead, it leverages modern diffusion editors such as InstructPix2Pix 7 and FlowEdit 24 , which support multi-prompt conditioning to modify only the regions implied by the instructions. This design yields privacy-neutral reconstructions, removing identity cues and other sensitive or unsafe attributes while retaining structural fidelity and avoiding the semantic drift often observed in full image regeneration. Figure 1 illustrates how Unsafe2Safe selectively anonymizes sensitive regions while preserving recognizability and downstream task relevance. Through extensive experiments on Caltech-101 13 , MIT Indoor-67 37 , MS-COCO 29 and OK-VQA 32 , we show that Unsafe2Safe achieves strong anonymization while maintaining or even improving downstream task performance. Models trained on Unsafe2Safe data match the accuracy of models trained on raw images, and in some cases exceed them due to removal of spurious correlations. Unsafe2Safe also reduces face similarity and text similarity, increases demographic diversity, and avoids semantic drift. Compared to baseline anonymization techniques, our approach leverages multimodal reasoning, dual-caption conditioning, and targeted diffusion editing, which together provide finer control, higher fidelity, and better scalability. Our main contributions are summarized as follows: (1) We introduce Unsafe2Safe , a controllable privacy-preserving diffusion framework that combines VLM-guided privacy inspection, public caption generation, and LLM-derived edit instructions to remove identity-sensitive content while preserving task-relevant semantics and spatial layout. (2) We develop a scalable anonymization pipeline and dataset construction process that enables safe proxy generation across diverse domains. We release privacy-safe generated datasets along with tools for anonymizing additional datasets, supporting reproducibility and broad community adoption. (3) We propose a unified evaluation framework for anonymization, introducing four metric groups that jointly quantify image quality, leakage pathways, demographic diversity, and downstream utility. (4) Through extensive experiments on Caltech-101, MIT Indoor-67, MS-COCO, and OK-VQA , we show that Unsafe2Safe preserves or improves downstream accuracy while substantially reducing face similarity, text similarity, and demographic predictability , outperforming existing anonymization baselines. 2 Related Work 2.1 Privacy-Preserving Data Generation Our work follows a data-centric approach to privacy: instead of protecting models after training, we anonymize images beforehand so that sensitive information never enters model weights. Most anonymization pipelines rely on face or person detectors 22 ; 21 ; 25 ; 18 and then apply blurring, masking, pixelation 17 ; 33 , or generative inpainting 36 ; 21 . Others detect risks via embedding similarity 23 . Yet, their reliance on the closed-set detector limits their applicability to open-domain imagery where privacy-encoding signals extend far beyond faces. VLMs and LLMs 52 ; 8 ; 16 ; 3 offer a promising alternative for zero-shot privacy inspection and natural-language reasoning about sensitive content 46 ; 33 . However, existing pipelines use such models only for detection, without leveraging their semantic understanding to guide how sensitive content should be modified. Unsafe2Safe fills this gap by coupling VLM-based privacy inspection with LLM-generated edit instructions and executing the transformation using text-guided diffusion editing, fostering structured, context-aware anonymization in open-set categories. 2.2 Privacy Evaluation Frameworks Prior work measures privacy via re-identification attacks 25 ; 11 or identity/attribute classification degradation 9