PrivacyBench : A Conversational Benchmark for Evaluating Privacy in Personalized AI Conference: ; ; CCS: Security and privacy Privacy protections CCS: Security and privacy Usability in security and privacy CCS: Human-centered computing HCI theory, concepts and models
Srija Mukhopadhyay, Sathwik Reddy, Shruthi Muthukumar, Jisun An, Ponnurangam Kumaraguru · 2025
Read PrivacyBench : A Conversational Benchmark for Evaluating Privacy in Personalized AI Conference: ; ; CCS: Security and privacy Privacy protections CCS: Security and privacy Usability in security and privacy CCS: Human-centered computing HCI theory, concepts and models by Srija Mukhopadhyay, Sathwik Reddy, Shruthi Muthukumar, Jisun An, Ponnurangam Kumaraguru (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2512-24848.txt.
Introduction
Srija Mukhopadhyay, Sathwik Reddy, Shruthi Muthukumar, and colleagues introduce PrivacyBench, a conversational benchmark testing whether personalized AI assistants keep user secrets. Such agents read private email, chats, and purchases, yet may leak sensitive facts when social context calls for discretion. Their datasets embed secrets in socially grounded material and probe them over multiple turns.
Retrieval-augmented assistants leaked secrets in up to about a quarter of interactions in their tests. A privacy-aware prompting approach is also examined. Readers building personal agents get a measurable warning and a test harness. Keeping secrets across a conversation is shown to be a distinct skill from answering questions well.
Catalog record
| CREATOR | Srija Mukhopadhyay, Sathwik Reddy, Shruthi Muthukumar, Jisun An, Ponnurangam Kumaraguru |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 52 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2512-24848
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
PrivacyBench : A Conversational Benchmark for Evaluating Privacy in Personalized AI Conference: ; ; CCS: Security and privacy Privacy protections CCS: Security and privacy Usability in security and privacy CCS: Human-centered computing HCI theory, concepts and models Srija Mukhopadhyay, Sathwik Reddy, Shruthi Muthukumar, Jisun An, Ponnurangam Kumaraguru Abstract . Personalized AI agents rely on access to a user’s digital footprint, which often includes sensitive data from private emails, chats and purchase histories. Yet this access creates a fundamental societal and privacy risk: systems lacking social-context awareness can unintentionally expose user secrets, threatening digital well-being. We introduce PrivacyBench , a benchmark with socially grounded datasets containing embedded secrets and a multi-turn conversational evaluation to measure secret preservation. Testing Retrieval-Augmented Generation (RAG) assistants reveals that they leak secrets in up to 26.56% of interactions. A privacy-aware prompt lowers leakage to 5.12%, yet this measure offers only partial mitigation. The retrieval mechanism continues to access sensitive data indiscriminately, which shifts the entire burden of privacy preservation onto the generator. This creates a single point of failure, rendering current architectures unsafe for wide-scale deployment. Our findings underscore the urgent need for structural, privacy-by-design safeguards to ensure an ethical and inclusive web for everyone. Code Dataset Figure 1. PrivacyBench provides a critical resource for pushing the boundaries of personalized generation, enabling research into systems that are not only accurate but also temporally adaptive, contextually aware, and respectful of social contexts. A user privately discusses resignation plans with a colleague, and later tasks their AI assistant with the composition of a professional email to a manager regarding "future goals." Driven by an optimization for semantic relevance, the system fetches the resignation chat and integrates the secret into the professional draft: ``My main priority is the preparation for a smooth transition, as I have accepted a new role elsewhere.'' This failure exposes the inadequacy of static benchmarks and highlights the urgent need for the assessment of privacy preservation within realistic, dynamic dialogues. 1. Introduction The rapid advancement of Artificial Intelligence (AI), specifically Large Language Models (LLMs), has catalyzed a transition towards hyper-personalization, enabling AI assistants to leverage a user’s entire digital footprint for tailored responses and actions ( 35 ; 33 ) . Emerging agentic systems, such as Pin AI or Notion AI autonomously execute tasks and interact with users’ online environments. While integrating these agents into daily workflows offers substantial potential to improve societal productivity and individual empowerment, it inevitably grants them access to highly sensitive personal data. Consequently, the central challenge for the next generation of responsible web assistants lies in reconciling the immense utility of personalization with the ethical imperative of preserving user privacy ( 28 ; 34 ) . The deployment of personalized assistants typically follows a standard architectural paradigm ( 27 ; 25 ) . In this setting, a dedicated LLM-based agent serves a primary user by maintaining continuous access to their evolving digital footprint, a dynamic repository that includes private chats, professional emails, and online purchase histories. To manage the scale of this data, Retrieval-Augmented Generation (RAG) serves as the structural backbone, which allows for the dynamic retrieval of historical context without the prohibitive cost of model retraining. However, this design fundamentally treats the user’s diverse web activity as a flat, unified knowledge base. By unifying these distinct data streams, the system views all information as equally retrievable, disregarding the implicit social boundaries and norms that governed their original creation. This disregard for social context directly violates the core principles of Contextual Integrity Theory ( 21 ) , which defines privacy as the adherence to appropriate information flow norms. While recent studies highlight the struggle of LLMs with the enforcement of these boundaries ( 13 ; 19 ) , the risk is significantly amplified in personalized assistants that aggregate data from various sources. The architectural inability to distinguish between public data and private confessions leads to a potential for leaking sensitive information shared in confidence to unintended audiences or inappropriate contexts. Such breaches pose severe risks to the digital well-being of users. This ethical challenge drives our central research question: To what extent do current personalized assistants maintain privacy boundaries during realistic, multi-turn interactions? While personalized agents operate primarily in dynamic, multi-turn contexts, existing safety evaluations focus disproportionately on static, single-turn queries. This methodological gap is critical: the preservation of privacy is significantly more difficult in fluid interactions where the gradual accumulation of context facilitates the bypassing of standard safety filters ( 2 ; 14 ) . As a dialogue progresses, the retrieval mechanism continuously injects historical user data for the maintenance of conversational continuity, a process that effectively blurs the distinction between relevant context and private secrets. Figure 1 illustrates this failure in a realistic workflow: a user privately discusses resignation plans with a colleague, and later tasks their AI assistant with the composition of a professional email to a manager regarding "future goals." Driven by an optimization for semantic relevance, the system fetches the resignation chat and integrates the secret into the professional draft: “My main priority is the preparation for a smooth transition, as I have accepted a new role elsewhere.” This failure exposes the inadequacy of static benchmarks and highlights the urgent need for the assessment of privacy preservation within realistic, dynamic dialogues. However, effective personalization requires more than just preventing leaks; agents must also maintain utility by facilitating appropriate information sharing with authorized individuals. This dual objective creates a tension between two distinct failure modes: leakage, the disclosure of a secret to unauthorized parties, and over-secrecy, the unwarranted withholding of information from trusted confidant. Thus, the ultimate goal is not total data lockdown, but the preservation of Contextual Integrity: ensuring that information flows align precisely with a user’s nuanced social norms ( 21 ) . Addressing these challenges demands a new evaluation paradigm. Prior work on LLM personalization, such as LaMP ( 25 ) , LongLaMP ( 12 ) , and PersonaBench ( 27 ) , primarily focuses on tasks like persona-consistent text generation and recommendation. However, these benchmarks define success solely by functional quality, thereby overlooking the critical privacy risks. Specifically, they lack two essential components for robust privacy evaluation: (1) ground truth secrets to quantify Contextual Integrity failures like leakage or over-secrecy; and (2) multi-turn interactions to capture privacy erosion during realistic conversations. To fill these gaps, we introduce PrivacyBench , a novel framework designed to generate evaluation benchmarks with ground-truth secrets embedded in realistic social contexts. Using this framework, we conduct a multi-turn conversational evaluation of five state-of-the-art models. Our findings reveal a critical vulnerability: without explicit safeguards, personalized assistants leaked secrets in 15.80% of conversations. Furthermore, we identify a practical defense; adding a simple privacy-aware system prompt mitigated this risk, significantly reducing the average leakage rate to 5.12%. To our knowledge, this is the first quantitative analysis of this failure mode, providing a foundational baseline for developing future privacy safeguards. Our key contributions are: • PrivacyBench : We introduce the first benchmark designed for the scalable audit of contextual privacy, featuring socially grounded contexts and ground-truth secrets to rigorously evaluate information flow norms. • Dynamic Evaluation Framework: We propose a multi-turn conversational framework that moves beyond static queries to assess privacy erosion