Privacy Leakage Overshadowed by Views of AI: A Study on Human Oversight of Privacy in Language Model Agent
Zhiping Zhang, Bingcan Guo, Tianshi Li · 2024
Read Privacy Leakage Overshadowed by Views of AI: A Study on Human Oversight of Privacy in Language Model Agent by Zhiping Zhang, Bingcan Guo, Tianshi Li (2024) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2411-01344.txt.
Description
Abstract Language model (LM) agents that act on users’ behalf for personal tasks (e.g., replying emails) can boost productivity, but are also susceptible to unintended privacy leakage risks. We present the first study on people’s capacity to oversee the privacy implications of the LM agents.
Catalog record
| CREATOR | Zhiping Zhang, Bingcan Guo, Tianshi Li |
|---|---|
| YEAR | 2024 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 92 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2411-01344
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Privacy Leakage Overshadowed by Views of AI: A Study on Human Oversight of Privacy in Language Model Agent Zhiping Zhang, Bingcan Guo, Tianshi Li Abstract Language model (LM) agents that act on users’ behalf for personal tasks (e.g., replying emails) can boost productivity, but are also susceptible to unintended privacy leakage risks. We present the first study on people’s capacity to oversee the privacy implications of the LM agents. By conducting a task-based survey ( N = 300 N=300 ), we investigate how people react to and assess the response generated by LM agents for asynchronous interpersonal communication tasks, compared with a response they wrote. We found that people may favor the agent response with more privacy leakage over the response they drafted or consider both good, leading to an increased harmful disclosure from 15.7% to 55.0%. We further identified six privacy profiles to characterize distinct patterns of concerns, trust, and privacy preferences in LM agents. Our findings shed light on designing agentic systems that enable privacy-preserving interactions and achieve bidirectional alignment on privacy preferences to help users calibrate trust. 1 Introduction Recent advances in language models have led to new applications of Language Model agents (LM agents) such as AutoGPT [ 47 ] , AgentGPT [ 2 ] and personal.ai [ 21 ] . Unlike basic language models or non-agentic AI systems, LM agents are inherently endowed with agency, allowing them to (semi-)automatically handle complex real-world tasks, such as accessing and retrieving information from connected databases (e.g., a user’s calendar) to generate and reply to emails [ 40 , 63 ] . These LM agents free users from having to instruct the LM step by step, potentially increasing productivity. However, this increased agency also means that LM agents can make decisions with limited human supervision, which raises new privacy challenges, especially in interpersonal communication where the agents act on behalf of the user to share information with other people. What if LM agents share information the users did not intend to disclose? Prior studies found that even without malicious attackers, LM agents can have unintended privacy leakage in their actions [ 39 , 55 ] . For example, Shao et al. 2024 demonstrated a case where an LM agent accesses the user John’s calendar data to generate an email reply, which shares the information that John is “talking to a few companies about switching jobs” in an email to John’s manager without John’s explicit consent. This issue not only risks violating one’s own privacy but also impacts bystanders , as LM agents might inadvertently share information about other people in the users’ connected database. Such unintentional privacy leaks occur because LMs lack the ability to reason about and operate under contextual privacy norms [ 55 , 39 ] . Introducing human oversight to AI systems has generated interest among researchers and regulators [ 29 , 20 , 14 ] . The European AI Act has begun to discuss keeping human in the loop to prevent harm and ensure ethical alignment with human values [ 14 ] . In LM agents, allowing humans to monitor the actions of an LM agent, such as reviewing and editing the reply generated by an email agent, could potentially address the unintentional privacy leakage issues. However, employing human oversight over LM agents to prevent privacy leakage presents challenges [ 67 , 20 ] . People could hold flawed mental models of the LM systems, which leads to incorrect expectations of the systems’ behavior [ 74 , 32 ] . Furthermore, overtrust in technological systems, often associated with “misunderstanding the risks associated with an action” [ 66 ] , can result in people overlooking privacy risks and becoming vulnerable to privacy violations [ 75 , 68 ] . The subjectivity and ambiguity in privacy also raise a normative question: What should be defined as privacy leakage in text disclosure? According to the contextual integrity (CI) theory [ 42 ] , any information shared by LM agents that does not conform to the contextual privacy norms can be considered leakage. Some existing studies operationalized CI to measure privacy leakage in LM agents by defining privacy norms based on regulations or social norms [ 55 , 17 , 39 ] . These methods take into account the general moral values, while overlooking the fact that privacy preferences vary among individuals [ 41 , 64 ] . Gabriel 2020 introduced multiple approaches to measure human values for model alignment. The revealed preferences, which are directly reflected in a person’s sharing behavior, while easy to collect, could suffer from the privacy paradox and may not represent the users’ best interest. Alternatively, informed preferences refer to preferences that a person would have if they were fully informed and rational, which may better reflect an individual’s true intentions, though the informing process can be challenging to implement. The multiplicity of privacy preference definitions calls for a better understanding of their influences on the quantifiable privacy leakage, which further affects the the alignment efforts. Despite the research that uncovers the novel privacy threats related to LM agents [ 39 , 55 ] , there has been no in-depth investigation into the efficacy of human oversight in addressing this issue. In this work, we take the initial step to address this gap and formulate the following research questions: RQ1 What are people’s primary concerns with LM agents, and what is the role of privacy? RQ2 How well do people oversee privacy leakage in LM agents’ actions? What distinct patterns emerge among users? RQ3 How does privacy leakage measured based on revealed preferences, informed preferences, and general privacy norms differ? To answer these questions, we conducted a task-based online survey with 300 participants. Each participant was asked to complete an asynchronous interpersonal communication task, such as replying an email. As shown in Figure 1 , they were introduced to an LM agent and asked to choose between their own response and a response generated by an LM agent, which included sensitive items that violate contextual privacy norms. They were then asked to explain their reasoning. Afterward, participants were informed of the privacy tuples in the scenario and asked to rate the perceived harmfulness of the sensitive items, which we used as the ground truth to evaluate each individual’s privacy leakage. Figure 1 : In the study, we guide the participants to experience an LM agent by asking them to first draft a response based on the synthetic user data in the presented scenario, and then review and compare with a response generated by an LM agent. We show a participant’s drafted response, preference, and justification (P137) to demonstrate the cases where the user failed to recognize the privacy leakage caused by the AI. Our results show that participants often included fewer privacy leakages in their own draft, while many selected the response written by LM agents. 48.0% of participants preferred the LM agent’s response or considered both responses equally good, despite the agent’s response containing more privacy leakages, leading to an increase in privacy leakage from 15.7% to a range between 38.4% (only considering the strict preference of the agent’s response) and 55.0% (also considering the “both are good” selections). These findings suggest that