Understanding Privacy Norms Around LLM-Based Chatbots: A Contextual Integrity Perspective
Sarah Tran, Hongfan Lu, Isaac Slaughter, Bernease Herman, Aayushi Dangol, Yue Fu, Lufei Chen, Biniyam Gebreyohannes, Bill Howe, Alexis Hiniker, Nicholas Weber, Robert Wolfe · 2025
Read Understanding Privacy Norms Around LLM-Based Chatbots: A Contextual Integrity Perspective by Sarah Tran, Hongfan Lu, Isaac Slaughter, Bernease Herman, Aayushi Dangol, Yue Fu, Lufei Chen, Biniyam Gebreyohannes, Bill Howe, Alexis Hiniker, Nicholas Weber, Robert Wolfe (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2508-06760.txt.
Introduction
Sarah Tran and co-authors survey how users of AI chatbots think about the privacy of their chats. Their 2025 study ran an experiment with three hundred United States users of ChatGPT, viewing the question through a lens called contextual integrity. That lens asks whether data flows fit the norms of the setting they occur in.
A wide gap showed up between worry and behavior. Over four fifths called their chats sensitive, more than email or social posts, yet nearly half had discussed health and over a third had discussed money with the bot. Users voiced strong concern and doubted their words would stay private. Still, they firmly rejected sharing their wider personal data with others.
Catalog record
| CREATOR | Sarah Tran, Hongfan Lu, Isaac Slaughter, Bernease Herman, Aayushi Dangol, Yue Fu, Lufei Chen, Biniyam Gebreyohannes, Bill Howe, Alexis Hiniker, Nicholas Weber, Robert Wolfe |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 60 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2508-06760
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Understanding Privacy Norms Around LLM-Based Chatbots: A Contextual Integrity Perspective Sarah Tran, Hongfan Lu, Isaac Slaughter, Bernease Herman, Aayushi Dangol, Yue Fu, Lufei Chen, Biniyam Gebreyohannes, Bill Howe, Alexis Hiniker, Nicholas Weber, Robert Wolfe Abstract LLM-driven chatbots like ChatGPT have created large volumes of conversational data, but little is known about how user privacy expectations are evolving with this technology. We conduct a survey experiment with 300 US ChatGPT users to understand emerging privacy norms for sharing chatbot data. Our findings reveal a stark disconnect between user concerns and behavior: 82% of respondents rated chatbot conversations as sensitive or highly sensitive — more than email or social media posts — but nearly half reported discussing health topics and over one-third discussed personal finances with ChatGPT. Participants expressed strong privacy concerns (t(299) = 8.5, p < < .01) and doubted their conversations would remain private (t(299) = -6.9, p < < .01). Despite this, respondents uniformly rejected sharing personal data (search history, emails, device access) for improved services, even in exchange for premium features worth $200. To identify which factors influence appropriate chatbot data sharing, we presented participants with factorial vignettes manipulating seven contextual factors. Linear mixed models revealed that only the transmission factors such as informed consent, data anonymization, or the removal of personally identifiable information, significantly affected perceptions of appropriateness and concern for data access. Surprisingly, contextual factors including the recipient of the data (hospital vs. tech company), purpose (research vs. advertising), type of content, and geographic location did not show significant effects. Our results suggest that users apply consistent baseline privacy expectations to chatbot data, prioritizing procedural safeguards over recipient trustworthiness. This has important implications for emerging agentic AI systems that assume user willingness to integrate personal data across platforms. Introduction Figure 1: We used the contextual integrity framework to construct factorial vignettes rated by participants based on whether they were appropriate or concerning. This approach allowed us to isolate factors associated with the transmission principle - the set of norms and expectations surrounding data exchange - as the primary determinants of privacy norms for chatbot data. Conversational AI systems like ChatGPT have rapidly become a source of user-generated data, with over 400 million weekly active users creating billions of chat interactions ( Beaty 2025 ) . This conversational data represents a fundamentally new category of personal information — more intimate than search queries, more extensive than social media posts, and often containing highly sensitive disclosures about health, finances, and personal relationships. The commercial and strategic value of chatlog data spans targeted advertising ( Matz et al. 2024 ) , personalized interfaces ( Chen et al. 2024 ) , training next-generation language models ( Nasr et al. 2023 ; Nasr et al. 2025 ) , and even geopolitical leverage when nations control chat data from other countries’ citizens ( Mok 2025 ; Burgess and Newman 2025 ) . Yet users remain uncertain about how their conversational data is handled. While chatbot providers may offer privacy protections through terms of service agreements, research indicates most users neither understand these policies nor trust that their data will be used responsibly. A 2023 Pew Research Center study found that large majorities of Americans familiar with AI worry about personal information being used in “unintended ways and ways people are not comfortable with” ( Faverio and Tyson 2023 ) . These concerns have proved prescient: In December 2024, the chatbot provider WotNot inadvertently exposed over 340,000 private chat logs through an unprotected cloud system ( Arntz 2024 ) , and in August 2025, researchers discovered that Google was indexing and making discoverable ChatGPT conversations that users had inadvertently shared - exposing passport numbers, medical records, and employment histories ( Cox 2025 ) . Despite the apparent risks, prior research finds that users of AI chatbots often disclose sensitive information in chat logs, sometimes relying on flawed mental models of how data will be processed and retained by a model’s provider ( Zhang et al. 2024 ) . Indeed, the presence of user social security numbers and other personally identifiable information (PII) resulted in the withdrawal of ShareGPT, an open dataset of interactions with ChatGPT used in prior privacy studies ( Zhang et al. 2024 ) . The apparent gap between many users’ behavior when interacting with chatbots ( e.g., sharing sensitive PII) and the public’s expressed concern with the privacy of chat data motivates a systematic investigation of privacy norms surrounding user chatbot data. Understanding such norms can help inform the design of chat interfaces as well as policy-level considerations for chat data. The present work thus investigates chatbot privacy norms using the contextual integrity framework, which posits that privacy norms exist in the context of information flows . In short, an information exchange can be reduced to a set of transmission principles that govern the exchange of a certain kind of information (content) with a certain entity (role) for a certain reason (purpose)- collectively these influence whether the exchange of data is appropriate, or a cause for concern ( Nissenbaum 2004 ; Malkin 2022 ) . Drawing on this theoretical framework, we set out to answer three research questions: 1. RQ1: How do users perceive the sensitivity of chatbot data, and how does this compare to their perception of other forms of sensitive personal information? 2. RQ2: What factors ( e.g., content, purpose, and consent, drawing on a contextual integrity framework) influence users’ concerns about the handling of their chatbot data? 3. RQ3: How willing are users to integrate chatbot data with data from their device, browser, and social accounts to improve their online experience - either via personalization or by gaining access to premium features? To address these questions, we drew on methods introduced in past work that addressed the contextual integrity of social media information flows ( Gilbert, Vitak, and Shilton 2021 ) , deploying a large-scale survey experiment with N =300 ChatGPT users residing in the U.S. In addition to a battery of questions that directly probe privacy attitudes, each respondent was also presented with 30 vignettes, short stories that each include a randomly drawn level from seven factors of interest, as illustrated in Figure 1 . These vignettes allowed us to isolate the factors that influence the two dependent variables rated by respondents: the appropriateness of the information flow, and the level of concern associated with it. Based on the results of our data analysis, we make three contributions: 1. Most respondents (82%) consider chatbot conversations sensitive data, and on average agree that they have privacy concerns about ChatGPT conversations. Respondents also disagreed on average that their conversations with chatbots like ChatGPT would remain private. Furthermore, respondents agreed on average that they were concerned about people receiving inaccurate information from AI; not understanding what AI can do; and having personal information misused by AI. 2. Only Informed Consent , Anonymity , and Privacy (elements of a transmission principle) influence appropriateness perceptions. Fitting linear mixed models (LMMs) to vignette ratings reveals that only providing informed consent, anonymizing data, and removing personal information improve respondents’ perceptions of the appropriateness of the information flows described by our vignettes. The Role, Purpose, Content Type, and Location factors had no detectable effect