Semantically-Aware LLM Agent to Enhance Privacy in Conversational AI Services
Jayden Serenari, Stephen Lee · 2025
Read Semantically-Aware LLM Agent to Enhance Privacy in Conversational AI Services by Jayden Serenari, Stephen Lee (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2510-27016.txt.
Introduction
Jayden Serenari and Stephen Lee present a privacy guard for chats with large language models. Users often paste personal details such as names and addresses into prompts, and that data can leak. Their framework, called LOPSIDED, swaps sensitive entities for realistic stand-ins that fit the surrounding meaning before anything leaves the device.
Once the remote model replies, the stand-ins are swapped back, so the user sees a normal answer. Tests use real shared conversations, expanded and labeled to check whether each entity matters for the reply. The method cuts semantic utility errors fivefold versus baseline techniques while improving privacy. For anyone building chat services, it offers a way to protect users without dulling answers.
Catalog record
| CREATOR | Jayden Serenari, Stephen Lee |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 27 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2510-27016
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Semantically-Aware LLM Agent to Enhance Privacy in Conversational AI Services Jayden Serenari, Stephen Lee Abstract With the increasing use of conversational AI systems, there is growing concern over privacy leaks, especially when users share sensitive personal data in interactions with Large Language Models (LLMs). Conversations shared with these models may contain Personally Identifiable Information (PII), which, if exposed, could lead to security breaches or identity theft. To address this challenge, we present the Local Optimizations for Pseudonymization with Semantic Integrity Directed Entity Detection (LOPSIDED) framework, a semantically-aware privacy agent designed to safeguard sensitive PII data when using remote LLMs. Unlike prior work that often degrade response quality, our approach dynamically replaces sensitive PII entities in user prompts with semantically consistent pseudonyms, preserving the contextual integrity of conversations. Once the model generates its response, the pseudonyms are automatically depseudonymized, ensuring the user receives an accurate, privacy-preserving output. We evaluate our approach using real-world conversations sourced from ShareGPT, which we further augment and annotate to assess whether named entities are contextually relevant to the model’s response. Our results show that LOPSIDED reduces semantic utility errors by a factor of 5 compared to baseline techniques, all while enhancing privacy. I Introduction Conversational AI systems are rapidly being adopted across domains ranging from customer service and healthcare to creative writing and programming assistance. These systems rely on Large Language Models (LLMs) hosted on remote servers, requiring user inputs that may include potentially sensitive data, to be transmitted for inference. When users unknowingly share Personally Identifiable Information (PII), such as names and addresses, they expose themselves to privacy risks, regulatory liabilities, and possible data breaches [ 1 ] . Fig. 1 : The LOPSIDED privacy agent system design. Pseudonymization is a common approach [ 2 ] used to protect users’ privacy by replacing PII with entities of the same class. For example, a city name like Chicago might be substituted with Los Angeles to obscure the original data while maintaining the overall structure of the input. However, if a system relies on specific details for accuracy, altering key information may lead to misleading or incorrect results. For instance, if a user asks, ”What is the population of Chicago?” and the system modifies it to ”What is the population of Los Angeles?”, the semantic integrity of the query is compromised. This highlights a key challenge in privacy-preserving techniques — ensuring that user data remains protected without distorting the intended semantic meaning of their input. More recently, LLMs have been explored for PII removal in conversational AI services [ 3 ] . For example, Hide and Seek (HAS) [ 4 ] anonymizes any PII within a prompt before it is transmitted to a cloud-based language model and then de-anonymizes the LLM’s response. This approach is subject to the same pitfall as stated previously. This gap motivates a key research question: How can we preserve user privacy through pseudonymization while ensuring that LLM responses remain semantically faithful to the original query? Currently, all techniques rely on an all-or-nothing approach , where all private entities are removed or none are. This can render the system unusable for users, as they may not be able to effectively use the tool if essential information is removed. Our key insight is to develop a more nuanced approach that selectively pseudonymizes PII data and generates a semantically similar response. For example, if a user asks about the weather in Palo Alto, it could be replaced with San Jose, maintaining privacy while preserving the accuracy of the response. To address this challenge, we propose LOPSIDED, a lightweight framework that balances PII removal and semantic response preservation. Our work focuses on maximizing user privacy by locally pseudonymizing sensitive information before it is transmitted to remote LLMs. The privacy agent, shown in Figure 1 , operates as an intermediary between the user and the cloud. It intercepts the request, pseudonymizes the prompt by replacing PII, and de-anonymizes it before presenting it to the user. This ensures that privacy-sensitive data is not exposed, while maintaining the relevance of the system’s response. We note that there are situations where a replacement could completely alter the meaning. In such cases, we prioritize maintaining the utility of the response. Since pseudonymization must occur locally, we explore the use of small models that can be deployed on the user’s device. Our key contributions are as follows: LOPSIDED Design : We formulate the problem of semantic-aware privacy for conversational AI, where the goal is to pseudonymize named entities while preserving the utility of the LLM response. To address this problem, we introduce LOPSIDED, a framework which ensures that named entities can be modified without disrupting the semantic integrity of the response, making it both privacy-preserving and semantically accurate. Semantic-aware Privacy Dataset : We augment the ShareGPT dataset, which contains real-world ChatGPT conversation histories, by annotating named entities for relevance to the prompt. This results a novel 866-sample evaluation dataset, designed for testing semantic-aware privacy agents. Evaluation and Analysis : We evaluate our technique using real-world prompts and compare it against several baseline methods. Our analysis shows that prior work often prioritizes privacy at the expense of utility, while our approach reduces utility errors by a factor of 5 while still enhancing privacy. II Background Personally Identifiable Information refers to any information that can be used to identify an individual, either directly or indirectly. PII is a key concept in privacy and data protection, as its exposure can lead to identity theft, fraud, and other security risks [ 5 , 6 ] . The definition of what constitutes PII can vary, but generally, it includes both direct and indirect identifiers [ 7 ] . PII can be used in many malicious attacks, including the recent uptick in “Pig Butchering” scams [ 8 ] . Attackers can use PII to lure unsuspecting victims into a false state of trust by using personal information against them, such as the identities of close family, workplace details, or contact information. Prior studies have highlighted that identifying PII is a significant challenge [ 9 , 7 ] . To address these challenges, most techniques rely on Named Entity Recognition (NER), a method used to identify and classify entities such as names, locations, and organizations within text. Existing methods, such as SpaCy [ 10 ] and NLTK [ 11 ] , leverage language models to perform NER. These models identify entities such as names, locations, organizations, and other relevant categories, classifying them into predefined labels like [PERSON] or [GPE] (Geopolitical Entity). Prior work has adopted spaCy as part of their pipeline to identify these named entities [ 4 ] . III Related Work Prior work has extensively studied the leakage of PII and privacy breaches through traffic originating from mobile devices and websites [ 12 ] . These studies highlight how data transmitted over networks can expose sensitive information. Such leaks enable third parties — including advertisers, data brokers, and malicious actors — to track users across different services. Such information can be correlated to build detailed user profiles without explicit consent. Recent research has also highlighted similar privacy risks arising from LLMs [ 13 ] . Today’s growing reliance on LLMs introduces new challenges, as these models can