Deploying Privacy Guardrails for LLMs: A Comparative Analysis of Real-World Applications

Shubhi Asthana, Bing Zhang, Ruchi Mahindru, Chad DeLuca · 2025

Read Deploying Privacy Guardrails for LLMs: A Comparative Analysis of Real-World Applications by Shubhi Asthana, Bing Zhang, Ruchi Mahindru, Chad DeLuca (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2501-12456.txt.

Introduction

Shubhi Asthana and colleagues compared in 2025 two real deployments of a privacy guardrail system for large language models, called OneShield Privacy Guard. Enterprise use of these models must meet privacy laws and catch risks in both user inputs and model outputs. The study spans an enterprise system and an open-source project.

In the enterprise case, the guard spotted sensitive items such as names, dates, and phone numbers across twenty-six languages with a high score, beating known tools by up to twelve percent. In the open-source case, it flagged privacy risks in code pull requests and cut manual review effort by hundreds of hours in three months. Adaptability across settings is the main claim.

Catalog record

CREATORShubhi Asthana, Bing Zhang, Ruchi Mahindru, Chad DeLuca
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE34 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2501-12456

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Deploying Privacy Guardrails for LLMs: A Comparative Analysis of Real-World Applications Shubhi Asthana, Bing Zhang, Ruchi Mahindru, Chad DeLuca Abstract The adoption of Large Language Models (LLMs) has revolutionized AI applications but poses significant challenges in safeguarding user privacy. Ensuring compliance with privacy regulations such as GDPR and CCPA while addressing nuanced privacy risks requires robust and scalable frameworks. This paper presents a detailed study of OneShield Privacy Guard, a framework designed to mitigate privacy risks in user inputs and LLM outputs across enterprise and open-source settings. We analyze two real-world deployments: (1) a multilingual privacy-preserving system integrated with Data and Model Factory , focusing on enterprise-scale data governance; and (2) PR Insights , an open-source repository emphasizing automated triaging and community-driven refinements. In Deployment 1, OneShield achieved a 95% F1 score in detecting sensitive entities like dates, names, and phone numbers across 26 languages, outperforming state-of-the-art tools such as StarPII and Presidio by up to 12%. Deployment 2, with an average F1 score of 0.86, reduced manual effort by over 300 hours in three months, accurately flagging 8.25% of 1,256 pull requests for privacy risks with enhanced context sensitivity. These results demonstrate OneShield’s adaptability and efficacy in diverse environments, offering actionable insights for context-aware entity recognition, automated compliance, and ethical AI adoption. This work advances privacy-preserving frameworks, supporting user trust and compliance across operational contexts. Introduction As Large Language Models (LLMs) gain widespread adoption in consumer and enterprise applications, ensuring data privacy has become a critical challenge. These models often process vast amounts of unstructured data that may contain sensitive Personally Identifiable Information (PII). For example, the latest Common Crawl dataset ( Smith et al. 2013 ) encompasses data from over 3 billion web pages and has vast user PII data. This raises concerns about data leakage, compliance with global regulations, and ethical use ( Carlini et al. 2021 ; Carlini et al. 2022 ) . While LLMs enable innovative applications, their ability to unintentionally store and recall sensitive data during inference creates vulnerabilities that must be addressed through robust privacy-preserving mechanisms. LLMs process unstructured data, ranging from text prompts to conversational outputs, often contains sensitive details like names, phone numbers, and credit card information. Furthermore, data sourced globally may inadvertently violate region-specific privacy regulations, including the General Data Protection Regulation (GDPR) ( gdp 2024 ) , California Consumer Privacy Act (CCPA) ( ccp 2024 ) , and Personal Information Protection and Electronic Documents Act (PIPEDA) ( pip 2024 ) . The scale and complexity of LLM deployments amplify compliance challenges. Moreover, traditional privacy techniques, such as rule-based regular expression (regex) methods or standalone machine learning models, often struggle to balance precision, contextual understanding, and computational efficiency. Detecting nuanced privacy risks, such as context-dependent sensitivity, requires advanced mechanisms. Simple entity recognition is not enough. For example, consider an office phone number publicly listed on a company website for media inquiries. This is different from the personal phone number of a private individual, like John Doe. Handling these cases requires careful analysis to comply with privacy regulations and prevent unintended disclosure. Consequently, deploying privacy-preserving systems for LLMs demands innovative framework that address these multifaceted challenges while ensuring scalability and real-time responsiveness. To address these questions, we designed a framework OneShield Privacy Guard . The key contributions of the different stages in the framework are as follows: • A detailed analysis of two distinct OneShield Privacy Guard deployments addressing different application scenarios. • A comparison of their technical architectures, privacy-preservation methods, and effectiveness. • Insights into scalability, compliance, and lessons learned that inform best practices for future LLM deployments. This work advances the understanding of deploying privacy safeguards in diverse LLM environments, contributing to the field of privacy-preserving AI and supporting the broader ethical use of LLM technologies. This paper focuses on two deployments of the OneShield Privacy Guard framework, examining their distinct approaches to addressing privacy risks in real-world environments. By analyzing these cases, the paper aims to shed light on best practices and lessons learned from deploying privacy-preserving solutions in diverse operational contexts. Specifically, the objectives include: 1. Exploring Deployment Scenarios: (a) Deployment 1: Safeguarding LLM outputs in the Data and Model Factory platform, which is an internal platform designed to provide AI models and datasets with robust data governance capabilities. The safeguard focuses on enterprise-scale data governance and multilingual privacy compliance. (b) Deployment 2: Securing community-contributed datasets in the open-source repository name anonymized , ensuring adherence to project codes of conduct and automated privacy triaging. 2. Comparative Analysis of Deployments: (a) Identify the similarities and differences in their architectures, privacy-preservation methods, and performance metrics. (b) Highlight the strengths and limitations of each deployment in addressing contextual privacy concerns. 3. Scalability and Adaptability: (a) Assess the scalability of the OneShield Privacy Guard framework across languages, domains, and operational environments. (b) Discuss the adaptability of these techniques to evolving privacy standards and emerging data protection laws. In the remainder of this paper, we first review the existing state-of-the-art PII works, followed by descriptions of our two deployments. We then perform a comparison study between the two deployments and discuss our lessons learned from them. Finally, we conclude the paper and discuss future work. State of the Art Existing Techniques for PII Detection in LLMs: Detecting Personally Identifiable Information (PII) within Large Language Models (LLMs) is an area of growing interest, with research focusing on developing scalable and accurate solutions. Traditional methods rely on regular expression (regex)-based tools like ( com 2023 ) and ( neu 2023 ) , and customized machine learning models to identify PII entities like names, addresses, date, social security numbers, Vehicle Identification Number (VIN), bank account numbers etc. For instance, tools like Presidio Analyzer ( mic 2023 ) and StarPII ( sta 2023 ) extend Named Entity Recognition (NER) capabilities by incorporating predefined dictionaries and regex patterns. However, such methods are inherently limited due to their dependence on static patterns, which fail to adapt to new contexts or nuanced PII definitions. There is also added complexity when a phone number may appear which may be identified correctly, but may not be considered as PII based on the context (e.g., phone number of an organization versus a private individual). Context-aware PII detection has emerged as a significant advancement, leveraging machine learning techniques to incorporate the surrounding context of entities in text. Studies have explored BiLSTM-based models for analyzing forward and backward contexts, enabling more robust identification of PII in unstructured text ( et al. US11127403B2, 2019 ) as well as other methods ( Gupta et al. 2021 ; Chen et al. 2023 ; Yan, Yu, and Chen 2024 ) . This contextual approach is essential for resolving ambiguities, such as distinguishing between a person’s name and a location when dealing with entities like ”Paris” or ”Jordan.” Despite advancements, existing tools still face critical gaps that limit their effectiveness in diverse real-world scenarios. One major issue is limited coverage across languages and jurisdictions. For example, a tool might perform well in detecting PII in English but struggle with languages like Arabic or Mandarin, and fail to adapt to regional regulations such as GDPR in Europe or CCPA in California. Another challenge lies in resolving conflicts between overlapping PII

Read the complete text