Token-Level Privacy in Large Language Models
Re’em Harel, Niv Gilboa, Yuval Pinter · 2025
Read Token-Level Privacy in Large Language Models by Re’em Harel, Niv Gilboa, Yuval Pinter (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2503-03652.txt.
Introduction
Re'em Harel, Niv Gilboa, and Yuval Pinter published a computing paper in 2025 on keeping private text safe when using remote language models. Sending text to an outside provider risks leaks to the provider or to eavesdroppers. Older protection methods lean on word meaning alone and ignore context. The team built a token-level method that uses both.
Their method, named dchi-stencil, works under a formal privacy frame and carries a stated privacy guarantee. Tests with current models on varied datasets show a balance of privacy and usefulness as good as or better than older methods. The authors present it as a candidate standard for privacy-safe language work in high-risk settings.
Catalog record
| CREATOR | Re’em Harel, Niv Gilboa, Yuval Pinter |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 42 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2503-03652
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Token-Level Privacy in Large Language Models Re’em Harel, Niv Gilboa, Yuval Pinter Abstract The use of language models as remote services requires transmitting private information to external providers, raising significant privacy concerns. This process not only risks exposing sensitive data to untrusted service providers but also leaves it vulnerable to interception by eavesdroppers. Existing privacy-preserving methods for natural language processing (NLP) interactions primarily rely on semantic similarity, overlooking the role of contextual information. In this work, we introduce d χ d_{\chi} -Stencil , a novel token-level privacy-preserving mechanism that integrates contextual and semantic information while ensuring strong privacy guarantees under the d χ d_{\chi} differential privacy framework, achieving 2 ϵ 2\epsilon - d χ d_{\chi} -privacy. By incorporating both semantic and contextual nuances, d χ d_{\chi} -Stencil achieves a robust balance between privacy and utility. We evaluate d χ d_{\chi} -Stencil using state-of-the-art language models and diverse datasets, achieving comparable and even better trade-off between utility and privacy compared to existing methods. This work highlights the potential of d χ d_{\chi} -Stencil to set a new standard for privacy-preserving NLP in modern, high-risk applications. 1 Introduction Natural language processing (NLP) models as a service, such as ChatGPT OpenAI 2021 ; Ouyang et al. 2022 , present notable privacy challenges. These models often require users to send their data to external servers, which can result in potential exposure, misuse, or insufficient protection of personal and sensitive information Sousa and Kern 2023 . For example, information can be leaked by model inversion Li et al. 2017 ; Devlin et al. 2019 , exploitation of feature memorization within the large language model (LLM) Carlini et al. 2021 , and more. This issue affects not only users but also service providers, who are required to adhere to regulations such as the General Data Protection Regulation (GDPR) Voigt and Von dem Bussche 2017 and the California Consumer Privacy Act (CCPA) Barrett 2019 . Moreover, the dependence on third-party services further complicates compliance, as users must rely on these providers to ensure that they follow these privacy standards and manage data responsibly. One common approach to addressing privacy concerns in NLP is the application of differential privacy mechanisms Dwork 2006 , which offer formal guarantees by limiting the amount of information that can be inferred about any individual user. These mechanisms can be employed during either the training or inference phases of an LLM, typically by perturbing the input data with carefully calibrated noise. This process effectively conceals the original input, ensuring that private details cannot be reconstructed from the model’s outputs. Differential privacy can operate in both remote and local settings. However, given the inherent trust issues with remote servers, a more practical and secure alternative is to apply differential privacy locally ( Arachchige et al. 2019 , LDP;) . In this local setting, users preprocess their data using differential privacy techniques before sharing sanitized results with a server. This approach ensures that raw data remains protected from potential breaches, significantly reducing privacy risks while still enabling useful contributions to the model’s functionality. Input tokens t_{i-2} t_{i-1} t_{i} t_{i+1} t_{i+2} Embedding table Embedding vectors + w Context vector and noise vector + Un-Embedding \tilde{t}_{i} Perturbed token Figure 1: Schematic overview of our proposed method, d_{\chi} -Stencil , which integrates contextual and semantic information to enhance privacy preservation while maintaining the utility of the NLP models. In this work, we introduce d_{\chi} -Stencil , a novel differential privacy-based technique for token-level privacy preservation. The core idea of d_{\chi} -Stencil is to integrate contextual and semantic information while ensuring strong privacy guarantees that comply with LDP, as seen in Figure 1 . Specifically, for each token in a sentence, we encode the embedding vectors of its nearest neighbors to capture contextual relationships. To further enhance privacy, we introduce a noise vector sampled from a random distribution, which perturbs the encoded vector and makes it harder to reconstruct. We show that for a noise vector sampled from a Laplacian distribution, and some privacy parameter \epsilon , the mechanism is 2\epsilon - d_{\chi} -private. Finally, we locate the closest token (the process of un-embedding an embedding vector) to the perturbed vector, thereby preserving semantic integrity. By embedding information from neighboring tokens directly into each token, d_{\chi} -Stencil effectively maintains contextual information, an aspect critical to the success of LLMs but often overlooked by existing privacy-preserving techniques that primarily focus on semantic similarity. This approach offers the potential for improved utility compared to such techniques, striking a more effective balance between privacy and utility. To evaluate the efficacy of the proposed privacy-preserving mechanism, we conduct a comprehensive set of experiments on novel benchmarks designed to assess the full capabilities of LLMs, for instance, the SWAG Zellers et al. 2018a benchmark. We compare the d_{\chi} -Stencil mechanism against existing privacy-preserving techniques, including the CusText + Chen et al. 2023 text sanitization method and the d_{\chi} -differential privacy approach proposed by Feyisetan et al. 2020 . The experimental results show that the d_{\chi} -Stencil mechanism achieves comparable and even better utility-privacy tradeoffs compared to these alternative methods, demonstrating the potential of incorporating contextual information in addition to semantic information in privacy preserving techniques. 2 Preserving Privacy in Models There have been many attempts to preserve privacy which primarily focus on anonymization techniques Liu et al. 2017 ; Friedrich et al. 2019 and methods that introduce noise into embedding vectors Zhou et al. 2023 ; Zhou et al. 2022 . Although these approaches provide some degree of privacy protection, they often lack rigorous mechanisms to quantify the guarantees they offer, which may make them insufficient for compliance with strict privacy regulations. By contrast, the LDP mechanism offers a robust framework with formal guarantees, making it a reliable and regulation-compliant approach for safeguarding privacy in NLP applications Cummings and Desai 2018 . Since the primary goal is to maintain the usefulness of the NLP model, the privacy-preserving technique must achieve a good balance between privacy and utility. In this regard, the LDP mechanism imposes a high privacy standard, requiring that any two samples produce similar and indistinguishable output distributions Feyisetan et al. 2020 . This often results in outputs that lack sufficient information, thereby harming downstream tasks. To address this issue, a common practice is to implement a relaxed version of LDP known as d_{\chi} -privacy Feyisetan et al. 2020 ; Qu et al. 2021 ; Yiwen et al. 2018 . d_{\chi} -privacy requires the output distribution to be proportionate with respect to the distance between any two samples. Consequently, the outputs are likely to preserve more semantic information as related by the distance function. Qu et al. 2021 proposed a d_{\chi} -LDP mechanism that incorporates noise from a random distribution into the user’s input. The noise introduced complies with formal privacy guarantees under the privacy parameter \epsilon , which determines the amount of noise introduced to the embedding vector. They outline three distinct methods for noise application across LLM components: the input text, token embeddings, and sequence embeddings. Although applying noise at the sequence embedding level has the least impact on performance, it requires access to the remote LLM, which may be restricted due to proprietary limitations. While this approach is straightforward