Differentially Private In-context Learning via Sampling Few-shot Mixed with Zero-shot Outputs
James Flemings, Haosheng Gan, Hongyi Li, Meisam Razaviyayn, Murali Annavaram · 2025
Read Differentially Private In-context Learning via Sampling Few-shot Mixed with Zero-shot Outputs by James Flemings, Haosheng Gan, Hongyi Li, Meisam Razaviyayn, Murali Annavaram (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2501-19287.txt.
Introduction
James Flemings and co-authors build a private way to teach language models from examples without leaking them, in a 2025 paper. Example-fed prompts can carry secrets that the model may spit back. Privacy math can guard them, but open-ended writing makes the guard hard to hold.
Their sampler blends outputs from many one-example prompts with a no-example output and draws from the mixed spread. The blend caps what any one example can leak, and the draw's own chance supplies the privacy cover with no added noise. On a summary data set, the method held a firm privacy level while scores dipped by a hair against the unguarded form.
Catalog record
| CREATOR | James Flemings, Haosheng Gan, Hongyi Li, Meisam Razaviyayn, Murali Annavaram |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 60 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2501-19287
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Differentially Private In-context Learning via Sampling Few-shot Mixed with Zero-shot Outputs James Flemings, Haosheng Gan, Hongyi Li, Meisam Razaviyayn, Murali Annavaram Abstract In-context learning (ICL) has shown promising improvement in downstream task adaptation of LLMs by augmenting prompts with relevant input-output examples (demonstrations). However, the ICL demonstrations can contain privacy-sensitive information, which can be leaked and/or regurgitated by the LLM output. Differential Privacy (DP), a widely adopted privacy safeguard, has emerged to mitigate this privacy leakage, with recent work demonstrating strong privacy-utility tradeoffs in classification tasks for ICL. However, generation tasks for ICL are challenging due to the high-dimensional output space of open-ended generation. To this end, we propose DPS-MOZO , Differentially Private Sampling by Mixing One-shot with Zero-shot Outputs, a decoding framework that generates DP text by sampling from the product of multiple one-shot outputs mixed with a zero-shot output. This mixing effectively reduces the amount of information that can be leaked by each demonstration. By utilizing the inherent randomness in sampling from the mixed distributions, we can achieve DP without adding noise, thereby improving the privacy-utility tradeoff. Our experimental evaluations show DPS-MOZO can achieve a strong privacy guarantee, ϵ = 2 \epsilon=2 , with minimal utility degradation compared to non-private few-shot learning, 0.3% ROUGE-L F1 score decrease on the SAMSum dataset with Gemma 2 2B. 1 Introduction Large language models (LLMs) exhibit an In-Context Learning (ICL) ability, enabling them to adapt to downstream tasks without modifying model parameters by conditioning the model on relevant input-output examples (demonstrations) ( Brown et al., 2020 ; Min et al., 2022 ) . This impressive ICL ability is largely attributed to 1) the semantic prior knowledge obtained through pre-training of vast public data ( Devlin et al., 2018 ; Radford et al., 2019 ) and 2) the input-output mapping from the demonstrations learned by the model during inference ( Wei et al., 2023 ) . As a result, ICL is gaining popularity as an alternative to fine-tuning, thereby avoiding the complexities involved in LLM training. Figure 1 : Problem setup for privacy-preserving in-context learning. A growing concern raised is that the downstream dataset could contain sensitive information, such as Personal Identifiable Information. Although ICL does not modify an LLM’s parameters to incorporate this private information, the LLM can still leak privacy by regurgitating the ICL demonstrations ( Duan et al., 2023 ; Wang et al., 2023 ; Priyanshu et al., 2023 ) . Consider the setup illustrated in Figure 1 , where a provider hosts an LLM to respond to client input queries. To enhance the quality of the responses, the provider uses demonstrations from its downstream dataset for ICL by the LLM. However, the output queries now contain information about the downstream dataset through the demonstrations, which can be leaked to the client. Hence, there is a need to develop privacy safeguards for ICL. (a) (b) Figure 2 : (a) A high-level overview of DPS-MOZO broken down into four phases: (1) Subsample Demonstrations. First, n_{\text{shots}} demonstrations are randomly selected without replacement from a private dataset, then one-shot prompts containing a demonstration, the input query, and the current output are generated. Also, a zero-shot prompt containing only the input query and the current output is generated. (2) Inference. An LLM performs inference on each prompt to generate n_{\text{shots}}+1 output distributions. (3) Mixing. Each one-shot output is mixed with the zero-shot output such that the resulting mixed output has a bounded divergence from the zero-shot output. (4) Decode. Lastly, the next token is sampled from the product of the mixed one-shot output distributions. (b) Two applications of DPS-MOZO on an ICL pipeline. The first, DPS-MOZO+On , applies DPS-MOZO during the online phase to answer input queries. The second, DPS-MOZO+Off , applies DPS-MOZO during the offline phase with public inputs to generate DP outputs, both of which are to be used in the online phase. Differential Privacy (DP) ( Dwork, 2006 ; Dwork et al., 2014 ) has emerged as a practically relevant solution to ameliorate privacy leakage. For classification tasks, prior work has proposed DP ICL solutions with strong privacy-utility tradeoffs ( Wu et al., 2023 ; Tang et al., 2023 ; Duan et al., 2024 ) . Nevertheless, achieving DP for text-generation tasks is inherently more challenging due to the high-dimensional output space of each LLM response. As an example, an LLM with a vocab size of 50,000 and a maximum generation length of 100 can produce 50,000^{100} possible different generations. Wu et al. (2023) overcame this by performing DP once on the entire LLM response rather than for every token in the response. However, their solution requires upwards of 100x more inferences than non-private ICL, and the final response relies heavily on the zero-shot capabilities of LLMs. To address the privacy concerns for ICL on generation tasks with the above-mentioned limitations, we present DPS-MOZO , Differentially Private Sampling by Mixing One-shot with Zero-shot Ouputs, a simple but effective DP decoding framework shown in Figure 2(a) . The key idea of DPS-MOZO is to bound multiple one-shot output probability distributions, the next token output distribution conditioned on an input-output example, by mixing it with the zero-shot output, the next token output that is not conditioned on any input-output examples and represents an LLM’s parametric knowledge. We show that sampling from this mixed distribution can achieve DP for ICL demonstrations. Our work is broadly aligned with prior work that has shown that sampling from a bounded distribution intrinsically provides privacy ( Wang et al., 2015 ; Husain et al., 2020 ; Flemings et al., 2024 ) . We propose two DP ICL solutions by applying DPS-MOZO in two different ICL settings, offline and online, to achieve DP query outputs, as shown in Figure 2(b) . In the first setting, DPS-MOZO+On , DPS-MOZO is applied during the online phase (inference time) to generate DP query outputs for every input query received. In the second, DPS-MOZO+Off , DPS-MOZO is applied to public example inputs during the offline phase to generate DP example outputs as few-shot demonstrations, which will be used during the online phase. A summary of our contributions is the following: • We introduce DPS-MOZO , a private decoding framework that mixes multiple one-shot output distributions with zero-shot output distributions and then samples from it. This approach avoids additive noise, commonly found in DP solutions, by relying upon the inherent stochasticity of LLMs to achieve better generation quality while still achieving DP for ICL demonstrations. Moreover, DPS-MOZO does not depend on a large ensemble of demonstrations found in current DP ICL solutions. • We demonstrate that we can apply DPS-MOZO in two different settings of the ICL pipeline to achieve DP responses: 1) the online setting, termed as DPS-MOZO+On , which uses DPS-MOZO to answer input queries directly and achieve DP; 2) the offline setting, DPS-MOZO+Off , which uses DPS-MOZO to generate DP few-shots offline that will be used for ICL in the online phase to answer input queries. • Our experimental results demonstrate that we can achieve strong privacy guarantees with minimal utility degradation. In particular, we show that our framework can achieve \epsilon=2 while only losing \mathbf{0.3}\% ROGUE-L compared to non-private few-shot ICL for the SAMSum dataset with Gemma 2 2B. •