Empirical Calibration and Metric Differential Privacy in Language Models
PEDRO FAUSTINI 1, NATASHA FERNANDES 1, ANNABELLE MCIVER 1, MARK DRAS 1 · 2025
Read Empirical Calibration and Metric Differential Privacy in Language Models by PEDRO FAUSTINI 1, NATASHA FERNANDES 1, ANNABELLE MCIVER 1, MARK DRAS 1 (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2503-13872.txt.
Introduction
Pedro Faustini and co-authors measure real privacy in language models in a 2025 paper. Privacy training reports a budget number that means little on its own and cannot be compared across methods. The team tests whether attacks can calibrate that number for text models, as done for images.
Membership guessing attacks proved weak guides, while reconstruction attacks revealed more. The authors also build a directional noise method based on a sphere distribution, which scrambles angles instead of adding round noise. Each method wins in different spots of the privacy-utility trade-off. Readers get a more honest ruler for private AI.
Catalog record
| CREATOR | PEDRO FAUSTINI 1, NATASHA FERNANDES 1, ANNABELLE MCIVER 1, MARK DRAS 1 |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 60 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2503-13872
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Empirical Calibration and Metric Differential Privacy in Language Models PEDRO FAUSTINI 1, NATASHA FERNANDES 1, ANNABELLE MCIVER 1, MARK DRAS 1 Abstract NLP models trained with differential privacy (DP) usually adopt the DP-SGD framework, and privacy guarantees are often reported in terms of the privacy budget ϵ \epsilon . However, ϵ \epsilon does not have any intrinsic meaning, and it is generally not possible to compare across variants of the framework. Work in image processing has therefore explored how to empirically calibrate noise across frameworks using Membership Inference Attacks (MIAs). However, this kind of calibration has not been established for NLP. In this paper, we show that MIAs offer little help in calibrating privacy, whereas reconstruction attacks are more useful. As a use case, we define a novel kind of directional privacy based on the von Mises-Fisher (VMF) distribution, a metric DP mechanism that perturbs angular distance rather than adding (isotropic) Gaussian noise, and apply this to NLP architectures. We show that, even though formal guarantees are incomparable, empirical privacy calibration reveals that each mechanism has different areas of strength with respect to utility-privacy trade-offs. † † history: Date of publication xxxx 00, 0000, date of current version xxxx 00, 0000. † † doi: 10.1109/ACCESS.2024.0429000 † † address: Macquarie University, Sydney, NSW 2109 Australia † † titlenote: This work has been submitted to the IEEE for possible publication. Copyright may be transferred without notice, after which this version may no longer be accessible. † † corresponding: Corresponding author: Pedro Faustini (e-mail: [email protected]). I Introduction Research has shown how Differential Privacy (DP) [ 1 ] , and DP-SGD specifically as a method for training machine learners [ 2 ] , can protect sensitive information, including in NLP tasks [ 3 , 4 ] . Works usually adopt the Laplace or Gaussian mechanisms, where noise determined by the mechanism type and a privacy budget \epsilon is added to the output of the model, to its input data, or to the gradients during training. The challenge is finding an acceptable tradeoff between preserving privacy and keeping utility. Various work has observed [ 5 ] that it is difficult to know what a privacy budget \epsilon ‘means’, and how to choose one. [ 6 ] consequently proposed a calibration of DP with respect to a concrete empirical task that captures privacy leakage; this additionally allows clearer comparison and understanding of privacy-utility trade-offs where various DP frameworks (e.g., Rényi DP [ 7 ] ) may be used in privacy accounting. 1 1 1 Note that this distinct from the goal of privacy auditing: the early work of [ 8 ] characterises auditing as aiming to determine precise privacy guarantees of DP-SGD, and contrasts it with the goal of [ 6 ] of comparing privacy guarantees across frameworks. Their proposal to use Membership Inference Attacks (MIAs), where success in attacking a machine learning model indicates potential privacy leakage in the form of knowing whether an individual is in the training set or not, has been widely taken up in the image processing literature [ 9 , 10 , 11 ] . Usually, works presuppose that the attack is ‘well-behaved’ with respect to \epsilon : smaller privacy budgets, and consequent larger noise, should correspond roughly monotonically to less effective attacks, so that effects of \epsilon values can be distinguished and compared within and across DP variants. The image data of [ 6 ] and subsequent work follows this pattern (see Fig 1 ). However, a calibration in the style of [ 6 ] has not been established for language data, despite MIAs having been used as supplementary evaluations for empirically assessing privacy leakage in NLP works (for example, [ 12 , 13 ] ). Suggesting that a straightforward use as in the image domain is unlikely, some other work [ 14 ] has shown that DP may have its peculiarities when applied to language data, and further, some very recent work has suggested that MIAs in general work poorly against large language models [ 15 ] . We note that membership inference is not the only attack type suitable for calibration; another potential type of attack is the reconstruction of training data, with [ 16 ] characterising it as particularly important. In this paper, then, we investigate the behaviour of both MIAs and reconstruction attacks with respect to DP-SGD on NLP datasets, to see how useful they are for calibration in these contexts. Fig. 1: [ 6 ] ’s Fig 2(a): calibration plotting \epsilon against privacy leakage for several DP-SGD variants. As noted, an empirical calibration allows the comparison of noise mechanisms outside the standard DP-SGD framework. One example is metric DP (which generalises the distance function; standard DP is then the special case of metric DP with the Hamming distance), which has been seen in NLP works mostly dealing with local DP (for instance, [ 17 , 18 , 19 ] ). Gaussian noise is isotropic: its noise vector is equally likely to point in any direction in the high-dimensional space of the gradients. In contrast, we might expect that the utility of the model would be better served by a mechanism which is designed to preserve the direction of the gradients. Intuitively, the more their directions are preserved, the better is the gradient descent algorithm going to minimise the loss function. The idea of tuning the shape of the noise arises from metric differential privacy or d -privacy [ 20 ] , a generalisation of DP in which the notion of adjacency is generalised to a distinguishability metric d . Metric differential privacy is a unifying definition which subsumes both central and local DP, the former recoverable by choosing d to be the Hamming metric on databases, and the latter by choosing d to be the Discrete metric on individual data points. By careful choice of the metric d , d -privacy mechanisms can provide a better privacy-utility trade-off than with standard DP. We thus compare the Gaussian-noise mechanism of [ 2 ] with the (directional privacy) von Mises-Fisher mechanism of [ 21 ] , who applied it to recurrent temporal data rather than deep learning. As a use case of our calibration framework, then, we define and apply for the first time a directional privacy mechanism to fine-tuning Transformers and to NLP tasks, and compare it to the standard DP-SGD. The paper is structured as followed. In Sec II , we review related work on use of DP in ML and specifically NLP, and also different kinds of model attacks and how privacy has been calibratd. In Sec III , we assess the suitability of different attacks for privacy calibration in NLP. In Sec IV , we outline a novel application of metric DP for privately training NLP models; we then empirically compare this metric DP approach to standard baselines using the privacy calibration established in Sec III . We conclude in Sec V . In this paper, our contributions in summary are as follows. 1. We show that MIAs are not well suited for calibration of DP in language data. We also show that reconstruction attacks provide a well-behaved (albeit limited) alternative. 2. We define and apply a metric DP mechanism based on