Adversarial Sample-Based Approach for Tighter Privacy Auditing in Final Model-Only Scenarios
Sangyeon Yoon, Wonje Jeung, Albert No · 2024
Read Adversarial Sample-Based Approach for Tighter Privacy Auditing in Final Model-Only Scenarios by Sangyeon Yoon, Wonje Jeung, Albert No (2024) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2412-01756.txt.
Introduction
Sangyeon Yoon, Wonje Jeung, and Albert No sharpen how auditors test the privacy of machine learning models. Differential privacy promises a bound on what a trained model can leak about its data. Checking that promise on a finished model is hard, and standard audits often give loose lower bounds.
Their method crafts worst-case test samples by searching the input space with the model's own loss. It needs no extra assumptions and works when only the final model is available. With a stated budget of epsilon ten on the MNIST data set, their bound reached about 4.9, above the 4.4 from canary-based checks. The authors offer the method as a practical audit tool for private learning.
Catalog record
| CREATOR | Sangyeon Yoon, Wonje Jeung, Albert No |
|---|---|
| YEAR | 2024 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 29 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2412-01756
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Adversarial Sample-Based Approach for Tighter Privacy Auditing in Final Model-Only Scenarios Sangyeon Yoon, Wonje Jeung, Albert No Abstract Auditing Differentially Private Stochastic Gradient Descent (DP-SGD) in the final model setting is challenging and often results in empirical lower bounds that are significantly looser than theoretical privacy guarantees. We introduce a novel auditing method that achieves tighter empirical lower bounds without additional assumptions by crafting worst-case adversarial samples through loss-based input-space auditing. Our approach surpasses traditional canary-based heuristics and is effective in final model-only scenarios. Specifically, with a theoretical privacy budget of ε = 10.0 \varepsilon=10.0 , our method achieves empirical lower bounds of 4.914 4.914 , compared to the baseline of 4.385 4.385 for MNIST. Our work offers a practical framework for reliable and accurate privacy auditing in differentially private machine learning. 1 Introduction Differentially Private Stochastic Gradient Descent (DP-SGD) ( Abadi et al., 2016 ) was introduced to prevent sensitive information leakage from training data in models trained using Stochastic Gradient Descent (SGD) ( Shokri et al., 2017 ; Hayes et al., 2017 ; Yeom et al., 2018 ; Bichsel et al., 2021 ; Balle et al., 2022 ; Haim et al., 2022 ; Carlini et al., 2022 ) . DP-SGD mitigates privacy leakage by clipping individual gradients and adding Gaussian noise to the aggregated gradient updates. However, correctly implementing DP-SGD is challenging, and several implementations have revealed bugs that compromise its privacy guarantees ( Ding et al., 2018 ; Bichsel et al., 2021 ; Stadler et al., 2022 ; Tramer et al., 2022 ) . These issues introduce potential privacy leakage, making thorough audits essential to ensure that privacy guarantees hold in practice. Audits commonly employ membership inference attacks (MIAs) ( Shokri et al., 2017 ) , using success rates to empirically estimate privacy leakage and compare it against theoretical upper bounds ( Jagielski et al., 2020 ; Nasr et al., 2021 ; Nasr et al., 2023 ) . If the empirical privacy leakage, represented by the lower bound on \varepsilon , is significantly lower than the theoretical upper bound, the audit results are considered loose ; conversely, when the lower bound closely approaches the upper bound, the results are regarded as tight . Figure 1 : Overview of privacy auditing process. The Crafter crafts a canary and constructs two neighboring datasets, D and D^{\prime} , where D^{\prime} contains the canary. The Trainer trains a model on either D or D^{\prime} using the DP-SGD. The Distinguisher observes the loss values for specific input to infer whether the model was trained on D or D^{\prime} . We suggests an adversarial sample for tighter auditing. In intermediate model settings, where adversaries can observe gradients throughout the entire training process, previous works ( Nasr et al., 2021 ; Nasr et al., 2023 ) have shown that achieving tight privacy auditing is feasible. However, this approach is often impractical, as real-world scenarios usually restrict adversaries’ access to only the final model. Achieving rigorous audit results in the final model settings remains challenging, leaving the feasibility of final model auditing for DP-SGD an unresolved issue. To address these challenges, the final model setting has gained attention in practical applications as a promising method to improve the privacy-utility trade-off. By concealing intermediate steps and exposing only the final model, this approach aligns more closely with real-world constraints. Theoretically, in the final model setting, privacy amplification by iteration can provide additional privacy protection for data points involved in earlier stages of training ( Feldman et al., 2018 ; Balle et al., 2019 ; Chourasia et al., 2021 ; Ye and Shokri, 2022 ; Altschuler and Talwar, 2022 ; Bok et al., 2024 ) . However, this effect has primarily been observed in convex or smooth loss functions, leaving its applicability to non-convex problems unresolved. Building on this theoretical foundation, several works ( Nasr et al., 2021 ; Nasr et al., 2023 ; Andrew et al., 2023 ; Steinke et al., 2024b ; Steinke et al., 2024a ; Cebere et al., 2024 ) have explored privacy amplification in final model settings with non-convex loss functions. They observed a significant gap between empirical lower bounds and theoretical upper bounds, suggesting that privacy amplification by iteration may extend to general non-convex loss functions. However, there is no formal proof to support this, and the observed gap may instead reflect that adversaries have not fully exploited their potential capabilities. Conversely, recent works ( Annamalai, 2024 ; Cebere et al., 2024 ) have observed that privacy amplification does not occur in non-convex settings. These findings, however, apply only to specific cases, such as constructing a worst-case non-convex loss function for DP-SGD where information from all previous iterations is encoded in the final iteration, or manually designing a gradient sequence. These efforts remain constrained to additional assumptions or highly specific scenarios, leaving the broader question of whether privacy amplification can occur in general non-convex settings unresolved. In this work, we introduce a novel auditing approach designed to establish tighter lower bounds in general non-convex settings for final model scenarios, without requiring any additional assumptions. Our method leverages loss-based input-space auditing to craft worst-case adversarial samples, enhancing the precision of privacy auditing results. Previous works ( Jagielski et al., 2020 ; Nasr et al., 2021 ; Tramer et al., 2022 ; Aerni et al., 2024 ) on input-space auditing have implicitly assumed that the canary sample represents the worst-case privacy leakage, using it as an adversarial sample for MIAs based on its loss outputs. However, relying solely on canary-based approaches may not yield tighter empirical lower bounds. By leveraging loss outputs from an alternative adversarial sample, our approach identifies increased privacy leakage. Consequently, we craft worst-case adversarial samples without relying on canaries, providing tighter empirical lower bounds for auditing. Our auditing method addresses practical challenges in the final model setting. Adversaries have access to the final model’s weights, as in open-source cases. We craft the worst-case adversarial sample directly using these weights. For example, with a theoretical privacy budget of 10.0 , our approach achieves an empirical lower bound of 4.914 , significantly surpassing the 4.385 obtained with the canary approach. 2 Preliminary Differentially Private Training. Differential Privacy (DP) ( Dwork et al., 2006 ) is a widely used standard for preserving individual privacy in data analysis. A randomized mechanism \mathcal{M} is considered (\varepsilon,\delta) -differentially private if, for any two neighboring datasets D and D^{\prime} that differ by only one data point known as the canary ( c ), and for any possible output subset O\subseteq\textit{Range}(\mathcal{M}) , the following inequality holds: \mathrm{Pr}[\mathcal{M}(D)\in O]\leq e^{\varepsilon}\,\mathrm{Pr}[\mathcal{M}(D^{\prime})\in O]+\delta In this formulation, \varepsilon represents the loss of privacy, with smaller \varepsilon indicating stronger privacy guarantees. The parameter \delta\in(0,1) represents a small probability that the privacy guarantee may not hold. Numerous mechanisms have been developed to ensure differential privacy during the training of machine learning models. ( Abadi et al., 2016 ; Papernot et al., 2016 ; Zhu et al., 2020 ) . Differentially Private Stochastic Gradient Descent (DP-SGD) ( Abadi et al., 2016 ) is widely used to achieve differential privacy. DP-SGD clips gradients to a maximum norm C to limit the impact of individual data points on model parameters, then adds Gaussian noise scaled by