Spurious Privacy Leakage in Neural Networks
Chenxiang Zhang [email protected], Jun Pang [email protected], Sjouke Mauw [email protected] · 2025
Read Spurious Privacy Leakage in Neural Networks by Chenxiang Zhang [email protected], Jun Pang [email protected], Sjouke Mauw [email protected] (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2505-20095.txt.
Introduction
Chenxiang Zhang, Jun Pang, and Sjouke Mauw uncover a 2025 privacy flaw in neural networks trained on data with false links. When a trait that only seems tied to a label is in the data, groups marked by that trait can leak more in attacks that probe what a model recalls. The team names this spurious privacy leakage.
The gap grows when the task is simple, they show. Methods built to cut bias did not close the gap, since models still memorize the false trait while learning. The choice of network shape also changed how much leaked. Their work warns that fairness fixes and privacy checks must be run together.
Catalog record
| CREATOR | Chenxiang Zhang [email protected], Jun Pang [email protected], Sjouke Mauw [email protected] |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 65 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2505-20095
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Spurious Privacy Leakage in Neural Networks Chenxiang Zhang [email protected], Jun Pang [email protected], Sjouke Mauw [email protected] Abstract Neural networks are vulnerable to privacy attacks aimed at stealing sensitive data. The risks can be amplified in a real-world scenario, particularly when models are trained on limited and biased data. In this work, we investigate the impact of spurious correlation bias on privacy vulnerability. We introduce spurious privacy leakage , a phenomenon where spurious groups are significantly more vulnerable to privacy attacks than non-spurious groups. We further show that group privacy disparity increases in tasks with simpler objectives (e.g. fewer classes) due to the persistence of spurious features. Surprisingly, we find that reducing spurious correlation using spurious robust methods does not mitigate spurious privacy leakage. This leads us to introduce a perspective on privacy disparity based on memorization, where mitigating spurious correlation does not mitigate the memorization of spurious data, and therefore, neither the privacy level. Lastly, we compare the privacy of different model architectures trained with spurious data, demonstrating that, contrary to prior works, architectural choice can affect privacy outcomes. 1 Introduction Neural networks are applied across diverse domains such as face recognition, medical prognosis, or personalized advertisement. All these applications are built using user-sensitive data that can be of interest to attackers. Additionally, real-world collected data in specific domains can be limited and biased towards specific groups, a subset of the dataset sharing a common characteristic (e.g. gender, ethnicity, or geographic location). On top of the privacy concerns, neural networks trained on real-world data can inherit different kind of biases, causing failures at test time ( Sagawa et al. 2019 ; Geirhos et al. 2020 ; Shah et al. 2020 ) . These observations suggest that models trained and deployed in sensitive domains should satisfy multiple constraints, such as ensuring fair performance across groups and protection of sensitive data. In this work, we examine the spurious correlation bias, a statistical relationship between two variables that appears to be causal but is either caused by a third confounding variable or randomness. Such correlations are prevalent in real-world datasets, particularly in domains with limited and imbalanced group representation. To address the problem, several mitigation methods have been proposed in the literature with the objective to improve the performance of the worst-group ( Sagawa et al. 2019 ; Nam et al. 2020 ; Liu et al. 2021a ; Izmailov et al. 2022 ; Yang et al. 2023 ) . However, their privacy implications have not been studied. On the other hand, privacy research typically focuses on unbiased datasets such as CIFAR10 or CIFAR100 ( Krizhevsky 2009 ; Hu et al. 2022 ) , overlooking the privacy risks of realistic applications that make use of limited and biased real-world datasets. We address this gap by investigating the privacy of neural networks trained on real-world spurious correlated datasets using membership inference attacks (MIA), a family of privacy attacks commonly used for their simplicity and versatility ( Murakonda & Shokri 2020 ; Carlini et al. 2021 ) . Contributions. We identify a phenomenon we term spurious privacy leakage , where groups with spurious correlation are significantly more vulnerable to MIA than other groups, creating a privacy disparity ( Section 3.1 ). This phenomenon adds a fairness requirement on top to the existing privacy challenges, modeling a realistic scenario for data-sensitive applications. Prior works demonstrated how privacy auditing may naively conclude that a model satisfies the privacy requirements by evaluating on an aggregated average metric over the entire dataset ( Carlini et al. 2022 ; Ye et al. 2022 ; Sablayrolles et al. 2019 ) . We demonstrate that this also applies for realistic spurious data, where an average metric evaluation is misleading for evaluating spurious and non-spurious groups. Addressing this privacy disparity is important to bridge and advance the privacy and spurious correlation research, to characterize the risks of the model, and to improve the auditing process. We continue with further analysis on spurious privacy leakage . Prior works suggested that improving fairness can, in limited scenarios, bound privacy disparity ( Kulynych et al. 2022 ) . Given the range of methods proposed to mitigate spurious correlations (equivalent as improving fairness between different data groups) ( Sagawa et al. 2019 ; Kirichenko et al. 2022 ; Liu et al. 2021a ; Zhang et al. 2022 ) , we investigate whether these mitigations can also reduce privacy leakage. To explore this, we apply robust training methods to mitigate spurious correlations and re-evaluate the privacy vulnerability. Surprisingly, we find no consistent and significant privacy improvement ( Section 4.1 ). This observation leads us to introduce a new perspective on group privacy disparity based on memorization ( Zhang et al. 2021 ) : spurious robust training improves worst-class performance but does not reduce the memorization level of data compared to standard training ( Figure 7 ), and thus neither can mitigate the privacy risks. Additional results demonstrate that robust training essentially learns the same features as standard training. We also assess the effectiveness of differential privacy to mitigate the spurious privacy leakage. Given a pretrained model, a stricter privacy guarantee can offer worst group privacy protection at the expense of reduced utility ( Section 4.2 ). Lastly, while prior works mostly focus on ResNet-like architecture for privacy analysis ( Carlini et al. 2022 ; Liu et al. 2022a ) , we fairly compare the performance and privacy of different model families. Our evaluation include state-of-the-art convolutional and transformer-based architectures, pretrained using both supervised and self-supervised training. Contrary to prior works showing that the best shadow architecture always matches the target one ( Carlini et al. 2022 ; Liu et al. 2022a ) , our results provide counterexamples when models are trained on spurious correlated data. This highlights a key difference in privacy behavior when models are trained on biased real-world data. We release the code at https://anonymous.4open.science/r/spurious-mia-4FB5 . 2 Background & Related work We provide a concise introduction to key topics necessary to follow the rest of this work, including neural networks, membership inference attacks, and spurious correlation. Neural networks represent functions f_{\bm{\theta}}\colon{\mathcal{X}}\rightarrow{\mathcal{Y}} that map the input data \bm{x}\in{\mathcal{X}} to a label \bm{y}\in{\mathcal{Y}} . The dataset \mathcal{D}=\{(\bm{x}_{i},\bm{y}_{i})\} is a set of labeled pairs used for estimating the model parameters. The neural network is parametrized by \bm{\theta}\in\mathbb{R}^{n} and it is updated using a first-order optimizer to minimize a loss function \ell\colon{\mathcal{Y}}\times{\mathcal{Y}}\rightarrow\mathbb{R} . We focus on the classification setting where the cross-entropy loss is commonly used. Formally, the objective is the empirical risk minimization (ERM) ( Vapnik 1991 ) : \hat{\bm{\theta}}_{\text{ERM}}=arg\min_{\bm{\theta}}\mathbb{E}_{(\bm{x},\bm{y})\in\mathcal{D}}(\ell(\bm{y},f_{\bm{\theta}}(\bm{x})) Spurious correlation is a statistical relationship between two variables X and Y that first appears to be causal but in reality is either caused by a third confounding (e.g. spurious) variable Z or due to random chance. This relationship is in contrast with causality, where the change of the variable X leads to a direct and predictable outcome of Y while ruling out the presence of any confounding factors Z . For a given dataset with spurious correlation, a feature \bm{z} is called spurious if it is correlated with the target label \bm{y} in the training data but not in the test data. For example, in a binary bird classification dataset