Logistic Regression Model for Differentially-Private Matrix Masking Data
Linh H. Nghiem, Aidong Adam Ding, Samuel S. Wu · 2024
Read Logistic Regression Model for Differentially-Private Matrix Masking Data by Linh H. Nghiem, Aidong Adam Ding, Samuel S. Wu (2024) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2412-15520.txt.
Introduction
Linh Nghiem, Aidong Ding, and Samuel Wu published this statistics paper in 2024. It concerns data collected under a privacy scheme where noise is added at the source and values are masked by matrix mixing. Under that scheme, no one, not even the data manager, sees the true records. Simple models can still be fit to such data, but logistic regression, a standard tool for yes-or-no outcomes, had no valid method.
The authors build the missing method by leaning on a known link between logistic regression and linear regression estimates. They prove the approach works when samples are large, even as the added noise grows to meet strict privacy demands. Tests on simulated and real data showed their estimates beat the plain approach of running logistic regression on the protected data as if it were clean.
Catalog record
| CREATOR | Linh H. Nghiem, Aidong Adam Ding, Samuel S. Wu |
|---|---|
| YEAR | 2024 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 80 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2412-15520
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Logistic Regression Model for Differentially-Private Matrix Masking Data Linh H. Nghiem, Aidong Adam Ding, Samuel S. Wu Abstract A recently proposed scheme utilizing local noise addition and matrix masking enables data collection while protecting individual privacy from all parties, including the central data manager. Statistical analysis of such privacy-preserved data is particularly challenging for nonlinear models like logistic regression. By leveraging a relationship between logistic regression and linear regression estimators, we propose the first valid statistical analysis method for logistic regression under this setting. Theoretical analysis of the proposed estimator confirmed its validity under an asymptotic framework with large noise variances to account for strict privacy requirements. Simulations and real data analyses demonstrate the superiority of the proposed estimators over naive logistic regression methods on privacy-preserved datasets. Keywords: privacy, measurement error, generalized linear model, mixture model 1 Introduction In the age of data-driven innovation, privacy-protected data collection methods have become essential for balancing the interests of organizations and individuals. These methods ensure that personal and sensitive information is collected, stored, and processed in ways that protect user privacy and comply with regulatory frameworks like GDPR, HIPAA, and CCPA. Protecting privacy is not only necessary for legal compliance but also critical to building and maintaining consumer trust. To ensure that data analysis yields valid statistical inferences from privacy-protected data, new statistical methods are required. Here, we propose a statistical inference procedure for logistic regression on data collected using matrix masking combined with noise addition. Over the years, many privacy-protected data collection methods have been developed. Common approaches include anonymization (removing personal identifiers) and pseudonymization (replacing sensitive information with fictitious identifiers). While widely used, these methods do not offer formal mathematical privacy guarantees. Differential privacy ( Dwork,, 2006 ) addresses this gap by adding statistical noise to datasets, providing a theoretical, quantitative guarantee of privacy. It has since become one of the most popular privacy-preserving methods, with noise addition to summary statistics enabling privacy-preserving logistic regression studies, such as that of Zhang et al., (2012) . However, noise addition at the summary level requires a trustworthy central data manager, which does not protect user privacy from data collectors themselves. To achieve user privacy at the individual level, local differential privacy adds noise to data before it is sent to the data collector. Local differential privacy schemes have been successfully implemented by companies like Apple ( Thakurta et al.,, 2017 ) and Google ( Erlingsson et al.,, 2014 ) to protect individual-level data. Another privacy protection scheme, matrix masking uses a random orthogonal matrix to scramble raw data and prevent identification of individual records ( Ting et al.,, 2008 ) . Ding et al., (2020) proved that a triple matrix-masking (TM 2 ) data collection procedure ( Wu et al.,, 2017 ) , using multi-party computing concepts, can collect a matrix masked data set while preserving individual-level privacy. This method, combined with local differential privacy’s noise addition, achieves the same quantitative privacy guarantees with less noise than local differential privacy only scheme ( Ding et al.,, 2023 ) . The combined scheme is referred to here as the TM²+Noise scheme. The TM 2 +Noise data presents unique challenges for logistic regression. Although simple noise addition to the covariates results in data that follows a measurement error model, which has been studied extensively, the existing statistical methods for logistic regression models with covariate measurement errors, such as those in ( Buonaccorsi,, 2010 , Chapter 6) , are not directly applicable to this setting for the following reason. First, under this TM 2 +Noise scheme, the observed response variable is no longer binary, but a real-valued continuous variable. Secondly, the matrix masking only preserves the first and second sample moments of the data, the presence of a non-linear link function in the logistic regression model (and a non-linear model in general) typically requires other information to conduct statistical inference for the model parameters. This other information, however, is not preserved under matrix masking. Hence, incorporating matrix masking with noise addition requires new statistical methods for non-linear models like logistic regression. In this paper, we propose the first valid statistical estimator and inference for the logistic regression model under the TM 2 +Noise setting. Building upon the seminal work of Haggstrom, (1983) who relate the logistic regression to the linear regression through a Gaussian mixture model, we generalize it to a conditional Gaussian mixture model that allows confounding covariates, and obtain the corresponding form of the logistic regression model. Then, we establish both a point and an interval estimator of this model using only the first and second sample moments of the raw data, whose expectations are preserved under the TM 2 +Noise scheme. We study the properties of this estimator under an asymptotic framework that does not assume the variance of added noise to be fixed, demonstrating the tradeoff between statistical accuracy and privacy. Simulation studies and a data application on the relationship between hypertension prevalence and various covariates in a medical database, illustrate the desirable performance of the proposed statistical approach. Our paper also contributes to a growing statistics literature on making inferences from summary statistics ( Bhowmik et al.,, 2015 ; Whitaker et al.,, 2021 ) . 2 Notations and model setup of logistic regression data collected from TM 2 +Noise scheme Data is collected from n individuals, where for each individual i=1,...,n , the data includes a binary outcome variable y_{i}^{*}\in\{0,1\} and and a set of p covariates represented by a p -dimensional row vector \mathbf{X}^{*}_{i}\in\mathbb{R}^{1\times p} . The logistic regression model defines the relationship between the binary response variable and the covariates as follows: P(y^{*}_{i}=1\mid\mathbf{X}_{i}^{*})=\dfrac{\exp(\beta_{0}+\mathbf{X}_{i}^{*}\bm{\beta}_{1})}{1+\exp(\beta_{0}+\mathbf{X}_{i}^{*}\bm{\beta}_{1})}=\dfrac{1}{1+\exp\left\{-\left(\bm{\beta}_{0}+\mathbf{X}_{i}^{*}\bm{\beta}_{1}\right)\right\}}, (1) where \beta_{0}\in\mathbb{R} is the intercept and \bm{\beta}_{1}\in\mathbb{R}^{p\times 1} is the coefficient vector. The primary goal is to make statistical inferences on the parameters \bm{\beta}_{1} based on observed data. If collected directly, the raw data from the n individuals can be represented as the response vector \mathbf{y}^{*}=(y_{1}^{*},\ldots,y_{n}^{*})^{\top} and the n\times p covariate matrix \mathbf{X}^{*} with \mathbf{X}^{*}_{i} as its i -th row. However, for privacy protection, the data is collected using the TM 2 +Noise scheme, so the analyst only has access to \mathbf{X}=\mathbf{M}\mathbf{X}^{*}+\mathbf{U},~\mathbf{y}=\mathbf{M}\mathbf{y}^{*}+\mathbf{v}, (2) where \mathbf{M} is an unknown n\times n orthogonal matrix that preserves row sums, i.e \mathbf{M}^{\textsc{T}}\mathbf{M}=\mathbf{I}_{n} and \mathbf{M}^{\top}\mathbf{1}_{n}=\mathbf{1}_{n} , and \mathbf{U} and \mathbf{v} are noise terms with dimensions n\times p and n\times 1 , respectively. The elements of these noise terms \mathbf{U} and \mathbf{v} are drawn from the normal distribution N(0,\sigma_{x}^{2}) and N(0,\sigma_{y}^{2}) respectively, with \sigma_{x} and \sigma_{y} known to the analyst. Due to the masking matrix \mathbf{M} and added noise, all elements in the masked data \mathbf{X} and \mathbf{y} are real-valued. The focus is on estimating and making inferences for \beta_{0} and \bm{\beta}_{1} based on the masked data \mathbf{X} and \mathbf{y} . If the raw data \mathbf{X}^{*} and \mathbf{y}^{*} were directly available, the conditional maximum likelihood estimator (cMLE) would typically be used to estimate \beta_{0} and \bm{\beta}_{1} . This involves minimizing the conditional negative log-likelihood function \ell^{*}(\beta_{0},\bm{\beta}_{1}\mid\mathbf{y}^{*},\mathbf{X}^{*})=-\sum_{i=1}^{n}\log P(y_{i}^{*}=1\mid\mathbf{X}_{i}^{*}) . Equivalently, this estimator solves the score equation \mathbf{S}(\beta_{0},\bm{\beta}_{1};\mathbf{y}^{*},\mathbf{X}^{*})=\dfrac{1}{n}\sum_{i=1}^{n}\left(\mathbf{y}^{*}_{i}\widetilde{\mathbf{X}}_{i}^{*}-\dfrac{\widetilde{\mathbf{X}}_{i}^{*}}{1+\exp\left\{-\widetilde{\mathbf{X}}_{i}^{*}\bm{\beta}\right\}}\right)=\bm{0}. (3) Here and in the following, for any matrix \mathbf{B} , we let \widetilde{\mathbf{B}}=(\mathbf{1},\mathbf{B}) denote the matrix formed by stacking a column of one to the left of \mathbf{B} , and \bm{\beta}=(\beta_{0},\bm{\beta}_{1}^{T})^{T}\in\mathbb{R}^{(p+1)\times 1} denote the