Private Model Personalization Revisited
Conor Snedeker Xinyu Zhou Raef Bassily · 2025
Read Private Model Personalization Revisited by Conor Snedeker Xinyu Zhou Raef Bassily (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2506-19220.txt.
Introduction
Conor Snedeker, Xinyu Zhou, and Raef Bassily studied private model personalization in 2025. In their setting, many users each hold varied data, yet their ideal model settings share a hidden low-dimensional form. The goal is to learn that shared form and each user's small personal part, across devices, while guaranteeing user-level differential privacy for every participant.
The authors give a lean federated method built on an earlier shared-version method, adapted so it satisfies differential privacy and still works when labels are noisy. Their guarantees cover a broader group of user data spreads than earlier work needed. And in natural settings they boost the privacy error term of the previous best result by a large factor. A second setting, including binary data, is also treated in the work.
Catalog record
| CREATOR | Conor Snedeker Xinyu Zhou Raef Bassily |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 178 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2506-19220
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Private Model Personalization Revisited Conor Snedeker Xinyu Zhou Raef Bassily Abstract We study model personalization under user-level differential privacy (DP) in the shared representation framework. In this problem, there are n n users whose data is statistically heterogeneous, and their optimal parameters share an unknown embedding U ∗ ∈ ℝ d × k U^{*}\in\mathbb{R}^{d\times k} that maps the user parameters in ℝ d \mathbb{R}^{d} to low-dimensional representations in ℝ k \mathbb{R}^{k} , where k ≪ d k\ll d . Our goal is to privately recover the shared embedding and the local low-dimensional representations with small excess risk in the federated setting. We propose a private, efficient federated learning algorithm to learn the shared embedding based on the FedRep algorithm in [ CHM+21 ] . Unlike [ CHM+21 ] , our algorithm satisfies differential privacy, and our results hold for the case of noisy labels. In contrast to prior work on private model personalization [ JRS+21 ] , our utility guarantees hold under a larger class of users’ distributions (sub-Gaussian instead of Gaussian distributions). Additionally, in natural parameter regimes, we improve the privacy error term in [ JRS+21 ] by a factor of O ~ ( d k ) \widetilde{O}(dk) . Next, we consider the binary classification setting. We present an information-theoretic construction to privately learn the shared embedding and derive a margin-based accuracy guarantee that is independent of d d . Our method utilizes the Johnson-Lindenstrauss transform to reduce the effective dimensions of the shared embedding and the users’ data. This result shows that dimension-independent risk bounds are possible in this setting under a margin loss. 1 Introduction The rapid advances in machine learning have revolutionized domains such as healthcare, finance, and personalized services. However, this progress relies heavily on the availability of user data, which presents two critical challenges. First, user data is often statistically heterogeneous, that is, individual users have distinct data distributions. This heterogeneity hinders the training of a single global model that performs well for all users. Second, the reliance on sensitive user data raises significant privacy concerns, necessitating robust mechanisms that offer strong privacy protections. Model personalization has emerged as a key strategy to address the challenge of statistical heterogeneity by adapting models to individual users, rather than relying on a single global model that may perform suboptimally across diverse data distributions. A widely used framework for personalization is shared representation learning, where users collaborate to learn a low-dimensional embedding that captures commonalities in their tasks. This shared embedding allows users to train their local models more efficiently, leveraging the embedding to complement their unique data. However, despite its effectiveness in handling heterogeneity, model personalization introduces additional privacy concerns, as learning shared representations involves user collaboration, which can inadvertently expose sensitive information. Ensuring rigorous privacy protections in this setting requires formal guarantees, such as user-level differential privacy (DP) [ DMN+06 ] , which prevents adversaries from inferring an individual’s presence in the training dataset based on the trained model. Existing approaches to private model personalization suffer from limitations such as restrictive assumptions about data distributions, centralized processing requirements, and suboptimal privacy-utility trade-offs. In this work, we study model personalization under user-level differential privacy via shared representation learning. In this problem, there are n users, where user i\in[n] is associated with a data distribution \mathcal{D}_{i} and a dataset S_{i}=(z_{1},\dots z_{m})\sim\mathcal{D}_{i}^{m} . The optimal parameter of user i is denoted as w_{i}^{*}\in\mathbb{R}^{d} . We aim at learning a shared low-dimensional embedding U^{*}\in\mathbb{R}^{d\times k} , where k\ll d , and user-specific parameters v^{*}_{1},\ldots,v^{*}_{n}\in\mathbb{R}^{k} such that w^{*}_{i}=U^{*}v^{*}_{i} for each user i\in[n] , under user-level DP. We propose a novel private federated algorithm for model personalization via shared representation that achieves superior privacy-utility trade-off under relaxed assumptions. Our contributions are summarized as follows: 1.1 Contributions Efficient private federated algorithm for regression: We extend the FedRep algorithm of [ CHM+21 ] to ensure user-level DP while addressing the case of noisy labels in statistically heterogeneous user data. The algorithm is iterative, where in each iteration t , it alternates between local updates by the users to their local vectors v_{1,t},\dots,v_{n,t}\in\mathbb{R}^{k} and private aggregated gradient update to the shared embedding U_{t}\in\mathbb{R}^{d\times k} . Previous work [ JRS+21 ] required performing exact minimization with respect to both the local vectors and the shared embedding in a centralized manner. Since the embedding update in [ JRS+21 ] involves a minimization over data from all the users, transforming their algorithm to a federated algorithm is not directly feasible. Meanwhile, ours is naturally a federated algorithm since the gradient is computed by each user locally, which the server privately aggregates. In natural parameter regimes, we show that our algorithm attains excess population risk of \widetilde{O}\left(\frac{d^{2}k}{n^{2}\epsilon^{2}\sigma_{\min,*}^{4}}+\frac{d}{nm\sigma_{\min,*}^{4}}\right)+\frac{k}{m} where \sigma_{\min,*} is k -th largest singular value of \frac{1}{\sqrt{n}}V^{*} . Compared to [ JRS+21 ] , under the same parameter regimes, this reduces the privacy error term by a factor of \widetilde{O}(dk) . Robustness to broader data distributions: Unlike prior work [ JRS+21 ] that assumes Gaussian data, the utility guarantees of our private federated algorithm apply to a broader class of sub-Gaussian distributions. Moreover, our results apply to statistically heterogeneous users in the sense that individual users may have distinct sub-Gaussian distributions, while the work of [ JRS+21 ] assumes that the features of all the users are standard Gaussian. Moreover, we extend the results of [ CHM+21 ] not only to the private case but also to the non-realizable case of noisy labels. Improved initialization under privacy constraints: We introduce a private initialization algorithm for our federated algorithm. Our initialization algorithm is based on the subspace recovery approach of [ DFH+22 ] . Our algorithm leverages a top-k singular value decomposition (SVD) with added Gaussian noise. This ensures a feasible starting point for the shared embedding, even under relaxed data assumptions. Dimension-independent risk guarantees for classification: We provide an information-theoretic construction for the binary classification setting under the margin loss and derive an risk bound that benefits from margin guarantees. We incorporate the Johnson-Lindenstrauss transform to reduce the effective dimensionality of the embedding space. This leads to a margin-based risk bound independent of the input dimension d . This result holds for arbitrary distributions where the feature vectors are bounded in the Euclidean norm. 1.2 Related Work Our work builds on and extends several lines of research in federated learning, model personalization, and differential privacy: Federated personalization via shared representations: Shared representation learning has proven effective in addressing user heterogeneity [ CHM+21 , JRS+21 ] . Additionally, centralized subspace recovery methods [ TJJ21 , DFH+22 ] may be used to provide a good initialization for the shared representation for better federated personalization guarantees. While prior work focuses on non-private or centralized settings, our approach incorporates user-level DP in a federated framework and achieves better utility guarantees. Differentially private model personalization: [ JRS+21 ] introduced a private algorithm for model personalization with centralized processing and restrictive Gaussian assumptions. Our method improves excess risk guarantees and extends applicability to sub-Gaussian data distributions. There are other prior works studying private model personalization like [ BWD+22 , HWS21 ] , but these works are not under the representation learning framework and their results are not comparable to ours. Optimization methods