Stabilization of Perturbed Loss Function: Differential Privacy without Gradient Noise
Salman Habib † , Rémi A. Chou ∗ , and Taejoon Kim † · 2025
Read Stabilization of Perturbed Loss Function: Differential Privacy without Gradient Noise by Salman Habib † , Rémi A. Chou ∗ , and Taejoon Kim † (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2508-15523.txt.
Introduction
Salman Habib and co-authors privatize model training by noising a smoothed loss instead of its gradients in a 2025 paper. Standard private training injects noise step by step into gradients, which costs time and steadiness. Their SPOF method first flattens the loss into a polynomial portrait, then adds calibrated noise to its coefficients.
Every users data is shielded at once under local privacy rules. The approach also shrugs off noisy sensor channels during training. Tested on a body-sensor network against private gradient descent, it reconstructed signals more accurately and trained far faster. Privacy, here, gets cheaper and calmer.
Catalog record
| CREATOR | Salman Habib † , Rémi A. Chou ∗ , and Taejoon Kim † |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 81 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2508-15523
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Stabilization of Perturbed Loss Function: Differential Privacy without Gradient Noise Thanks: The material is based upon work supported in part by the National Science Foundation (NSF) and Office of the Under Secretary of Defense (OUSD) Research and Engineering, ITE2326898, and ITE2226447 as part of the NSF Convergence Accelerator Track G: Securely Operating Through 5G Infrastructure Program. The simulation code used in this work is available at https://github.com/theorycoder/distributedautoencoders. Salman Habib † , Rémi A. Chou ∗ , and Taejoon Kim † Abstract We propose SPOF ( S tabilization of P erturbed L o ss F unction), a differentially private training mechanism intended for multi-user local differential privacy (LDP). SPOF perturbs a stabilized Taylor expanded polynomial approximation of a model’s training loss function, where each user’s data is privatized by calibrated noise added to the coefficients of the polynomial. Unlike gradient-based mechanisms such as differentially private stochastic gradient descent (DP-SGD), SPOF does not require injecting noise into the gradients of the loss function, which improves both computational efficiency and stability. This formulation naturally supports simultaneous privacy guarantees across all users. Moreover, SPOF exhibits robustness to environmental noise during training, maintaining stable performance even when user inputs are corrupted. We compare SPOF with a multi-user extension of DP-SGD, evaluating both methods in a wireless body area network (WBAN) scenario involving heterogeneous user data and stochastic channel noise from body sensors. Our results show that SPOF achieves, on average, up to 3.5 % 3.5\% higher reconstruction accuracy and reduces mean training time by up to 57.2 % 57.2\% compared to DP-SGD, demonstrating superior privacy-utility trade-offs in multi-user environments. I Introduction To address the open challenge of achieving differential-privacy (DP) in multi-user learning systems, we introduce SPOF ( S tabilization of P erturbed L o ss F unction), a DP approach that incorporates local differential-privacy (LDP). SPOF ensures that each user’s data is privatized independently, without any aggregation, by injecting calibrated noise directly into the coefficients of a Taylor expanded polynomial approximation of a model’s loss function. The coefficients of this loss function encode each user’s contribution, which are perturbed locally using Laplace noise. This formulation avoids the need for perturbing gradients at every optimization step, as required in differentially private stochastic gradient descent (DP-SGD), leading to improved computational efficiency and training stability. Multi-user privacy is crucial in distributed systems where multiple data sources independently contribute to sensitive information. Examples include wireless body area networks (WBANs), where physiological signals from body sensors are wirelessly transmitted to a central server [ 1 , 2 , 3 ] , smart cities [ 4 ] , and collaborative filtering [ 5 ] . In such applications, protecting individual user data from inference attacks requires mechanisms that guarantee privacy jointly over all users. Several mechanisms have been proposed to enhance DP. For instance, [ 6 ] enforces privacy via inferential separation, whereas [ 7 ] derives optimal perturbation strategies that minimize utility loss under fixed privacy budgets. The authors of [ 8 ] added artificial DP noise to client-side model parameters before aggregation, whereas [ 9 ] defends against membership and inversion attacks by privatizing and normalizing model confidence scores under a tunable budget. While prior approaches like the functional mechanism (FM) [ 10 , 11 , 12 ] perturb the objective function instead of the gradients, they are inherently designed for single-user settings and lack stabilization techniques. In contrast, SPOF adds a loss stabilization constant to the model parameters that enhance the model’s prediction accuracy. Moreover, SPOF generalizes to any number of users m . Thus, FM naturally arises as a special case of SPOF in the single-user scenario (m=1) without loss stabilization. We study SPOF using autoencoders due to their ubiquity in practical systems and their analytically tractable loss function, which allows for theoretical derivations of privacy guarantees. To accommodate a realistic multi-user setting, we adopt a distributed autoencoder (DA) architecture, where m users each possess an independent encoder and share a common decoder that reconstructs the original data. DAs have gained attention for their efficiency in distributed learning systems, including schemes such as distributed image compression [ 13 ] and task-aware source coding [ 14 ] . The DA framework enables multi-user privacy, as each user’s input contributes independently to the local privacy budget, making it a natural choice for providing LDP privacy guarantees. In contrast, FM, is suitable only for standard autoencoders (SAs) that consist of a single encoder and a decoder [ 11 ] . For simulations, we apply SPOF to a DA-based wireless body area network (e.g. WBAN) in which multiple encoders are employed to compress physiological data from the Fitbit dataset [ 15 ] , where environmental noise from body sensors and channels introduces realistic training data corruption [ 16 , 17 ] . A common decoder (e.g., hospital) collects all the compressed signals for post processing. As a benchmark for SPOF, we extend DP-SGD [ 18 , 19 , 20 , 21 , 22 ] to the multi-user setting, by incorporating advanced techniques such as bookkeeping [ 23 ] and group clipping [ 24 ] . While recent methods such as DP based on zeroth order optimization (DP-ZO) [ 25 ] also apply DP at the loss level, they lack the analytical tractability of SPOF, and whether DP-ZO leverages environmental noise for privacy is unknown. In this paper, we consider privacy mechanisms for SPOF and DP-SGD, both using Laplace noise. Notably, the use of Laplace noise in the context of DP-SGD has also been explored in [ 25 ] . Our analysis shows that SPOF satisfies DP under both noiseless and noisy input conditions, where the latter is caused due to environmental noise. The proposed loss stabilization technique biases the approximated loss of a DA and improves the model’s prediction accuracy. We further derive sensitivity expressions and show that SPOF can yield lower sensitivity than DP-SGD for a certain value of the Taylor-approximated loss function parameter, enabling lower noise injection for a fixed privacy budget. Simulations using the Fitbit dataset confirm that SPOF outperforms DP-SGD in both accuracy and training time. Notably, when environmental noise standard deviation (s.d.) is large enough, SPOF requires less additional DP noise with a probability of at least 0.5 due to a multiplicative scaling factor that reduces the overall noise variance. On the other hand, our simulations indicate that DP-SGD suffers utility degradation under noisy conditions, presumably due to large fluctuations in gradient magnitudes during early training epochs. Our main contributions are as follows: • We derive a multi-user cross-entropy loss function that depends on all m+1 trainable parameters, i.e., weights of the m encoders and a decoder of the DA, and approximate it using a Taylor-expanded loss \tilde{L} . SPOF perturbs the coefficients of \tilde{L} , while DP-SGD perturbs its gradients. In Section IV and VII , we show that both mechanisms satisfy DP guarantees under noiseless and noisy input conditions. • In Section V , we discuss an essential component of SPOF that involves adding a loss stabilization constant into a model’s learnable parameters. This constant stabilizes the direction of gradient updates and improves a DA’s convergence and reconstruction accuracy, without compromising the privacy guarantee. • We analyze the effect of environmental noise \mathbf{n}