GeoClip: Geometry-Aware Clipping for Differentially Private SGD
Atefeh Gilani, Naima Tasnim, Lalitha Sankar, Oliver Kosut · 2025
Read GeoClip: Geometry-Aware Clipping for Differentially Private SGD by Atefeh Gilani, Naima Tasnim, Lalitha Sankar, Oliver Kosut (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2506-06549.txt.
Introduction
Atefeh Gilani, Naima Tasnim, Lalitha Sankar, and Oliver Kosut present GeoClip, a smarter way to clip gradients when training models with formal privacy guarantees, in a 2025 paper. Private training adds noise after clipping each example's gradient, and the clipping threshold strongly affects the balance between privacy and accuracy. Recent adaptive methods adjust that threshold but treat each coordinate separately.
GeoClip instead rotates the gradients into a basis matched to their actual shape, so correlations between coordinates are respected, then clips and adds noise there. The rotation is estimated from already-released noisy gradients, so it costs no extra privacy. The authors prove convergence and derive the best rotation in closed form. Experiments beat existing adaptive clippers at the same budget.
Catalog record
| CREATOR | Atefeh Gilani, Naima Tasnim, Lalitha Sankar, Oliver Kosut |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 55 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2506-06549
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
GeoClip: Geometry-Aware Clipping for Differentially Private SGD Atefeh Gilani, Naima Tasnim, Lalitha Sankar, Oliver Kosut Abstract Differentially private stochastic gradient descent (DP-SGD) is the most widely used method for training machine learning models with provable privacy guarantees. A key challenge in DP-SGD is setting the per-sample gradient clipping threshold, which significantly affects the trade-off between privacy and utility. While recent adaptive methods improve performance by adjusting this threshold during training, they operate in the standard coordinate system and fail to account for correlations across the coordinates of the gradient. We propose GeoClip, a geometry-aware framework that clips and perturbs gradients in a transformed basis aligned with the geometry of the gradient distribution. GeoClip adaptively estimates this transformation using only previously released noisy gradients, incurring no additional privacy cost. We provide convergence guarantees for GeoClip and derive a closed-form solution for the optimal transformation that minimizes the amount of noise added while keeping the probability of gradient clipping under control. Experiments on both tabular and image datasets demonstrate that GeoClip consistently outperforms existing adaptive clipping methods under the same privacy budget. 1 Introduction As machine learning models are increasingly trained on sensitive user data, ensuring strong privacy guarantees during training is essential to reduce the risk of misuse, discrimination, or unintended data exposure. Differential privacy (DP) ( Dwork et al. 2006a ; Dwork et al. 2006b ) offers a principled framework for protecting individual data, and has become a cornerstone of privacy-preserving machine learning. In deep learning, the most widely used approach for DP is differentially private stochastic gradient descent (DP-SGD) ( Abadi et al. 2016 ) , which clips per-sample gradients and adds calibrated noise to their average. Despite its widespread use, standard DP-SGD has a key limitation: it relies on a fixed clipping threshold to bound the sensitivity of individual gradients. Selecting this threshold poses a challenging privacy-utility tradeoff—setting it too low discards useful gradient information, while setting it too high increases sensitivity and necessitates injecting more noise, ultimately degrading model performance. This trade-off was observed empirically by McMahan et al. 2018 and later analyzed theoretically and shown to be a fundamental limitation of differentially private learning by Amin et al. 2019 . Moreover, the optimal threshold can vary over the course of training, across tasks, and between datasets, limiting the effectiveness of a fixed setting. To address this, recent work has proposed adaptive strategies that dynamically adjust the clipping threshold during training. One class of methods uses decay schedules to reduce the threshold over time. Yu et al. 2018 and Du et al. 2021 propose linear and near-linear decay rules, respectively, where the schedule is predefined and does not depend on the dataset—hence, no privacy budget is required. Lin et al. 2022 introduce a nonlinear decay schedule, along with a transfer strategy that leverages public data to guide threshold selection. Recently, methods have been introduced which set the clipping level based on the data during the training process. These include AdaClip ( Pichapati et al. 2019 ) , which applies coordinate-wise clipping based on estimated gradient variances. Yet another is quantile-based clipping ( Andrew et al. 2021 ) , which sets the threshold using differentially private quantiles of per-sample gradient norms. Although ( Andrew et al. 2021 ) was designed for federated learning, it can be adapted to centralized DP-SGD. These adaptive strategies have been shown to improve model utility while preserving privacy guarantees. Despite these advances, existing adaptive clipping methods remain agnostic to the geometry of the gradient distribution. They operate in the standard basis—treating each coordinate independently. This overlooks dependencies between coordinates, especially when gradients exhibit strong correlations across dimensions. In such cases, independently clipping and perturbing each coordinate can introduce redundant noise without improving privacy, ultimately degrading model utility. To address this, we propose GeoClip , a method that transforms gradients into a decorrelated basis that better reflects their underlying geometry. By applying DP mechanisms in this transformed space, GeoClip allocates noise more effectively, achieving a better privacy-utility tradeoff. A different approach to correlations in gradients from the literature considers introducing correlations—dependencies across iterations and between entries of the noise vector—into the noise, rather than injecting i.i.d. Gaussian noise. This has been shown to improve the utility of private training ( Denisov et al. 2022 ; Kairouz et al. 2021 ) . Choquette-Choo et al. 2024 strengthen this direction by analytically characterizing near-optimal spatio-temporal correlation structures that lead to provably tighter privacy-utility tradeoffs. However, these approaches have been developed independently of adaptive clipping, and the effect of combining both methods remains unexplored. In addition, ( Choquette-Choo et al. 2024 ) uses pre-determined correlations in the noise, rather than being tailored to the data, as our approach is. GeoClip is data-driven but does not require any additional privacy budget to compute the clipping transformation. Instead, it reuses the noisy gradients already released during training to estimate the mean and the covariance of the gradient distribution. By reusing these privatized gradients, GeoClip adapts its basis over time without accessing raw data or incurring additional privacy cost. We list our main contributions below: 1. We propose GeoClip, a novel framework that applies differential privacy in a transformed basis rather than the standard coordinate system. To guide the choice of transformation, we derive a convergence theorem (Theorem 1 ) showing how the basis impacts convergence under DP-SGD, providing theoretical guidance for selecting transformations that improve utility. 2. Building on this insight, we formulate a convex optimization problem to find the transformation, and derive a closed-form solution (Theorem 2 ). 3. We introduce two algorithms to estimate the transformation using only previously released noisy gradients. The first is based on a moving average to estimate the gradient covariance matrix. 4. For large-scale models, the full gradient covariance matrix is prohibitively large to store. Thus, our second algorithm uses a streaming low-rank approximation of the covariance matrix. This second algorithm is thus suitable for deep models with large parameter counts. 5. We validate GeoClip through experiments on synthetic, tabular, and image datasets, showing that it consistently outperforms existing adaptive clipping methods under the same privacy budget. Notation. We denote the d\times d identity matrix by I_{d} . The notation N\sim\mathcal{N}(0,\sigma^{2}I_{d}) denotes a d -dimensional Gaussian with zero mean and covariance \sigma^{2}I_{d} . We use \|x\|_{2} for the Euclidean norm of a vector x , A^{-1} for the inverse of matrix A , and A^{\top} for its transpose. The trace of a matrix A is denoted by \Tr(A) , and \text{Cov}(x\mid y) refers to the conditional covariance of x given y . 2 General Framework Let \mathcal{D}=\{x_{k}\}_{k=1}^{N} be a dataset of N examples, and let f:\mathbb{R}^{d}\to\mathbb{R} denote the empirical loss function defined as the average of per-sample losses: \displaystyle f(\theta)=\frac{1}{N}\sum_{k=1}^{N}f_{k}(\theta), (1) where \theta\in\mathbb{R}^{d} is the model parameter vector and each f_{k} corresponds to the loss on the k -th data point x_{k} . In DP-SGD, the algorithm updates \theta using a noisy clipped stochastic gradient to ensure privacy. Let g_{t}\in\mathbb{R}^{d} denote the stochastic gradient at iteration t . The update rule is: \displaystyle\theta_{t+1}=\theta_{t}-\eta\tilde{g}_{t}, (2) where \tilde{g}_{t} is the privatized version of g_{t} , obtained by clipping and adding noise. GeoClip builds on the DP-SGD framework but changes how gradients are processed before