Differentially Private Hyperparameter Tuning using Local Bayesian Optimization
Getoar Sopa, Juraj Marusic *, Marco Avella Medina, John P. Cunningham · 2025
Read Differentially Private Hyperparameter Tuning using Local Bayesian Optimization by Getoar Sopa, Juraj Marusic *, Marco Avella Medina, John P. Cunningham (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2502-06044.txt.
Introduction
Private tuning of model knobs by local search is built by Getoar Sopa and co-authors in a 2025 arXiv paper. Tuning picks settings like learn rate, but checks on user data can leak that data. Old private tuners searched near at random. This team's method, DP-GIBO, guesses slopes with a smooth stand-in model and keeps each step local.
Proofs show it lands near the best local pick, off only by a privacy-set span. Its cost grows mildly with knob count, not in leaps. Tests beat private random search and global private search in mid to wide knob spaces.
Catalog record
| CREATOR | Getoar Sopa, Juraj Marusic *, Marco Avella Medina, John P. Cunningham |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 82 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2502-06044
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Differentially Private Hyperparameter Tuning using Local Bayesian Optimization Getoar Sopa, Juraj Marusic *, Marco Avella Medina, John P. Cunningham Abstract Hyperparameter tuning is a key component of machine learning procedures, but when validation data contain sensitive user information, search mechanisms can leak private information through the selected configuration. Existing differentially private hyperparameter tuning methods often rely on near-random search, while prior differentially private Bayesian optimization approaches are typically global and, therefore, scale poorly with the hyperparameter dimensionality. We study differentially private hyperparameter tuning using local Bayesian optimization, focusing on settings where the validation objective is available only through noisy black box evaluations and gradients are unavailable or impractical to compute. We introduce DP-GIBO, a differentially private local Bayesian optimization framework that privately approximates gradients using a Gaussian Process surrogate. Under suitable conditions, we prove that DP-GIBO converges to a locally optimal hyperparameter configuration up to a privacy-dependent error, with dimensional dependence that is polynomial rather than exponential. Empirically, we show that DP-GIBO provides scalable private hyperparameter tuning across multiple tasks, substantially outperforming non-private random search and global Bayesian optimization baselines in moderate-to-high-dimensional hyperparameter spaces. 1 Introduction Differentially private (DP) machine learning methods have increasingly seen use by practitioners due to the sensitive nature of the data used in some machine learning applications. Examples include healthcare data [ 1 ] , data collected at scale by companies that must comply with privacy regulations [ 2 ] , and more. Differential Privacy allows practitioners to reason about the amount of ‘data leakage’ to an individual by the inclusion of one data point to the input of an algorithm [ 3 ] . To preserve these guarantees end-to-end, hyperparameters must also be tuned under differential privacy, since the choice of hyperparameters can itself reveal information about the underlying data. Independent of this body of research, there have been many recent advances in optimizing black box functions using Bayesian optimization (BO) [ 4 ] . Often, these black box functions appear in the optimization of hyperparameters of machine learning algorithms [ 5 ] , reinforcement learning [ 6 , 7 ] , experimental design [ 8 ] , and more [ 4 ] . These methods allow for the efficient evaluation of points in the domain space, and therefore often find ‘good’ parameter configurations in fewer rounds of training than classical grid search methods. Unfortunately, classical global BO fits a single surrogate over the entire search space and selects each query to optimize a global acquisition function, an approach that breaks down in high dimensions, rendering unrestricted global optimization nearly intractable for as few as 10 dimensions [ 9 , 10 ] . Local Bayesian optimization approaches sidestep this by restricting attention to a neighborhood of the current iterate: TuRBO [ 11 ] maintains many small trust regions, while Gradient Informative Bayesian Optimization (GIBO) [ 7 , 12 ] uses BO to approximate the local gradient and descends along it. A unifying question, then, is: how can we tune hyperparameters in a differentially private way ? Recent work has mostly considered nonadaptive, random grid search methods [ 13 , 14 , 15 ] . These papers have developed powerful techniques for ensuring a constant privacy cost for training the model for given hyperparameter configurations, even as a random—potentially large—number of configurations are evaluated. Recently, this line of work was extended to adaptive hyperparameter search strategies by Wang et al. [16] . These methods have in common that in order to achieve a small fixed privacy cost in higher dimensions, the hyperparameter sampling mechanism has to be close to uniformly random, making the hyperparameter search intractable in these problems. Recent work has also laid a theoretical foundation for efficient differentially private bilevel optimization for structured smooth bilevel problems with access to first- and second-order derivative information [ 17 , 18 ] , though this information is often unavailable or infeasible to compute. Figure 1 : DP-GIBO scales to higher-dimensional private hyperparameter tuning. Final validation loss on a group LASSO tuning task with d\in\{2,10,20\} group regularization hyperparameters, where each evaluation refits the group LASSO on the training split and queries the validation loss on held-out users, yielding a black box objective with no usable gradient w.r.t. hyperparameter. Random search is competitive in low dimensions but degrades sharply as d grows, and offers no privacy protection by default, so any released hyperparameter is a function of the validation data and may reveal information about individual users. In contrast, DP-GIBO closely tracks its non-private counterpart (GIBO) throughout, paying only a modest privacy cost. On the other hand, Kusner et al. [19] devised a framework for selecting candidate hyperparameter configurations using Bayesian optimization when first- and second-order derivative information is unavailable, while maintaining privacy in the validation dataset. Making BO differentially private was a significant development, and the authors show that their algorithm provided rapid convergence to a minimizing parameter configuration for certain classes of problems. However, as a classic global Bayesian Optimization procedure, it too suffers in higher dimensions. Here we introduce a differentially private analog of GIBO (DP-GIBO), which can be considered a black box approximation of Noisy Gradient Descent when gradients are unknown or prohibitively expensive to compute, and hence unavailable to the practitioner. Our algorithm enables efficient, private optimization over high-dimensional continuous spaces, thus filling the gap left by traditional global Bayesian optimization or random search routines. This makes DP-GIBO applicable to a wide range of private hyperparameter tuning problems where the validation loss is queried as a black box—ranging from a single regularization parameter in vanilla LASSO, to per-group penalties in group LASSO, to the joint tuning of regularization strengths and kernel length-scales in kernel methods, where the hyperparameter dimension can easily exceed one hundred. We prove that, for sufficiently well-behaved objectives, the suboptimality gap of the algorithm’s final estimate converges at a rate that depends polynomially on the problem dimensionality, up to an error scaling with the magnitude of the injected privacy noise. Figure 1 illustrates this scaling behavior on a private group LASSO tuning task, where the curse of dimensionality clearly affects random search but not DP-GIBO. Contributions. In summary, in this work we: (1) Propose DP-GIBO, a drop-in differentially private algorithm for tuning (potentially high-dimensional) continuous hyperparameters in a black box manner. (2) Prove convergence rates for the suboptimality gap of DP-GIBO under standard regularity conditions, with explicit dependence on dimension and privacy noise. (3) Empirically demonstrate that DP-GIBO substantially outperforms existing private black box tuning methods in higher-dimensional regimes where global Bayesian optimization and random search collapse. (4) Introduce a modification to the GIBO framework that enables tighter error control and yields a substantial computational speedup in moderate dimensions. 2 Preliminaries and Motivation This section is organized as follows. We first provide a primer on Differential Privacy and Gaussian Processes. We then formally define the problem setting, using the private tuning of regularization hyperparameters in a group LASSO as a motivating example. Finally, we position our framework at the intersection of Bayesian optimization, noisy gradient descent, and private hyperparameter tuning. 2.1 Differential Privacy We adopt the Gaussian Differential Privacy (GDP) framework [ 20 ] . Let \mathcal{M}:\mathcal{X}^{n}\to\mathcal{Y} denote a randomized mechanism taking as input \mathbf{x}=(x_{1},\dots,x_{n})\in\mathcal{X}^{n} , and call \mathbf{x},\mathbf{x}^{\prime} neighboring